Skip to content

[SPARK-24402] [SQL] Optimize In expression when only one element in the collection or collection is empty #21442

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dbtsai wants to merge 9 commits into from
Closed

[SPARK-24402] [SQL] Optimize In expression when only one element in the collection or collection is empty #21442

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
7c44c70
Optimize `In`
dbtsai May 28, 2018
449613a
style
dbtsai May 28, 2018
7a354fc
Addressed feedback
dbtsai May 29, 2018
fa678f8
Merge branch 'master' into optimize-in
dbtsai Jul 16, 2018
61824f7
Merge branch 'master' into optimize-in
dbtsai Jul 16, 2018
23fedd8
Addressed feedback
dbtsai Jul 16, 2018
5079833
update
dbtsai Jul 16, 2018
c519c40
Add one more test
dbtsai Jul 16, 2018
9174a30
Remove duplicated code
dbtsai Jul 16, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Addressed feedback
  • Loading branch information
@dbtsai
dbtsai committed Jul 16, 2018
commit 23fedd8d65cb51201e1f032c938671ebc21eb432
8 changes: 4 additions & 4 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,14 +224,14 @@ object OptimizeIn extends Rule[LogicalPlan] {
If(IsNotNull(v), FalseLiteral, Literal(null, BooleanType))
case expr @ In(v, list) if expr.inSetConvertible =>
val newList = ExpressionSet(list).toSeq
if (newList.length == 1) {
if (newList.length == 1 && !list.isInstanceOf[ListQuery]) {
EqualTo(v, newList.head)
Copy link
Member

@gatorsmile gatorsmile May 29, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will fail, since the schema mismatches when the data type is struct. The test cases were added a few days ago. #21425

} else if (newList.size > SQLConf.get.optimizerInSetConversionThreshold) {
} else if (newList.length > SQLConf.get.optimizerInSetConversionThreshold) {
val hSet = newList.map(e => e.eval(EmptyRow))
InSet(v, HashSet() ++ hSet)
} else if (newList.size < list.size) {
} else if (newList.length < list.length) {
expr.copy(list = newList)
} else { // newList.length == list.length
} else { // newList.length == list.length && newList.length > 1
expr
}
}
Expand Down