Skip to content

[SPARK-24781][SQL] Using a reference from Dataset in Filter/Sort might not work #21745

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
viirya wants to merge 9 commits into from
Closed

[SPARK-24781][SQL] Using a reference from Dataset in Filter/Sort might not work #21745

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
97837a4
Resolved references from Dataset should be checked if it is missed fr…
viirya Jul 11, 2018
b99d0c7
Address comments and fix bug.
viirya Jul 12, 2018
38a935d
Fix bug.
viirya Jul 12, 2018
eff3af2
Merge remote-tracking branch 'upstream/master' into SPARK-24781
viirya Jul 12, 2018
6eda8d2
Remove added test.
viirya Jul 12, 2018
8432b00
Add more tests and deal with aggregate.
viirya Jul 12, 2018
860d433
Add comments.
viirya Jul 12, 2018
a98f416
Update comment.
viirya Jul 13, 2018
9e00db9
Split original test case to two test cases.
viirya Jul 13, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove added test.
  • Loading branch information
@viirya
viirya committed Jul 12, 2018
commit 6eda8d2e8e56a9d9027d6cac49d23684396cb0ea
12 changes: 0 additions & 12 deletions sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/LogicalPlanSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,16 +119,4 @@ class LogicalPlanSuite extends SparkFunSuite {
OneRowRelation())
assert(result.sameResult(expected))
}

test("Logical plan with missing inputs should be unresolved") {
// Normally we won't add a missing resolved reference into a logical plan,
// but a valid query like `df.select(df("name")).filter(df("id") === 0)` can make a query
// like this.
val relation = LocalRelation(AttributeReference("a", IntegerType, nullable = true)())
val plan = Project(Stream(AttributeReference("b", IntegerType, nullable = true)()), relation)
assert(plan.expressions.forall(_.resolved))
assert(plan.childrenResolved)
assert(plan.missingInput.nonEmpty)
assert(!plan.resolved)
}
}