Skip to content

[SPARK-16866][SQL] Infrastructure for file-based SQL end-to-end tests #14472

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
petermaxlee wants to merge 11 commits into from
Closed

[SPARK-16866][SQL] Infrastructure for file-based SQL end-to-end tests #14472

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
ba9b678
[SPARK-16866][SQL] Basic infrastructure for file-based SQL end-to-end…
petermaxlee Aug 3, 2016
9b360da
rat exclude
petermaxlee Aug 3, 2016
a1e1b57
Use a more concise and human readable format.
petermaxlee Aug 3, 2016
2352d6f
rat exclude
petermaxlee Aug 5, 2016
34b6704
Merge remote-tracking branch 'apache/master' into SPARK-16866
petermaxlee Aug 9, 2016
0359756
code review
petermaxlee Aug 9, 2016
5eb01fe
make it work for maven
petermaxlee Aug 9, 2016
26c7771
Update example
petermaxlee Aug 10, 2016
7497742
Simplify filter
petermaxlee Aug 10, 2016
14f4959
cr
petermaxlee Aug 10, 2016
288b699
better message for assert.
petermaxlee Aug 10, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Simplify filter
  • Loading branch information
@petermaxlee
petermaxlee committed Aug 10, 2016
commit 7497742617420da77932fe20fff3d422b1a52fde
2 changes: 1 addition & 1 deletion sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ class SQLQueryTestSuite extends QueryTest with SharedSQLContext {
// val cleaned = input.split("\n").filterNot(_.matches("--.*(?<=[^\\\\]);")).mkString("\n")
Copy link
Contributor

@cloud-fan cloud-fan Aug 10, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this line?

Copy link
Contributor Author

@petermaxlee petermaxlee Aug 10, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

val cleaned = input.split("\n").filterNot(_.startsWith("--")).mkString("\n")
// note: this is not a robust way to split queries using semicolon, but works for now.
cleaned.split("(?<=[^\\\\]);").map(_.trim).filterNot(q => q == "").toSeq
cleaned.split("(?<=[^\\\\]);").map(_.trim).filter(_ != "").toSeq
Copy link
Contributor

@cloud-fan cloud-fan Aug 10, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you explain a bit more about this regex? Why can't we just use ; here?

Copy link
Contributor Author

@petermaxlee petermaxlee Aug 10, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I copied this from HiveComparisonTest. I think it is done to avoid escaping e.g. ";". It is really not very robust, but seems fine for now.

Copy link
Contributor

@cloud-fan cloud-fan Aug 10, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you explain a bit more about this regex? Why can't we just use ; here?

Copy link
Contributor Author

@petermaxlee petermaxlee Aug 10, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I copied this from HiveComparisonTest. I think it is done to avoid escaping e.g. ";". It is really not very robust, but seems fine for now.

}

// Run the SQL queries preparing them for comparison.
Expand Down