Skip to content

[WIP] [skip ci] Fuzz testing in Spark SQL #7625

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JoshRosen wants to merge 87 commits into from
Closed

[WIP] [skip ci] Fuzz testing in Spark SQL #7625

Changes from 1 commit
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
6f2b909
Fix SPARK-9292.
JoshRosen Jul 24, 2015
03120d5
Check condition type in resolved()
JoshRosen Jul 24, 2015
e1f462e
Initial commit for SQL expression fuzzing harness
JoshRosen Jul 11, 2015
f8daec7
Apply implicit casts (in a hacky way for now)
JoshRosen Jul 11, 2015
df00e7a
More messy WIP prototyping on expression fuzzing
JoshRosen Jul 11, 2015
2dcbc10
Add some comments; speed up classpath search
JoshRosen Jul 11, 2015
c20a679
Move dummy type coercion to a helper method
JoshRosen Jul 11, 2015
95860de
More code cleanup and comments
JoshRosen Jul 11, 2015
abaed51
Use non-mutable interpreted projection.
JoshRosen Jul 11, 2015
129ad6c
Log expression after coercion
JoshRosen Jul 11, 2015
e1f91df
Run tests in deterministic order
JoshRosen Jul 13, 2015
adc3c7f
Test with random inputs of all types
JoshRosen Jul 23, 2015
ae5e151
Ignore BinaryType for now, since it led to some spurious failures.
JoshRosen Jul 23, 2015
a354208
Begin to add a DataFrame API fuzzer.
JoshRosen Jul 24, 2015
13f8c56
Don't puts nulls into the DataFrame
JoshRosen Jul 24, 2015
dd16f4d
Print logical plans.
JoshRosen Jul 24, 2015
7f2b771
Fuzzer improvements.
JoshRosen Jul 24, 2015
326d759
Fix SPARK-9293
JoshRosen Jul 24, 2015
4a2c684
Merge branch 'SPARK-9293' into fuzz-test
JoshRosen Jul 24, 2015
37e4ce8
Support methods that take varargs Column parameters.
JoshRosen Jul 24, 2015
558f04a
Merge remote-tracking branch 'origin/master' into fuzz-test
JoshRosen Jul 24, 2015
2f1b802
Add analysis rule to detect sorting on unsupported column types (SPAR…
JoshRosen Jul 24, 2015
c0889c0
Merge branch 'SPARK-9295' into fuzz-test
JoshRosen Jul 24, 2015
d7a3535
[SPARK-9303] Decimal should use java.math.Decimal directly instead of…
JoshRosen Jul 24, 2015
74bbc8c
Merge branch 'SPARK-9303' into fuzz-test
JoshRosen Jul 24, 2015
bfe1451
Update to allow sorting by null literals
JoshRosen Jul 24, 2015
7a7ec4d
Merge branch 'SPARK-9295' into fuzz-test
JoshRosen Jul 24, 2015
55221fa
Shouldn't use SortMergeJoin when joining on unsortable columns.
viirya Jul 24, 2015
a240707
Use forall instead of exists for readability.
viirya Jul 24, 2015
dc94314
Merge remote-tracking branch 'origin/pr/7645/head' into fuzz-test
JoshRosen Jul 24, 2015
16dfac9
Merge remote-tracking branch 'origin/master' into fuzz-test
JoshRosen Jul 24, 2015
a3f30bf
Merge remote-tracking branch 'origin/master' into fuzz-test
JoshRosen Jul 24, 2015
68c0e97
Commit some outstanding changes.
JoshRosen Jul 24, 2015
2d4ed76
Move to fuzzing package.
JoshRosen Jul 24, 2015
ac8dd74
Begin to clean up random DF generator
JoshRosen Jul 25, 2015
0b3938b
Add basic backtracking to improve chance of generating executable plan.
JoshRosen Jul 26, 2015
c836884
Hacky approach to try to execute child plans first.
JoshRosen Jul 26, 2015
fb1a666
Merge remote-tracking branch 'origin/master' into fuzz-test
JoshRosen Jul 26, 2015
396c235
Enable Unsafe by default
JoshRosen Jul 21, 2015
71fe0bb
Ignore failing ScalaUDFSuite test.
JoshRosen Jul 21, 2015
04fbe3e
Do not use UnsafeExternalSort operator if codegen is disabled
JoshRosen Jul 21, 2015
a7979dc
Disable unsafe Exchange path when RangePartitioning is used
JoshRosen Jul 21, 2015
4fcae4a
Reduce page size to make HiveCompatibilitySuite pass.
JoshRosen Jul 21, 2015
7e49d0e
Fix use-after-free bug in UnsafeExternalSorter.
JoshRosen Jul 26, 2015
0b95b71
Merge branch 'unsafe-by-default' into fuzz-test
JoshRosen Jul 26, 2015
454c921
Hack to enable join types to be tested
JoshRosen Jul 26, 2015
11f80a3
[SPARK-9368][SQL] Support get(ordinal, dataType) generic getter in Un…
rxin Jul 27, 2015
9989064
JoinedRow.
rxin Jul 27, 2015
24a3e46
Added support for DateType/TimestampType.
rxin Jul 27, 2015
fb6ca30
Support BinaryType.
rxin Jul 27, 2015
0f57c55
Reset the changes in ExpressionEvalHelper.
rxin Jul 27, 2015
3063788
Reset the change for real this time.
rxin Jul 27, 2015
5e3e266
Merge remote-tracking branch 'origin/pr/7682/head' into fuzz-test
JoshRosen Jul 27, 2015
6214682
Fixes to null handling in UnsafeRow
JoshRosen Jul 28, 2015
4c09a78
Enable Unsafe by default
JoshRosen Jul 21, 2015
54579b1
Disable unsafe Exchange path when RangePartitioning is used
JoshRosen Jul 21, 2015
601fcbd
Reduce page size to make HiveCompatibilitySuite pass.
JoshRosen Jul 21, 2015
e5f7464
Add task completion callback to avoid leak in limit after sort
JoshRosen Jul 27, 2015
c8eb2ee
Fix test in UnsafeRowConverterSuite
JoshRosen Jul 28, 2015
ef1c62d
Also match TungstenProject in checkNumProjects
JoshRosen Jul 28, 2015
203f1d8
Use TaskAttemptIds to track unroll memory
JoshRosen Jul 28, 2015
d7a2788
Use TaskAttemptIds to track shuffle memory
JoshRosen Jul 28, 2015
b38e70f
Roll back fix in PySpark, which is no longer necessary
JoshRosen Jul 28, 2015
d8bd892
Fix capitalization
JoshRosen Jul 28, 2015
56edb41
Move Executor's cleanup into Task so that TaskContext is defined when…
JoshRosen Jul 28, 2015
f4f5859
More thread -> task changes
JoshRosen Jul 28, 2015
e2b69c9
Fix ShuffleMemoryManagerSuite
JoshRosen Jul 28, 2015
63492c4
Fix long line.
JoshRosen Jul 28, 2015
6dc34f4
Merge branch 'unsafe-row-null-fixes' into unsafe-by-default
JoshRosen Jul 28, 2015
6ac2d82
Merge branch 'unsafe-by-default' into fuzz-test
JoshRosen Jul 29, 2015
18615a6
Merge remote-tracking branch 'origin/master' into fuzz-test
JoshRosen Jul 30, 2015
704abc1
Merge remote-tracking branch 'origin/master' into fuzz-test
JoshRosen Aug 14, 2015
b549b3e
Merge remote-tracking branch 'origin/master' into fuzz-test
JoshRosen Aug 16, 2015
ca8168a
Fix compilation with latest master.
JoshRosen Aug 16, 2015
0c7e9d0
Update to ignore some new analysis exceptions.
JoshRosen Aug 16, 2015
fb0671f
Move RandomDataFrameGenerator to own file.
JoshRosen Aug 16, 2015
78a71af
WIP
JoshRosen Aug 17, 2015
3b06849
Filter failing BinaryType array test.
JoshRosen Aug 17, 2015
574130b
Merge remote-tracking branch 'origin/master' into fuzz-test
JoshRosen Aug 23, 2015
a4c9b33
WIP
JoshRosen Aug 24, 2015
d36f8f5
Merge remote-tracking branch 'origin/master' into fuzz-test
JoshRosen May 27, 2016
ae5055a
Also ignore BRound expression.
JoshRosen May 27, 2016
dfdab5e
Fix serializability.
JoshRosen May 27, 2016
bb4cc2a
Merge remote-tracking branch 'origin/master' into fuzz-test
JoshRosen Jul 29, 2016
e60f231
More input type validation.
JoshRosen Jul 29, 2016
94087cb
Updates for DataSet API.
JoshRosen Jul 29, 2016
d1d3d53
Merge remote-tracking branch 'origin/master' into fuzz-test
JoshRosen Aug 16, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix test in UnsafeRowConverterSuite
  • Loading branch information
@JoshRosen
JoshRosen committed Jul 28, 2015
commit c8eb2ee505cef0f65b1a2865cc217270d2fcd45f
4 changes: 2 additions & 2 deletions ...st/src/test/scala/org/apache/spark/sql/catalyst/expressions/UnsafeRowConverterSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,8 +146,8 @@ class UnsafeRowConverterSuite extends SparkFunSuite with Matchers {
assert(createdFromNull.getShort(3) === 0)
assert(createdFromNull.getInt(4) === 0)
assert(createdFromNull.getLong(5) === 0)
assert(java.lang.Float.isNaN(createdFromNull.getFloat(6)))
assert(java.lang.Double.isNaN(createdFromNull.getDouble(7)))
assert(createdFromNull.getFloat(6) === 0.0f)
assert(createdFromNull.getDouble(7) === 0.0d)
assert(createdFromNull.getUTF8String(8) === null)
assert(createdFromNull.getBinary(9) === null)
// assert(createdFromNull.get(10) === null)
Expand Down