Skip to content
Closed
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
7bf45dd
Adds zip function to sparksql
DylanGuedes Apr 11, 2018
99848fe
Changes zip construction
DylanGuedes Apr 13, 2018
27b0bc2
Changes tests and uses builtin namespace in pyspark
DylanGuedes Apr 13, 2018
93826b6
fixes examples string and uses struct instead of arrays
DylanGuedes Apr 26, 2018
a7e29f6
working pyspark zip_lists
DylanGuedes May 11, 2018
7130fec
Fixes java version when arrays have different lengths
DylanGuedes May 11, 2018
d552216
remove unused variables
DylanGuedes May 11, 2018
1fecef4
rename zip_lists to zip
DylanGuedes May 11, 2018
f71151a
adds expression tests and uses strip margin syntax
DylanGuedes May 12, 2018
6b4bc94
Adds variable number of inputs to zip function
DylanGuedes May 15, 2018
1549928
uses foldleft instead of while for iterating
DylanGuedes May 15, 2018
9f7bba1
rewritten some notation
DylanGuedes May 16, 2018
3ba2b4f
fix dogencode generation
DylanGuedes May 17, 2018
3a59201
Adds new tests, uses lazy val and split calls
DylanGuedes May 17, 2018
6462fa8
uses splitFunction
DylanGuedes May 17, 2018
8b1eb7c
move arraytypes to private member
DylanGuedes May 18, 2018
2bfba80
adds binary and array of array tests
DylanGuedes May 18, 2018
c3b062c
uses stored array types names
DylanGuedes May 18, 2018
d9b95c4
split input function using ctxsplitexpression
DylanGuedes May 18, 2018
26bbf66
uses splitexpression for inputs
DylanGuedes May 19, 2018
d9ad04d
Refactor cases, add new tests with empty seq, check size of array
DylanGuedes May 22, 2018
f29ee1c
Check empty seq as input
DylanGuedes May 22, 2018
c58d09c
Uses switch instead of if
DylanGuedes May 23, 2018
38fa996
refactor switch and else methods
DylanGuedes May 23, 2018
5b3066b
uses if instead of switch
DylanGuedes May 30, 2018
759a4d4
Not using storedarrtype anymore
DylanGuedes Jun 4, 2018
68e69db
split between empty and nonempty codegen
DylanGuedes Jun 4, 2018
12b3835
remove ternary if
DylanGuedes Jun 4, 2018
643cb9b
Fixes null values evaluation and adds back tests
DylanGuedes Jun 4, 2018
5876082
move to else
DylanGuedes Jun 4, 2018
0223960
remove unused lines
DylanGuedes Jun 4, 2018
2b88387
use zip alias
DylanGuedes Jun 5, 2018
bbc20ee
using same docs for all apis
DylanGuedes Jun 8, 2018
8d3a838
adds transient to method
DylanGuedes Jun 8, 2018
d8f3dea
rename zip function to arrays_zip
DylanGuedes Jun 10, 2018
3d68ea9
adds pretty_name for arrays_zip
DylanGuedes Jun 11, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fixes java version when arrays have different lengths
Signed-off-by: DylanGuedes <[email protected]>
  • Loading branch information
DylanGuedes committed Jun 4, 2018
commit 7130fec16b833a47b7ba4390e29425f989cb6da6
Original file line number Diff line number Diff line change
Expand Up @@ -157,22 +157,49 @@ case class ZipLists(left: Expression, right: Expression)
val i = ctx.freshName("i")
val values = ctx.freshName("values")
val len1 = ctx.freshName("len1")
val schema = ctx.freshName("schema")
val len2 = ctx.freshName("len2")
val pair = ctx.freshName("pair")
val higher = ctx.freshName("higher")
val leftType = left.dataType.asInstanceOf[ArrayType].elementType
val rightType = right.dataType.asInstanceOf[ArrayType].elementType
val getValue1 = CodeGenerator.getValue(
arr1, left.dataType.asInstanceOf[ArrayType].elementType, i)
val getValue2 = CodeGenerator.getValue(
arr2, right.dataType.asInstanceOf[ArrayType].elementType, i)

s"""
int $len1 = $arr1.numElements();
int $len2 = $arr2.numElements();
int $higher = $len2;

Object[] $values;
$values = new Object[$len1];
for (int $i = 0; $i < $len1; $i ++) {
Object[] $pair;
$pair = new Object[2];
$pair[0] = $getValue1;
$pair[1] = $getValue2;
$values[$i] = new $genericInternalRow($pair);
if ($len1 > $len2) {
$values = new Object[$len1];
for (int $i = 0; $i < $len1; $i ++) {
Object[] $pair;
$pair = new Object[2];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can merge this with the previous line

$pair[0] = $getValue1;
if ($i >= $len2) {
$pair[1] = null;
} else {
$pair[1] = $getValue2;
}
$values[$i] = new $genericInternalRow($pair);
}
} else {
$values = new Object[$len2];
for (int $i = 0; $i < $len2; $i ++) {
Object[] $pair;
$pair = new Object[2];
$pair[1] = $getValue2;
if ($i >= $len1) {
$pair[0] = null;
} else {
$pair[0] = $getValue1;
}
$values[$i] = new $genericInternalRow($pair);
}
}
${ev.value} = new $genericArrayData($values);
"""
Expand Down