Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
7bf45dd
Adds zip function to sparksql
DylanGuedes Apr 11, 2018
99848fe
Changes zip construction
DylanGuedes Apr 13, 2018
27b0bc2
Changes tests and uses builtin namespace in pyspark
DylanGuedes Apr 13, 2018
93826b6
fixes examples string and uses struct instead of arrays
DylanGuedes Apr 26, 2018
a7e29f6
working pyspark zip_lists
DylanGuedes May 11, 2018
7130fec
Fixes java version when arrays have different lengths
DylanGuedes May 11, 2018
d552216
remove unused variables
DylanGuedes May 11, 2018
1fecef4
rename zip_lists to zip
DylanGuedes May 11, 2018
f71151a
adds expression tests and uses strip margin syntax
DylanGuedes May 12, 2018
6b4bc94
Adds variable number of inputs to zip function
DylanGuedes May 15, 2018
1549928
uses foldleft instead of while for iterating
DylanGuedes May 15, 2018
9f7bba1
rewritten some notation
DylanGuedes May 16, 2018
3ba2b4f
fix dogencode generation
DylanGuedes May 17, 2018
3a59201
Adds new tests, uses lazy val and split calls
DylanGuedes May 17, 2018
6462fa8
uses splitFunction
DylanGuedes May 17, 2018
8b1eb7c
move arraytypes to private member
DylanGuedes May 18, 2018
2bfba80
adds binary and array of array tests
DylanGuedes May 18, 2018
c3b062c
uses stored array types names
DylanGuedes May 18, 2018
d9b95c4
split input function using ctxsplitexpression
DylanGuedes May 18, 2018
26bbf66
uses splitexpression for inputs
DylanGuedes May 19, 2018
d9ad04d
Refactor cases, add new tests with empty seq, check size of array
DylanGuedes May 22, 2018
f29ee1c
Check empty seq as input
DylanGuedes May 22, 2018
c58d09c
Uses switch instead of if
DylanGuedes May 23, 2018
38fa996
refactor switch and else methods
DylanGuedes May 23, 2018
5b3066b
uses if instead of switch
DylanGuedes May 30, 2018
759a4d4
Not using storedarrtype anymore
DylanGuedes Jun 4, 2018
68e69db
split between empty and nonempty codegen
DylanGuedes Jun 4, 2018
12b3835
remove ternary if
DylanGuedes Jun 4, 2018
643cb9b
Fixes null values evaluation and adds back tests
DylanGuedes Jun 4, 2018
5876082
move to else
DylanGuedes Jun 4, 2018
0223960
remove unused lines
DylanGuedes Jun 4, 2018
2b88387
use zip alias
DylanGuedes Jun 5, 2018
bbc20ee
using same docs for all apis
DylanGuedes Jun 8, 2018
8d3a838
adds transient to method
DylanGuedes Jun 8, 2018
d8f3dea
rename zip function to arrays_zip
DylanGuedes Jun 10, 2018
3d68ea9
adds pretty_name for arrays_zip
DylanGuedes Jun 11, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
using same docs for all apis
Signed-off-by: DylanGuedes <[email protected]>
  • Loading branch information
DylanGuedes committed Jun 8, 2018
commit bbc20eec010c8709b426a221b117c079f7bf97e1
4 changes: 2 additions & 2 deletions python/pyspark/sql/functions.py
Original file line number Diff line number Diff line change
Expand Up @@ -2397,8 +2397,8 @@ def array_repeat(col, count):
@since(2.4)
def zip(*cols):
"""
Collection function: Merge two columns into one, such that the M-th element of the N-th
argument will be the N-th field of the M-th output element.
Collection function: Returns a merged array containing in the N-th position the
N-th value of each array given.

:param cols: columns in input
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: columns of arrays to be merged.


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -129,8 +129,10 @@ case class MapKeys(child: Expression)
}

@ExpressionDescription(
usage = """_FUNC_(a1, a2, ...) - Returns a merged array containing in the N-th position the
N-th value of each array given.""",
usage = """
_FUNC_(a1, a2, ...) - Returns a merged array containing in the N-th position the
N-th value of each array given.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This description looks a bit confusing to me. How about Returns a merged array of structs in which the N-th struct contains all N-th values of input arrays.?

""",
examples = """
Examples:
> SELECT _FUNC_(array(1, 2, 3), array(2, 3, 4));
Expand All @@ -151,7 +153,7 @@ case class Zip(children: Seq[Expression]) extends Expression with ExpectsInputTy

private lazy val arrayElementTypes = arrayTypes.map(_.elementType)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have more than one arrayElementTypes?


def mountSchema: StructType = {
private lazy val mountSchema: StructType = {
val fields = children.zip(arrayElementTypes).zipWithIndex.map {
case ((expr: NamedExpression, elementType), _) =>
StructField(expr.name, elementType, nullable = true)
Expand Down
4 changes: 2 additions & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/functions.scala
Original file line number Diff line number Diff line change
Expand Up @@ -3509,8 +3509,8 @@ object functions {
def map_entries(e: Column): Column = withExpr { MapEntries(e.expr) }

/**
* Merge multiple columns into a resulting one.
*
* Returns a merged array containing in the N-th position the N-th value
* of each array given.
* @group collection_funcs
* @since 2.4.0
*/
Expand Down