Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
eabb65b
WIP nothing worked, just recording the progress
HeartSaVioR Sep 4, 2018
c3076d2
WIP not working yet... lots of implementations needed
HeartSaVioR Sep 6, 2018
9d59c7a
WIP Finished implementing UpdatingSessionIterator
HeartSaVioR Sep 6, 2018
b38f2b9
WIP add verification on precondition "rows in iterator are sorted by …
HeartSaVioR Sep 7, 2018
668c1f5
Rename SymmetricHashJoinStateManager to MultiValuesStateManager
HeartSaVioR Sep 8, 2018
9f63a3c
Move package of UpdatingSessionIterator
HeartSaVioR Sep 8, 2018
5d17ac8
WIP add MergingSortWithMultiValuesStateIterator, now integrating with…
HeartSaVioR Sep 10, 2018
ec33265
WIP the first version of working one! Still have lots of TODOs and FI…
HeartSaVioR Sep 13, 2018
8b210d5
Add more explanations
HeartSaVioR Sep 13, 2018
7255bca
Silly bugfix & block session window for batch query as of now
HeartSaVioR Sep 13, 2018
7b57fe5
More works: majorly split out updating session to individual physical…
HeartSaVioR Sep 13, 2018
969859b
Fix a silly bug and also add check for session window against batch q…
HeartSaVioR Sep 13, 2018
f5ecbdd
WIP Fixed eviction on update mode
HeartSaVioR Sep 13, 2018
b180772
WIP found root reason of broken UT... fixed it
HeartSaVioR Sep 13, 2018
0a2d731
WIP remove printing "explain" on UTs
HeartSaVioR Sep 13, 2018
6ee901e
WIP address session to batch query (+ python) as well... not having t…
HeartSaVioR Sep 13, 2018
395606b
WIP add more test on session batch query
HeartSaVioR Sep 13, 2018
f6bb34d
WIP add UT for sessions with keys overlapped
HeartSaVioR Sep 14, 2018
22fffd2
WIP refactor a bit
HeartSaVioR Sep 14, 2018
847f69e
WIP add more FIXMEs for javadoc, and remove invalid FIXMEs
HeartSaVioR Sep 14, 2018
104df13
WIP Repackage & remove unnecessary field
HeartSaVioR Sep 17, 2018
8108fc5
WIP addressed UPDATE mode, but doesn't look like performant
HeartSaVioR Sep 17, 2018
35c8fef
WIP remove FIXME since it is not relevant
HeartSaVioR Sep 17, 2018
6b1d1e0
WIP update numOutputRows for Append mode
HeartSaVioR Sep 17, 2018
86b3060
WIP apply aggregations when merging sessions
HeartSaVioR Sep 18, 2018
f7c2deb
WIP simplify the code a bit
HeartSaVioR Sep 18, 2018
a81616b
WIP address batch distinct query for sessionization
HeartSaVioR Sep 18, 2018
013785d
WIP remove debug statements for test code
HeartSaVioR Sep 18, 2018
37fffef
WIP remove debug informations
HeartSaVioR Sep 18, 2018
df95e72
WIP port Sessionization example to UT of session window
HeartSaVioR Sep 18, 2018
16d6421
WIP remove unnecessary thing
HeartSaVioR Sep 18, 2018
dc43300
WIP fix all the issues with sessionization example UTs
HeartSaVioR Sep 19, 2018
3637f60
WIP apply merging session in each partition before shuffling
HeartSaVioR Sep 19, 2018
0d53831
Fix scala checkstyle
HeartSaVioR Sep 20, 2018
a781400
Fix python style check
HeartSaVioR Sep 20, 2018
918dad2
WIP add complete mode, fix tricky bugs, apply ExternalAppendOnlyUnsaf…
HeartSaVioR Oct 8, 2018
fd6377b
WIP add "session" function to exclude list for description
HeartSaVioR Oct 8, 2018
e029e12
WIP rename function & column name "session" to "session_window"
HeartSaVioR Oct 10, 2018
a2fc652
WIP reducing unnecessary codegen which seriously harmed performance
HeartSaVioR Oct 15, 2018
2dc413b
WIP reduce codegen once again for MergingSessionsIterator
HeartSaVioR Oct 15, 2018
4dd0e89
WIP optimize a bit more on codegen...
HeartSaVioR Oct 15, 2018
cf52044
WIP make the feature "merge session in local partition" optional
HeartSaVioR Oct 15, 2018
5c74609
WIP add "session_window" to exclude list
HeartSaVioR Oct 17, 2018
fb6c59f
WIP Enable versioning of session window state format
HeartSaVioR Oct 18, 2018
dd29af2
WIP some correction in comment
HeartSaVioR Oct 19, 2018
1f6e496
WIP cover all the cases for session window in UTs
HeartSaVioR Oct 20, 2018
0673b6e
WIP Add Linked List data structure for storing session windows
HeartSaVioR Oct 23, 2018
4698f6d
WIP add SessionWindowLinkedListStateStoreRDD
HeartSaVioR Oct 23, 2018
5c67f72
WIP add more functionalities to SessionWindowLinkedListState
HeartSaVioR Oct 24, 2018
7bb0060
WIP it works but a bit suboptimal
HeartSaVioR Oct 25, 2018
35c9712
WIP optimized!
HeartSaVioR Oct 25, 2018
ede078a
WIP remove requirement on sort, add UT to test linked list state with…
HeartSaVioR Oct 27, 2018
f8e8ff6
WIP add code to print out information when task crashes with dangling…
HeartSaVioR Oct 27, 2018
b05abc7
WIP fixed the issue with benchmark run
HeartSaVioR Oct 29, 2018
17570f2
WIP optimize a bit on storing new sessions
HeartSaVioR Oct 30, 2018
958de31
WIP Fixed critical bug which tasks don't respect preference on state …
HeartSaVioR Oct 31, 2018
ee67bca
WIP Fix critical perf. issue: remove codegen on generating session ro…
HeartSaVioR Nov 1, 2018
8a0331e
WIP Rolling back unnecessary changes
HeartSaVioR Nov 2, 2018
b6ccecd
WIP Apply removing codegen to UpdatingSessionIterator as well
HeartSaVioR Nov 2, 2018
75c7611
WIP remove state version for now: it will be reintroduced when actual…
HeartSaVioR Nov 2, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
WIP reduce codegen once again for MergingSessionsIterator
  • Loading branch information
HeartSaVioR committed Oct 15, 2018
commit 2dc413beff4630b98e4b718d2146b34855816752
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
package org.apache.spark.sql.execution.aggregate

import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.{Attribute, BindReferences, CreateNamedStruct, Expression, GenericInternalRow, Literal, MutableProjection, NamedExpression, PreciseTimestampConversion, UnsafeProjection, UnsafeRow}
import org.apache.spark.sql.catalyst.expressions.{Attribute, BindReferences, CreateNamedStruct, Expression, GenericInternalRow, JoinedRow, Literal, MutableProjection, NamedExpression, PreciseTimestampConversion, UnsafeProjection, UnsafeRow}
import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
import org.apache.spark.sql.catalyst.expressions.codegen.GenerateUnsafeProjection
import org.apache.spark.sql.execution.metric.SQLMetric
Expand Down Expand Up @@ -219,8 +219,12 @@ class MergingSessionsIterator(
}
}

private val join = new JoinedRow

private val groupingKeyProj = GenerateUnsafeProjection.generate(groupingExpressions,
groupingWithoutSessionAttributes :+ sessionExpression.toAttribute)

private def generateGroupingKey(): UnsafeRow = {
// FIXME: Convert to JoinRow if possible to reduce codegen for unsafe projection
val sessionStruct = CreateNamedStruct(
Literal("start") ::
PreciseTimestampConversion(
Expand All @@ -230,17 +234,10 @@ class MergingSessionsIterator(
Literal(currentSessionEnd, LongType), LongType, TimestampType) ::
Nil)

val convertedGroupingExpressions = groupingExpressions.map { x =>
if (x.semanticEquals(sessionExpression)) {
sessionStruct
} else {
BindReferences.bindReference[Expression](x, groupingWithoutSessionAttributes)
}
}
val joined = join(currentGroupingKey,
UnsafeProjection.create(sessionStruct).apply(InternalRow.empty))

val proj = GenerateUnsafeProjection.generate(convertedGroupingExpressions,
groupingWithoutSessionAttributes)
proj(currentGroupingKey)
groupingKeyProj(joined)
}

def outputForEmptyGroupingKeyWithoutInput(): UnsafeRow = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,7 @@ class UpdatingSessionIterator(

private def closeCurrentSession(keyChanged: Boolean): Unit = {
// FIXME: Convert to JoinRow if possible to reduce codegen for unsafe projection
// FIXME: Same approach on MergingSessionsIterator.generateGroupingKey doesn't work here, why?
val sessionStruct = CreateNamedStruct(
Literal("start") ::
PreciseTimestampConversion(
Expand Down