Skip to content

[SPARK-18016][SQL] Code Generation: Constant Pool Limit - reduce entries for mutable state #19811

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kiszk wants to merge 26 commits into from
Closed

[SPARK-18016][SQL] Code Generation: Constant Pool Limit - reduce entries for mutable state #19811

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
d441794
update API of addMutableState
kiszk Nov 24, 2017
870d106
eliminate initialization with default value
kiszk Nov 25, 2017
24d7087
allocate a global Java array to store a lot of mutable state in a class
kiszk Nov 26, 2017
3eb5842
fix scala style error
kiszk Nov 26, 2017
074d711
fix test failure of ExpressionEncoderSuite.NestedArray
kiszk Nov 27, 2017
eafa3f8
rebase with master
kiszk Nov 30, 2017
90d15f3
address review comments
kiszk Nov 30, 2017
c456c07
add useFreshname parameter to addMutableState
kiszk Nov 30, 2017
9ca5ab3
rebase with master
kiszk Nov 30, 2017
5b36c61
fix test failures
kiszk Dec 1, 2017
fd51d75
drop to creat a loop for initialization
kiszk Dec 7, 2017
effe918
fix test failure
kiszk Dec 8, 2017
9df109c
update comments
kiszk Dec 8, 2017
634d494
address review comment
kiszk Dec 10, 2017
d3438fd
address review comment
kiszk Dec 12, 2017
f4f3754
address review comment
kiszk Dec 12, 2017
f1e1fca
address review comments except test case
kiszk Dec 13, 2017
0937ef2
rebase with master
kiszk Dec 13, 2017
4bfcc1a
Do not use compaction as possible for frequently-accessed variable
kiszk Dec 13, 2017
49119a9
exclude mutable state from argument list for ExpressionCodegn
kiszk Dec 13, 2017
24f49c5
fix test failures
kiszk Dec 14, 2017
15e967e
address review comments
kiszk Dec 14, 2017
d6c1a97
address review comments
kiszk Dec 14, 2017
a9d40e9
address review comments
kiszk Dec 14, 2017
31914c0
address review comments
kiszk Dec 15, 2017
0e45c19
address review comments
kiszk Dec 19, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix test failures
  • Loading branch information
@kiszk
kiszk committed Dec 14, 2017
commit 5b36c615c15cbb8f6f4ea1b5cde53b6feb63d420
7 changes: 6 additions & 1 deletion ...lyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,11 +203,16 @@ class CodegenContext {
useFreshName: Boolean = true): String = {
val varName = if (useFreshName) freshName(variableName) else variableName
Copy link
Contributor

@cloud-fan cloud-fan Dec 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of calling freshName here and adding a useFreshName parameter, can we follow the previous style and ask the caller side to guarantee the given name is unique? i.e. call freshName at caller side

Copy link
Member Author

@kiszk kiszk Dec 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since I noticed that most of caller sides executes freshName, I decided to use the new style that can simply caller code. If a developer want to guarantee the given name is unique at caller site (currently, they are only several cases), it is OK by using useFreshName = true.

Do we need redundant code at caller side? WDYT? @cloud-fan

Copy link
Contributor

@cloud-fan cloud-fan Dec 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't it an existing problem? Let's fix it in another PR to make this PR more consistent with the previous code.

Copy link
Member Author

@kiszk kiszk Dec 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, let us discuss in another PR.

Copy link
Contributor

@mgaido91 mgaido91 Dec 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this can be moved in the if for clarity

Copy link
Member Author

@kiszk kiszk Dec 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch

val initCode = codeFunctions(varName)
if (javaType.contains("[][]")) {
Thread.dumpStack()
}

if (inline ||
Copy link
Contributor

@cloud-fan cloud-fan Dec 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

val canInlinePrimitive = isPrimitiveType(javaType) &&
  mutableStates.length < CodeGenerator.OUTER_CLASS_VARIABLES_THRESHOLD
if (forceInline || canInlinePrimitive || javaType.contains("[][]")) ...

// want to put a primitive type variable at outerClass for performance
isPrimitiveType(javaType) &&
(mutableStates.length < CodeGenerator.OUTER_CLASS_VARIABLES_THRESHOLD)) {
(mutableStates.length < CodeGenerator.OUTER_CLASS_VARIABLES_THRESHOLD) ||
// type is multi-dimensional array
Copy link
Contributor

@cloud-fan cloud-fan Dec 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why can't support multi-dimensional array?

Copy link
Member Author

@kiszk kiszk Dec 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are changing a declared type for compaction (i.e. adding one dimension at the first dimension) at declareMutableStates. In the logic at declareMutableStates assumes that a given variable is scalar or one-dimensional array.
We could support multi-dimensional array. However, it requires slightly complicated string operations. Should we support multi-dimensional array or keep it simple?

@cloud-fan WDYT?

Copy link
Contributor

@cloud-fan cloud-fan Dec 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok let's leave it

javaType.contains("[][]")) {
mutableStates += ((javaType, varName, initCode))
varName
} else {
Expand Down
5 changes: 3 additions & 2 deletions ...yst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,8 +183,9 @@ class RegexpExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
val ctx = new CodegenContext
RegExpReplace(Literal("100"), Literal("(\\d+)"), Literal("num")).genCode(ctx)
// four global variables (lastRegex, pattern, lastReplacement, and lastReplacementInUTF8)
// are always required
assert(ctx.mutableStates.length == 4)
// are always required, which are allocated in type-based global array
assert(ctx.mutableStates.length == 0)
assert(ctx.mutableStateArrayInitCodes.length == 3)
}

test("RegexExtract") {
Expand Down