-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-22570][SQL] Avoid to create a lot of global variables by using a local variable with allocation of an object in generated code #19797
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
41a591a
de5cd38
1046e1d
13e05b7
072cdd2
d4e4b07
36a330d
3b8459d
420b10d
cd4c696
a455ac3
747e215
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
- Loading branch information
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -184,6 +184,23 @@ class CodegenContext { | |
| mutableStates += ((javaType, variableName, initCode)) | ||
| } | ||
|
|
||
| /** | ||
| * Add a mutable state as a field to the generated class if the name has not been added | ||
| * | ||
| * @param javaType Java type of the field. | ||
| * @param variableName Name of the field. | ||
| * @param initCode The statement(s) to put into the init() method to initialize this field. | ||
| * If left blank, the field will be default-initialized. | ||
| */ | ||
| def reuseOrAddMutableState( | ||
|
||
| javaType: String, | ||
| variableName: String, | ||
| initCode: String = ""): Unit = { | ||
| if (!mutableStates.exists(s => s._1 == variableName)) { | ||
| addMutableState(javaType, variableName, initCode) | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Add buffer variable which stores data coming from an [[InternalRow]]. This methods guarantees | ||
| * that the variable is safely stored, which is important for (potentially) byte array backed | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what if we create a new wrapper every time?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would work well, too. We may have some overhead to create a small object
IntWrapperand collect it at GC.Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shouldn't GC work very well in this case? My concern is, if there is no significant performance benefit, reusing global variables may be too hacky.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. If we do not reuse global variables, we can use only a local variable.