Padding for TABLESWITCH can be applied multiple times due to grow on relocatables

[HeartSaVioR]

I have observed some compilation error on generated code for Spark, which is around switch statement.
(The generated code is compiled against Janino 3.0.8 but upgrading version to Janino 3.0.15 didn't seem to fix the issue.)

org.codehaus.janino.InternalCompilerException: Compiling "GeneratedClass": Compiling "expand_doConsume(short expand_expr_0, boolean expand_exprIsNull_0, int expand_expr_1, boolean expand_exprIsNull_1, Decimal expand_expr_2, boolean expand_exprIsNull_2, Decimal expand_expr_3, boolean expand_exprIsNull_3, Decimal expand_expr_4, boolean expand_exprIsNull_4, Decimal expand_expr_5, boolean expand_exprIsNull_5, Decimal expand_expr_6, boolean expand_exprIsNull_6, Decimal expand_expr_7, boolean expand_exprIsNull_7, Decimal expand_expr_8, boolean expand_exprIsNull_8, Decimal expand_expr_9, boolean expand_exprIsNull_9, long expand_expr_10, boolean expand_exprIsNull_10, long expand_expr_11, boolean expand_exprIsNull_11, long expand_expr_12, boolean expand_exprIsNull_12, long expand_expr_13, boolean expand_exprIsNull_13, long expand_expr_14, boolean expand_exprIsNull_14, long expand_expr_15, boolean expand_exprIsNull_15, long expand_expr_16, boolean expand_exprIsNull_16, long expand_expr_17, boolean expand_exprIsNull_17, long expand_expr_18, boolean expand_exprIsNull_18, long expand_expr_19, boolean expand_exprIsNull_19, Decimal expand_expr_20, boolean expand_exprIsNull_20, Decimal expand_expr_21, boolean expand_exprIsNull_21, Decimal expand_expr_22, boolean expand_exprIsNull_22, Decimal expand_expr_23, boolean expand_exprIsNull_23, Decimal expand_expr_24, boolean expand_exprIsNull_24, Decimal expand_expr_25, boolean expand_exprIsNull_25, Decimal expand_expr_26, boolean expand_exprIsNull_26, Decimal expand_expr_27, boolean expand_exprIsNull_27, Decimal expand_expr_28, boolean expand_exprIsNull_28, Decimal expand_expr_29, boolean expand_exprIsNull_29, Decimal expand_expr_30, boolean expand_exprIsNull_30, Decimal expand_expr_31, boolean expand_exprIsNull_31, Decimal expand_expr_32, boolean expand_exprIsNull_32, Decimal expand_expr_33, boolean expand_exprIsNull_33, Decimal expand_expr_34, boolean expand_exprIsNull_34, Decimal expand_expr_35, boolean expand_exprIsNull_35, Decimal expand_expr_36, boolean expand_exprIsNull_36, Decimal expand_expr_37, boolean expand_exprIsNull_37, Decimal expand_expr_38, boolean expand_exprIsNull_38, Decimal expand_expr_39, boolean expand_exprIsNull_39, Decimal expand_expr_40, boolean expand_exprIsNull_40, Decimal expand_expr_41, boolean expand_exprIsNull_41, long expand_expr_42, boolean expand_exprIsNull_42, long expand_expr_43, boolean expand_exprIsNull_43, long expand_expr_44, boolean expand_exprIsNull_44, long expand_expr_45, boolean expand_exprIsNull_45, long expand_expr_46, boolean expand_exprIsNull_46, long expand_expr_47, boolean expand_exprIsNull_47, long expand_expr_48, boolean expand_exprIsNull_48, long expand_expr_49, boolean expand_exprIsNull_49, long expand_expr_50, boolean expand_exprIsNull_50, long expand_expr_51, boolean expand_exprIsNull_51, long expand_expr_52, boolean expand_exprIsNull_52, long expand_expr_53, boolean expand_exprIsNull_53, long expand_expr_54, boolean expand_exprIsNull_54, long expand_expr_55, boolean expand_exprIsNull_55, long expand_expr_56, boolean expand_exprIsNull_56, long expand_expr_57, boolean expand_exprIsNull_57, long expand_expr_58, boolean expand_exprIsNull_58, long expand_expr_59, boolean expand_exprIsNull_59, long expand_expr_60, boolean expand_exprIsNull_60, long expand_expr_61, boolean expand_exprIsNull_61, long expand_expr_62, boolean expand_exprIsNull_62, long expand_expr_63, boolean expand_exprIsNull_63)"; expand_doConsume(short expand_expr_0, boolean expand_exprIsNull_0, int expand_expr_1, boolean expand_exprIsNull_1, Decimal expand_expr_2, boolean expand_exprIsNull_2, Decimal expand_expr_3, boolean expand_exprIsNull_3, Decimal expand_expr_4, boolean expand_exprIsNull_4, Decimal expand_expr_5, boolean expand_exprIsNull_5, Decimal expand_expr_6, boolean expand_exprIsNull_6, Decimal expand_expr_7, boolean expand_exprIsNull_7, Decimal expand_expr_8, boolean expand_exprIsNull_8, Decimal expand_expr_9, boolean expand_exprIsNull_9, long expand_expr_10, boolean expand_exprIsNull_10, long expand_expr_11, boolean expand_exprIsNull_11, long expand_expr_12, boolean expand_exprIsNull_12, long expand_expr_13, boolean expand_exprIsNull_13, long expand_expr_14, boolean expand_exprIsNull_14, long expand_expr_15, boolean expand_exprIsNull_15, long expand_expr_16, boolean expand_exprIsNull_16, long expand_expr_17, boolean expand_exprIsNull_17, long expand_expr_18, boolean expand_exprIsNull_18, long expand_expr_19, boolean expand_exprIsNull_19, Decimal expand_expr_20, boolean expand_exprIsNull_20, Decimal expand_expr_21, boolean expand_exprIsNull_21, Decimal expand_expr_22, boolean expand_exprIsNull_22, Decimal expand_expr_23, boolean expand_exprIsNull_23, Decimal expand_expr_24, boolean expand_exprIsNull_24, Decimal expand_expr_25, boolean expand_exprIsNull_25, Decimal expand_expr_26, boolean expand_exprIsNull_26, Decimal expand_expr_27, boolean expand_exprIsNull_27, Decimal expand_expr_28, boolean expand_exprIsNull_28, Decimal expand_expr_29, boolean expand_exprIsNull_29, Decimal expand_expr_30, boolean expand_exprIsNull_30, Decimal expand_expr_31, boolean expand_exprIsNull_31, Decimal expand_expr_32, boolean expand_exprIsNull_32, Decimal expand_expr_33, boolean expand_exprIsNull_33, Decimal expand_expr_34, boolean expand_exprIsNull_34, Decimal expand_expr_35, boolean expand_exprIsNull_35, Decimal expand_expr_36, boolean expand_exprIsNull_36, Decimal expand_expr_37, boolean expand_exprIsNull_37, Decimal expand_expr_38, boolean expand_exprIsNull_38, Decimal expand_expr_39, boolean expand_exprIsNull_39, Decimal expand_expr_40, boolean expand_exprIsNull_40, Decimal expand_expr_41, boolean expand_exprIsNull_41, long expand_expr_42, boolean expand_exprIsNull_42, long expand_expr_43, boolean expand_exprIsNull_43, long expand_expr_44, boolean expand_exprIsNull_44, long expand_expr_45, boolean expand_exprIsNull_45, long expand_expr_46, boolean expand_exprIsNull_46, long expand_expr_47, boolean expand_exprIsNull_47, long expand_expr_48, boolean expand_exprIsNull_48, long expand_expr_49, boolean expand_exprIsNull_49, long expand_expr_50, boolean expand_exprIsNull_50, long expand_expr_51, boolean expand_exprIsNull_51, long expand_expr_52, boolean expand_exprIsNull_52, long expand_expr_53, boolean expand_exprIsNull_53, long expand_expr_54, boolean expand_exprIsNull_54, long expand_expr_55, boolean expand_exprIsNull_55, long expand_expr_56, boolean expand_exprIsNull_56, long expand_expr_57, boolean expand_exprIsNull_57, long expand_expr_58, boolean expand_exprIsNull_58, long expand_expr_59, boolean expand_exprIsNull_59, long expand_expr_60, boolean expand_exprIsNull_60, long expand_expr_61, boolean expand_exprIsNull_61, long expand_expr_62, boolean expand_exprIsNull_62, long expand_expr_63, boolean expand_exprIsNull_63): Operand stack inconsistent at offset 578: Previous size 1, now 0

With some investigation I found that there's a "branch" before TABLESWITCH OP which requires "growing" - so below situation is happening:

1. In the first loop of fixUpAndRelocate
   1.1) The padding for TABLESWITCH is applied first in fixUp.
   1.2) In relocate, some branch requires "growing" and insert some spaces "before" TABLESWITCH. In result, TABLESWITCH OP and padded offsets are shifted, which "may" break padding.
2. In the second loop of fixUpAndRelocate
   2.1) The padding for TABLESWITCH is applied AGAIN in fixUp, which adds more spaces if padding was broken in 1.2)
3. fixUpAndRelocate is finished and flowAnalysis runs
4. flowAnalysis reads the part of default for TABLESWITCH which is 0 due to "double padding", hence the offset has been same as TABLESWITCH OP, and error is thrown.

Janino 3.0.8

error-codegen.log
disassembled-gencode.log

[HeartSaVioR]

Actually I've crafted a patch but I'm not expert of JVM (just dug up with JLS only on TABLESWITCH) so not sure I assumed correctly.

My assumptions are:

1. growing code only requires one stage - once all Relocatables check for grow and ensure the space, it will not require another growing
   (Branch.relocate() seems to be the only thing requiring grow, and it requires growing only once, at least based on the code implementation.)
2. after the stage for growing, fixUp is the only thing requiring grow - this can be also done in one stage
3. after running fixUp, there's no need for growing, hence relocation would also be done in another one stage

If my assumptions are all correct, fixUpAndRelocate can just execute three phases instead of loop - 1) grow 2) fixUp 3) relocate - and then we can add grow in Relocatable interface to insert the space "if needed". (Looks like Branch only needed this.)

If these all make sense I'll propose the PR. If there's something major I am missing here, I'd love to see your fix being landed to Janino and released.

Thanks in advance!

[HeartSaVioR]

disassembled-gencode-janino-3-1-1-snapshot.log

Also attached the disassembled gencode while running the code with Janino 3.1.1-SNAPSHOT. It doesn't throw exception (as flowAnalysis has been removed), but there's no code in "expand_doConsume" as well as no "tableswitch" is presented in the log.

It would be nice to avoid crashing in runtime - is there any chance to bring flowAnalysis back, or put alternative verification?

[aunkrig]

Your disagnosis is exactly right, and I just merged your PR #114 that fixes the problem.

Lucky me that my recent massive refactoring was complete before you started developing the fix ;-) ...

[HeartSaVioR]

Great! Do you have any remaining works before releasing? Since we've affected by this, I'd like to see the new release sooner. (Honestly Spark community is depending on 3.0.x and new 3.0.x release would be easier to consume, but looks like Janino does linear release with single version line so OK.)

Btw, I like the way Janino does some verification and throws compilation error, like flowAnalysis for this case. Could we please reconsider restoring verification logic on the new code?

[aunkrig]

A release would not be a problem at all... maybe I can do it today!

[aunkrig]

Regarding "flowAnalysis()": Its sole purpose was to calculate the max_stack field of a method's code attribute. That field is totally redundant because the JVM that loads the class must calculate the maximum operand stack size anyway (if only to compare it with the value of the max_stack field). The same holds true for the cursed StackMapTable attribute, which is mandatory for Java 7+ .class files and cost me hours and hours to implement.

The new StackMapTable logic supersedes flowAnalysis(), which was always sort of foreign matter, because it runs after code generation and has thus only very little context information. My observation is also that the VerifyError messages produced by recent JVMs are much more helpful than those of flowAnalysis().

Which reasons do you see why flowAnalysis() should come back?

[HeartSaVioR]

Please refer my previous comment:
#113 (comment)

I've even observed the case where the operation(s) is(are) silently removed due to bad case, Janino bug. (In above case, the content of static method expand_doConsume looks to be "empty".) It was throwing exception on flowAnalysis as there's some minimal assertion (and hopefully this case it hits the assertion) and user query will simply fail to run for Spark case, but now there's a possible chance the compiled code runs as normal but no longer be the same as input source code, leading wrong result and negative impact on business.

I'm not sure where it "drops" the operations (and #114 would have fixed it so we should find the place without applying #114) but I'd like to see Janino throw error instead of producing unintended output. "Fail to compile" is the thing we can be noticed early and try to investigate. "Run and produce wrong result" is hard to find out and once we find out it have been already impacting the business badly.

[HeartSaVioR]

The compiler is the one we heavily rely on while we use static type languages to avoid as many possible errors as possible in runtime. This was from Janino bug so may not easy to assert from itself, but whenever possible or in doubt it would be nice to throw out error instead of taking risk of going forward. I'll recheck whether my comment is a red-herring; if it's not a red-herring then I'd like to see how we can assert it and prevent dropping out operations.

[aunkrig]

Version 3.1.1 is out the door!

[aunkrig]

Producing loadable but incorrect code is certainly a no-no. Can you provide a unit test that reproduces the problem with JANINO 3.1.1? And in a new issue, please...