Skip to content

Commit d1a673a

Browse files
committed
[SPARK-29924][DOCS] Document Apache Arrow JDK11 requirement
### What changes were proposed in this pull request? This adds a note for additional setting for Apache Arrow library for Java 11. ### Why are the changes needed? Since Apache Arrow 0.14.0, an additional setting is required for Java 9+. - https://issues.apache.org/jira/browse/ARROW-3191 It's explicitly documented at Apache Arrow 0.15.0. - https://issues.apache.org/jira/browse/ARROW-6206 However, there is no plan to handle that inside Apache Arrow side. - https://issues.apache.org/jira/browse/ARROW-7223 In short, we need to document this for the users who is using Arrow-related feature on JDK11. For dev environment, we handle this via [SPARK-29923](#26552) . ### Does this PR introduce any user-facing change? Yes. ### How was this patch tested? Generated document and see the pages. ![doc](https://user-images.githubusercontent.com/9700541/73096611-0f409d80-3e9a-11ea-804b-c6b5ec7bd78d.png) Closes #27356 from dongjoon-hyun/SPARK-JDK11-ARROW-DOC. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
1 parent ed44926 commit d1a673a

File tree

1 file changed

+2
-0
lines changed

1 file changed

+2
-0
lines changed

docs/index.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,8 @@ For the Scala API, Spark {{site.SPARK_VERSION}}
5252
uses Scala {{site.SCALA_BINARY_VERSION}}. You will need to use a compatible Scala version
5353
({{site.SCALA_BINARY_VERSION}}.x).
5454

55+
For Java 11, `-Dio.netty.tryReflectionSetAccessible=true` is required additionally for Apache Arrow library. This prevents `java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available` when Apache Arrow uses Netty internally.
56+
5557
# Running the Examples and Shell
5658

5759
Spark comes with several sample programs. Scala, Java, Python and R examples are in the

0 commit comments

Comments
 (0)