[Java] java.lang.reflect.InaccessibleObjectException on Java 18

[asfimport]

Getting the following stack trace when running on Java 18.

BaseAllocator throws this when it calls the DefaultAllocationManagerFactory.

    private ArrowBuf createEmpty() {
        return this.allocationManagerFactory.empty();
    }

Caused by: java.lang.RuntimeException: Failed to initialize MemoryUtil.
	at org.apache.arrow.memory.util.MemoryUtil.<clinit>(MemoryUtil.java:136)
	... 20 more
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field long java.nio.Buffer.address accessible: module java.base does not \"opens java.nio\" to unnamed module @3453e479
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
	at org.apache.arrow.memory.util.MemoryUtil.<clinit>(MemoryUtil.java:84)
	... 20 more
 

Reporter: Daniel Glöckner

Related issues:

• [Java][CI]: Add support for Java 17 CI process (requires)

_{Note: This issue was originally created as ARROW-16449. Please see the migration documentation for further details.}

[asfimport]

David Li / @lidavidm:
Can you test with Arrow 8.0.0 once it releases (in the next few weeks)? Until ARROW-15755 was implemented there was no support for Java > 11

[asfimport]

David Li / @lidavidm:
CC @davisusanibar

[asfimport]

Raúl Cumplido / @raulcd:
Hi Daniel,

Thanks for your report. We are working towards supporting newer JDK versions.

Support for JDK 17 will be added to the next 8.0.0 release as you can see from this ticket https://issues.apache.org/jira/browse/ARROW-15755 .

Support for JDK 18 is expected to be added to a following release. You can follow how support for JDK 18 progresses on this ticket: https://issues.apache.org/jira/browse/ARROW-16267

[asfimport]

Daniel Glöckner:
Thanks @lidavidm and @raulcd. I will trace the Arrow release and re-test.

[asfimport]

Daniel Glöckner:
@lidavidm I re-tested with the Arrow 8.0 release and java 17.0.2.

The minimal code below only works if I add --add-opens java.base/java.nio=ALL-UNNAMED VM option. Is this expected?

import org.apache.arrow.memory.RootAllocator;

public class Arrow {

  public static void main(String[] args) {
    RootAllocator rootAllocator = new RootAllocator();
    var buf = rootAllocator.buffer(1024);
    System.out.println(buf);
  }
}

[asfimport]

David Li / @lidavidm:
[~[email protected]] I believe this is currently unavoidable because of Arrow/Netty's use of direct memory, cc @davisusanibar for comments. (JEP419 may help but we are a very long way off from being able to rely on that.)

[asfimport]

David Dali Susanibar Arce / @davisusanibar:
Hi [~[email protected]]  as per the jar artifacts are exposed as a legacy mode (it means any jar are running in the classpath is consider as a unnamed module) and base on JEP 200 The Modular JDK: yes, this is needed:

--add-opens ${module}/${package}=${target_modules}
Let the ${target_modules} access all types and members, regardless of visibility, from ${module}'s ${package}

Also this is added to the Arrow Java documentation 8.0.0 at https://arrow.apache.org/docs/java/install.html

 

—

 

This is a draft for implement JEP 200 The Modular JDK and be more restrict about to not give ALL-UNNAMED access but there is a lot of discussion we need to close before to convert this draft into PR: apache/arrow#13072

 

[asfimport]

David Dali Susanibar Arce / @davisusanibar:
Hi [~[email protected]] please could you help us if we could close this issue as resolved ?, thank you in advance.

[asfimport]

Fabian Mastenbroek:
Hi,

I stumbled upon this issue today and ideally I would like to avoid having to open java.base to unnamed modules. I took a look at the source code of MemoryUtil today and it seems there might be workaround for this issue.

The reflective call that is currently causing the failure is not necessary. Applying this patch to the Arrow tree seems to resolve this issue for me. However, without the suggested add-opens parameter, any call to MemoryUtil.directBuffer will now fail during runtime.

This patch resolves that issue for me. However, this is a rather hairy solution, since it overwrites fields of DirectByteBuffer using Unsafe to construct a buffer for an arbitrary memory address.

[asfimport]

David Li / @lidavidm:
Thanks [~fabianishere]!

The first patch seems reasonable. But the second patch seems rather risky, and from a quick look, without directBuffer, much of the Arrow library will still fail at runtime.

[asfimport]

David Li / @lidavidm:
CC @davisusanibar @lwhite1 for opinions

[lidavidm]

We can't solve this without rewriting the memory subsystem (unfortunately). FFM is a better potential solution.