"Swing-Shell" java.lang.InternalError: Could not initialize COM: HRESULT=0x80010106 - windows-store

Question

I have a Java 9 app that I'm trying to package for the Windows Store. The strange thing is that it works as expected when I run the exe-launcher directly, but I get the following strange error when I run the launcher via the APPX package:
Exception in thread "Swing-Shell" java.lang.InternalError: Could not
initialize COM: HRESULT=0x80010106
at java.desktop/sun.awt.shell.Win32ShellFolderManager2.initializeCom(Native Method)
at java.desktop/sun.awt.shell.Win32ShellFolderManager2$ComInvoker$1.run(Unknown Source at java.base/java.lang.Thread.run(Unknown Source)
HRESULT=0x80010106 means RPC_E_CHANGED_MODE which I guess means that COM is somehow already initialized in MTA mode. But why is this only an issue in the Windows Bridge sandbox? Does the Windows Bridge somehow pre-initialize COM somehow for some reason?
I'm not sure if this is a Java 9 issue, or a Desktop Bridge issue, or both. Does anybody have any ideas on how to identify the cause of the issue or workaround?
I have made a minimal Sample Project to reproduce the issue
The application works when executed directly, but not when executed via the APPX launcher. Why?

Answer 1

Analysis
The JDK parts, that rely on COM being initialized (D3DPipeline, Sound and windows shellfolder access) all follow the same pattern:
the run the code in a separate thread
for this thread CoInitialize is called
at the end of the user level code CoUnitialize is called
This is in line with the documentation the MSDN holds for COM and the analysis is correct, the error indicates, that the COM subsystem is already initialized to be MTA threaded.
So I modified the java launcher (jvm.dll) and inserted
debugging statements into some of the native methods in os_windows.cpp.
I focused on the threading methods. What I found was this:
create_main_thread, create_os_thread, pd_start_thread all
run with COM not yet initialized
the native thread initializer (thread_native_entry) is already
running with COM initialized
I looked more into in _beginthreadex and indeed I finally found a lead
on stackoverflow. That pointed me to the sourcecode of threadex.c,
which is part of the Visual Studio 2013 Express Installation.
There you find, that _beginthreadex does not directly start the
supplied thread function, but runs a library initializer
(_threadstartex) first. Part of this initializer reads:
_ptd->_initapartment = __crtIsPackagedApp();
if (_ptd->_initapartment)
{
_ptd->_initapartment = _initMTAoncurrentthread();
}
_callthreadstartex();
_crtIsPackagedApp detects via a kernel function if the application is
run as a "PackagedApp" (i.e. AppX package) and if so, then the
RoInitialize function is called, which from my understanding acts like the big
brother of CoInitialize.
Long story short: If your application is build with Visual Studio 2013
and run as a packaged application, you get a broken environment.
It was confirmed, that the working versions of the Oracle JDK were build with VS2010 SP1 and the broken version was build with VS2013SP4.
Workaround
With the above information google finally found a reference, that supports the analysis:
https://blogs.msdn.microsoft.com/vcblog/2016/07/07/using-visual-c-runtime-in-centennial-project/
Quote from that article:
Note that the existing VC++ 12.0 libraries created during the
Windows 8 timeframe have runtime checks to determine whether the
app is running under the app container or not. When running
desktop apps as a packaged app, these checks might limit the
functionality of the desktop app or cause it to behave like a UWA
(Universal Windows Application) (limited file system access or
create thread initializing MTA etc.). We have fixed this behavior
in the VC++ libraries contained in these framework packages and
thus removing the modern app limitations from your desktop
applications.
So I see to options to fix applications, that shall be distributed as AppX packages:
either force the users to have the updates VC++ 12.0 binaries installed (by introducing the dependency cited in the blog post) or
replace the msvcr120.dll that is bundled with Java 9 (and I assume also Java 10) with the fixed version from the update packages
I would go with the second version and I tested this. Starting with a clean up-to-date Windows 10 System, I installed JDK 9.0.4, I cloned the supplied testcase, modified it use the locally installed JRE (not the JDK!) and build the appx package. Running this, I reproduced the problem. I then replaced the msvcr120.dll's in the JRE folder of my systems installation with the one contained in the APPX container from:
https://www.microsoft.com/en-us/download/details.aspx?id=53176
Hint: *.appx are just ZIP files with additional signatures, so you can
just rename them and extract the contents.
I rebuild the testcase and it is working as it should (no COM
initialization errors anymore).