Android 10 stacked USB permission when using intent-filter to start activity when USB attached - android-10.0

Question

My setup
Pixel 3/Android 10 (but same problem on Samsung Galaxy S9/Android 9)
App targets API 29 (Android 10)
A USB device
On Android 10 there has been a change to the permission model, which means reading serial number for a USB device requires an explicit user permission, see: https://developer.android.com/preview/privacy/data-identifiers#usb-serial-user-permission.
So when receiving an intent that a USB device is attached I automatically request a permission to read the serial number (results in a permission dialog: "Allow X to access Virtual COM Port ?").
To also be able to launch the application when an appropriate USB device was attached it was previously possible to also do this (AndroidManifest.xml):
<activity
android:name="MainActivity"
android:screenOrientation="portrait"
>
<intent-filter>
<action android:name="android.intent.action.MAIN"/>
<category android:name="android.intent.category.LAUNCHER"/>
</intent-filter>
<intent-filter>
<action
android:name="android.hardware.usb.action.USB_DEVICE_ATTACHED"
/>
</intent-filter>
<meta-data
android:name="android.hardware.usb.action.USB_DEVICE_ATTACHED"
android:resource="#xml/my_device_filter"
/>
</activity>
... which results in a permission request dialog also when a new USB device is attached ("Open X to handle Com Port?").
But, as previously mentioned, the application must also request permission for reading the serial number which results in a second permission dialog.
These two dialogs are now stacked and if agreeing to the first dialog ("Open X to handle Virtual COM Port ?"), the second one ("Allow X to access COM Port ?") is dismissed automatically leading to broken functionality.
Any suggestions on how to handle these two permissions?