I have encountered a serialisation problem in Corda. My new code path contains no new classes that haven't already been serialised/deserialised, but blows up with the following exception. Note that it contains no CorDapp specific classes in the stack trace, which makes it quite hard to figure out what the problem is.
Are there any hints/tips as to how to start to investigate what the underlying cause might be for this sort of exception?
Thanks :-)
com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException
Serialization trace:
dataObject (co.paralleluniverse.fibers.Stack)
stack (net.corda.node.services.statemachine.FlowStateMachineImpl)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)
at com.esotericsoftware.kryo.serializers.CompatibleFieldSerializer.read(CompatibleFieldSerializer.java:147)
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:782)
at co.paralleluniverse.io.serialization.kryo.ReplaceableObjectKryo.readObjectOrNull(ReplaceableObjectKryo.java:107)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:540)
at co.paralleluniverse.fibers.Fiber$FiberSerializer.read(Fiber.java:2137)
at co.paralleluniverse.fibers.Fiber$FiberSerializer.read(Fiber.java:2067)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813)
at co.paralleluniverse.io.serialization.kryo.ReplaceableObjectKryo.readClassAndObject(ReplaceableObjectKryo.java:112)
at net.corda.nodeapi.internal.serialization.kryo.AbstractKryoSerializationScheme$deserialize$$inlined$use$lambda$1$1.invoke(KryoSerializationScheme.kt:97)
at net.corda.nodeapi.internal.serialization.kryo.AbstractKryoSerializationScheme$deserialize$$inlined$use$lambda$1$1.invoke(KryoSerializationScheme.kt:37)
at net.corda.nodeapi.internal.serialization.kryo.AbstractKryoSerializationScheme.withContext(KryoSerializationScheme.kt:80)
at net.corda.nodeapi.internal.serialization.kryo.AbstractKryoSerializationScheme.access$withContext(KryoSerializationScheme.kt:37)
at net.corda.nodeapi.internal.serialization.kryo.AbstractKryoSerializationScheme$deserialize$$inlined$use$lambda$1.execute(KryoSerializationScheme.kt:95)
at com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.run(KryoPoolQueueImpl.java:61)
at net.corda.nodeapi.internal.serialization.kryo.AbstractKryoSerializationScheme.deserialize(KryoSerializationScheme.kt:94)
at net.corda.nodeapi.internal.serialization.SerializationFactoryImpl$deserialize$1$1.invoke(SerializationScheme.kt:111)
at net.corda.core.serialization.SerializationFactory.withCurrentContext(SerializationAPI.kt:66)
at net.corda.nodeapi.internal.serialization.SerializationFactoryImpl$deserialize$1.invoke(SerializationScheme.kt:111)
at net.corda.nodeapi.internal.serialization.SerializationFactoryImpl$deserialize$1.invoke(SerializationScheme.kt:86)
at net.corda.core.serialization.SerializationFactory.asCurrent(SerializationAPI.kt:80)
at net.corda.nodeapi.internal.serialization.SerializationFactoryImpl.deserialize(SerializationScheme.kt:111)
at net.corda.node.services.statemachine.StateMachineManagerImpl.deserializeFiber(StateMachineManagerImpl.kt:715)
at net.corda.node.services.statemachine.StateMachineManagerImpl.access$deserializeFiber(StateMachineManagerImpl.kt:63)
at net.corda.node.services.statemachine.StateMachineManagerImpl$updateCheckpoint$2.run(StateMachineManagerImpl.kt:551)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsupportedOperationException
at java.util.AbstractCollection.add(AbstractCollection.java:262)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813)
at co.paralleluniverse.io.serialization.kryo.ReplaceableObjectKryo.readClassAndObject(ReplaceableObjectKryo.java:112)
at de.javakaffee.kryoserializers.UnmodifiableCollectionsSerializer.read(UnmodifiableCollectionsSerializer.java:71)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731)
at co.paralleluniverse.io.serialization.kryo.ReplaceableObjectKryo.readObject(ReplaceableObjectKryo.java:92)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:392)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:303)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731)
at co.paralleluniverse.io.serialization.kryo.ReplaceableObjectKryo.readObject(ReplaceableObjectKryo.java:92)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
... 28 more
[INFO ] 17:02:38,664 [Mock network] (FlowStateMachineImpl.kt:432) flow.[66a8e516-7d9e-4ff3-a922-c80dda86768e].initiateSession - Initiating flow session with party O=Notary Service, L=Zurich, C=CH. Session id for tracing purposes is SessionId(toLong=763122305110574148). {}
[ERROR] 17:02:38,660 [CheckpointChecker-1] (StateMachineManagerImpl.kt:417) flow.[66a8e516-7d9e-4ff3-a922-c80dda86768e].deserializeFiber - Encountered unrestorable checkpoint! {}
com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException
Post some awesome help from Kat and Andras # R3, we got to the bottom of this issue. I've been asked to post this here so that there is a bit more google help!
Solution (TL;DR)
I had a map of objects, and for each of those objects I wanted to call to a method that essentially kicked off two sub flows (under the handleEvent method).
This was failing:
modelDataMap.values.forEach { it.handleEvent(event, this, flowLogic, notary) }
but this fixes the problem:
modelDataMap.values.toList().forEach { it.handleEvent(event, this, flowLogic, notary) }
I think that in the first case, Kryo deserialised the values as an unmodifiable collection and was then unable to add elements to it. The toList resulted in Kryo creating a collection to which it could add values and everything was fine.
How to figure this out
I was aware of the code path that had changed that triggered this issue. In my case I had a log line that preceded the stack trace that gave me a hint about where to start.
Andras pointed out two useful tools at this point.
Missing #Suspendable annotations
If you add the following flag to your test:
-Dco.paralleluniverse.fibers.verifyInstrumentation=true
you will be informed of methods in the stack that do not have the #Suspendable annotation. This is actually really helpful.
FlowStackSnapshot
Andras has added a tool to FlowLogic that can be used to print out the stack. We had the idea of adding a method to this that will iterate through the stack, recursively serialising and deserialising everything, so that it's possible to identify where the problem is coming from.
For the moment, adding this to a suitable point just before you blow up will atleast allow visibility of the stack from Kryo's point of view:
val snapshot = flowLogic.flowStackSnapshot()
Note that as of Corda 3.2, there is an #Suspendable missing in the code path for flowStackSnapshot so don't use the verifyInstrumentation flag with flowStackSnapshot for the moment :-)
Once again - huge thanks to Kat and Andras for helping to get to the bottom of this.
Related
This is a simple send port, in which I'm trying to setup a filter over a promoted property. Whenever I attempt to enlist this port, I get the following error:
===================================
Could not update Send Port 'SendPort1' in Message Box. Exception from HRESULT: 0xC0C01B22 (Microsoft.BizTalk.ExplorerOM)
------------------------------
For help, click: http://go.microsoft.com/fwlink/?LinkId=47400&ProdName=Microsoft+BizTalk+Server+2013&ProdVer=3.10.229.0&EvtSrc=Microsoft.BizTalk.ExplorerOM.Resources&EvtID=IDS_ERR_SENDPORT_UPDATE
------------------------------
Program Location:
at Microsoft.BizTalk.ExplorerOM.BtsCatalogExplorer.SaveChangesWithTransaction(Object transactionObj)
at Microsoft.BizTalk.ExplorerOM.BtsCatalogExplorer.SaveChanges()
at Microsoft.BizTalk.Administration.SnapIn.Forms.Common.ExplorerPropertyPagesContainer.CommitChanges()
at Microsoft.BizTalk.SnapIn.Framework.Forms.PropertyPagesContainer.Store()
at Microsoft.BizTalk.SnapIn.Framework.Forms.SheetFramework.Store()
The filter simply checks for existence of the promoted property in the incoming message. There's only one post in MSDN forums about this exception: HRESULT 0xC0C01B22
Any idea what is causing this?
The answer is in this page: https://learn.microsoft.com/en-us/biztalk/core/promoting-properties
The note there says the following:
XSD Data Type of base64Binary, duration, ENTITES, hexBinary, IDREFS, long, NMTOKENS, and unsignedLong are not supported for promotion.
Turns out, in our case, the promoted property is of xs:long type. I changed the element type to xs:string then the issue in the question went away.
It would make sense to prevent the promotion of the elements if they are of unsupported type, in the beginning itself. But we don't get to see any alerts during the promotion phase for such elements.
Hope this helps someone.
I wrote a crawler with Scrapy.
There is a function in the pipeline where I write my data to a database. I use the logging module to log runtime logs.
I found that when my string have Chinese logging.error() will throw an exception. But the crawler keeps running!
I know this is a minor error but if there is a critical exception I will miss it if crawler keeps running.
My question is: Is there a setting that I can force Scrapy stop when there is an exception?
You can use CLOSESPIDER_ERRORCOUNT
An integer which specifies the maximum number of errors to receive
before closing the spider. If the spider generates more than that
number of errors, it will be closed with the reason
closespider_errorcount. If zero (or non set), spiders won’t be closed
by number of errors.
By default it is set to 0
CLOSESPIDER_ERRORCOUNT = 0
you can change it to 1 if you want to exit when you have the first error.
UPDATE
Read the answers of this question, you can also use:
crawler.engine.close_spider(self, 'log message')
for more information read :
Close spider extension
In the process_item function of your spider you have an instance of spider.
To solve your problem you could catch the exceptions when you insert your data, then neatly stop you spider if you catch a certain exeption like this:
def process_item(self, item, spider):
try:
#Insert your item here
except YourExceptionName:
spider.crawler.engine.close_spider(self, reason='finished')
I don't know of a setting that would close the crawler on any exception, but you have at least a couple of options:
you can raise CloseSpider exception in a spider callback, maybe when you catch that exception you mention
you can call crawler.engine.close_spider(spider, 'some reason') if you have a reference to the crawler and spider object, for example in an extension. See how the CloseSpider extension is implemented (it's not the same as the CloseSpider exception).
You could hook this with the spider_error signal for example.
This problem is perplexing me. I just can't seem to get the nameidentifier claim in my c# code after the user has authenticated. This is a .Net 4.5 web forms site. The only claim I can get is http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name
It's easy to iterate through the claims, and there is just one available. Looking at the trace clearly there is nameidentifier coming through, however it's just not accessible in my code when iterating through ClaimsPrincipal.Current.Claims or ClaimsPrincipal.Current.Identities[].Claims.
Here is the trace example:
<TraceRecord xmlns="http://schemas.microsoft.com/2009/10/IdentityModel/TraceRecord" Severity="Information">
<Description>Setting Thread.CurrentPrincipal from session token.</Description>
<AppDomain>/LM/W3SVC/2/ROOT/qat-5-131009300157520403</AppDomain>
<ClaimsPrincipalTraceRecord xmlns="http://schemas.microsoft.com/2009/06/IdentityModel/ClaimsPrincipalTraceRecord">
<ClaimsPrincipal Identity.Name="Adfs">
<ClaimsIdentity Name="Adfs" Label="" RoleClaimType="http://schemas.microsoft.com/ws/2008/06/identity/claims/role" NameClaimType="http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name">
<Claim Type="http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier" ValueType="http://www.w3.org/2001/XMLSchema#string" Value="something"/>
<Claim Type="http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name" ValueType="http://www.w3.org/2001/XMLSchema#string" Value="Adfs"/>
<Claim Type="http://schemas.microsoft.com/ws/2008/06/identity/claims/role" ValueType="http://www.w3.org/2001/XMLSchema#string" Value="Adfs.email#somewhere"/>
<Claim Type="http://schemas.microsoft.com/ws/2008/06/identity/claims/authenticationmethod" ValueType="http://www.w3.org/2001/XMLSchema#string" Value="urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport"/>
<Claim Type="http://schemas.microsoft.com/ws/2008/06/identity/claims/authenticationinstant" ValueType="http://www.w3.org/2001/XMLSchema#dateTime" Value="2016-02-26T01:53:04.289Z"/>
</ClaimsIdentity>
</ClaimsPrincipal>
</ClaimsPrincipalTraceRecord>
</TraceRecord>
My code can return the Identity and one claim for the name. But the role which contains an email address and nameidentifier are just not there. There is just one claim.
I've gone through everything. My app is just the simple single page test app. Nothing interesting, nothing configured to drop some claims or anything.
Can anyone suggest where I should look or anything that I can do to get the nameidentifier (or any other claims).
I can't understand why they are showing in the trace, but are not available in my application. I thought they would just be in one of the Claims collections available under ClaimsPrincipal.Current or in one of the ClaimsPrincipal.Current.Identities (there is just one identity matching the trace output).
Any help really appreciated I have little hair left! Thanks in advance.
I will be answering my own question. I don't have much hair left after this. The problem is possibly going to catch others out.
When using the below to access the ClaimsPrincipal, I was using the System.IdentityModel namespace. This was giving me only one claim. Although with WIF tracing enabled I could see many more claims being provided to my application.
using System.IdentityModel;
...
ClaimsPrincipal claimsPrincipal = ClaimsPrincipal.Current;
The solution was just to use the following instead:
using Microsoft.IdentityModel.Claims;
...
IClaimsPrincipal claimsPrincipal = (IClaimsPrincipal)Thread.CurrentPrincipal;
Then, suddenly all the other claims were available. On discovering this I have researched a little and found a few stack overflow articles about the differences between these two. One guy mentioned here that the Microsoft namespace is for extensions to the standard namespace that deal with Microsoft features and would explain this. Also this answer makes the most sense to me about this as well.
When starting my program for the first time since the associated table has been deleted I get this error:
An exception of type 'Microsoft.WindowsAzure.Storage.StorageException' occurred in Microsoft.WindowsAzure.Storage.dll but was not handled in user code
Additional information: The remote server returned an error: (409) Conflict.
However if I refresh the crashed page the table will successfully create.
Here is the code just in case:
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
Microsoft.WindowsAzure.CloudConfigurationManager.
GetSetting("StorageConnectionString"));
CloudTableClient tableClient = storageAccount.CreateCloudTableClient();
CloudTable table = tableClient.GetTableReference("tableTesting");
table.CreateIfNotExists();
I don't really understand how or why I'd be getting a conflict error if there's nothing there.
These errors appear elsewhere in my code as well when I'm working with blob containers, but I can't reproduce them as easily.
If you look at the status codes here: http://msdn.microsoft.com/en-us/library/azure/dd179438.aspx, you will notice that you get 409 error code in two scenarios:
Table already exists
Table is being deleted
If I understand correctly, table.CreateIfNotExists() only handles the 1st situation but not the 2nd one. Please check if that is not the case in your situation. One way to check this would be to see details of Storage Exception. Somewhere you should get the code which would match with the link I mentioned above.
Also one important thing to understand is that when you delete the table, it is actually marked for deletion and is actually deleted through a background process (much like garbage collection). If you try to create a table between these two steps, you will get the 2nd error.
I have used the code below:
Iterable<Module> modules = ImmutableSet.<Module> of(
new SshjSshClientModule());
ContextBuilder builder = ContextBuilder.newBuilder(provider).endpoint(endpoint)
.credentials(identity, credential)
.modules(modules);
System.out.printf(">> initializing %s%n", builder.getApiMetadata());
ComputeService compute = builder.buildView(ComputeServiceContext.class).getComputeService();
System.out.println(compute1.listImages());
but I am getting the following error message.........
com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected a string but was BEGIN_ARRAY at line 1 column 787
at org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory$DeserializeIntoParameterizedConstructor.read(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:181)
at org.jclouds.json.internal.NullFilteringTypeAdapterFactories$IterableTypeAdapter.readAndBuild(NullFilteringTypeAdapterFactories.java:92)
The code was working... before...
You've definitely hit a bug somewhere between the version of jclouds you're using and whatever version of whatever cloud you're using. We'll need more information to fix this. Please go through the instruction on how to Report a Bug to Apache jclouds.