We are using spring-kafka-2.2.7.RELEASE to produce and consume avro messages and using schema registry for schema validation with 'FORWARD_TRANSITIVE' as the compatibility type. Now, I'm trying to use 'ErrorHandlingDeserializer2 ' from spring-kafka to handle the exception/error when a deserializer fails to deserialize a message. Now I'm trying to write a component test to test this configuration. My component test expected to have below steps.
Spin up a local kafka cluster using docker containers
Send an avro message (using KafkaTemplate) with invalid schema to re-create/simulate the deserialization exception onto a test topic
Now what's happening is, since we have schema registry in place, if i send a message with new schema (invalid schema) it's validating the schema as per the compatibility type setting we have and not letting me producer the message onto kafka by throwing an exception at the producer level itself.
Now my question is, In this scenario, how can I create/simulate the creation of deserialization exception to test my configuration. Please suggest.
Note:- I don't want to disable/stop schema registry because that wouldn't simulate our prod setup.
Related
I have a project using Spring cloud stream with Kafka Streams binder. For the output of a stream, I am using Avro, with the Serde provided by Confluent(io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde).
I am able to use it with the Confluent Schema Registry. Serialization and Deserialization takes place correctly.
However, I wanted to see if we can use the Spring Cloud Schema Registry Server instead of the Confluent one. I configured a standalone Schema Registry server and set the schema registry in my project to it (changed the schemaRegistryClient.endpoint and schema.registry.url properties).
When I tried it out, it seems Spring Cloud is able to work with the standalone server. It registers the schema available in the resources folder as a .avsc file. However, when I send a message, it seems the Confluent serializer continues to approach it as a Confluent Schema Registry (which has different REST endpoints from Spring Schema Registry). As a result, it gets a 405 response code.
We get the following exception(partial stack-trace)
org.apache.kafka.common.errors.SerializationException: Error registering Avro schema: <my-avro-schema>
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
at [Source: (sun.net.www.protocol.http.HttpURLConnection$HttpInputStream); line: 1, column: 2]; error code: 50005
at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:230)
It seems to me that there are two possibilities:
Spring Schema Registry Server can work only with the content-type provided by Spring (specified as content-type: application/*+avro) and not with the native Serde provided by Confluent, or
There is an issue with the project configuration.
Can someone help me figure out which one is it? If it is the second one, can someone point out what is wrong?
Each schema registry provider requires a proprietary SerDe library. For example, if you would like to integrate AWS Glue Schema Registry with Kafka, then you would need Amazon's SerDe stuff. Hence, the Confluent's SerDe library expects Confluent's Schema Registry at the address specified in the schema.registry.url property.
Is there a way to programmatically determine from a DocumentClientException where StatusCode == HttpStatusCode.NotFound whether it was the document, the collection, or the database that was not found?
I'm trying to figure out whether I can implement on-demand collection provisioning and only call DocumentClient.CreateDocumentCollectionIfNotExistsAsync when I need to. I'm trying to avoid calling it before making every request (presumably this adds an extra network roundtrip to every request). Likewise, I'm trying to avoid calling it on error recovery when I know it won't help.
From experimentation with the local emulator, the only field I see varying in these three cases is DocumentClientException.Error.Message, and only when the database cannot be found. I generally try to avoid exception dispatching based on human-readable messages.
Wrong database name:
StatusCode: HttpStatusCode.NotFound
Error.Message: {\"Errors\":[\"Owner resource does not exist\"]}...
Correct database name, wrong collection name:
StatusCode: HttpStatusCode.NotFound
Error.Message: {\"Errors\":[\"Resource Not Found\"]}...
Correct database name, correct collection name, incorrect document ID:
StatusCode: HttpStatusCode.NotFound
Error.Message: {\"Errors\":[\"Resource Not Found\"]}...
I'm planning to use a database with its own offer. Since collections inside a database with its own offer are cheap, I'm trying to see whether I can segregate each tenant in my multi-tenant application into its own collection. Each tenant ends up having a different indexing and default TTL policy. The set of collections is not fixed and changes dynamically during runtime as new tenants sign up. I cannot predict when I will need to add a new collection. There's no new tenant notification: I just get a request that I need to handle by creating a document in a possibly non-existent collection. There's a process to garbage collect unused collections.
I'm using the NuGet package Microsoft.Azure.DocumentDB.Core Version 1.9.1 in a .NET Core 2.1 app targeting a SQL API Cosmos DB instance.
If you look at the Message property in detail, you should see following strings that informs whether 404 Not Found response was generated due to Document vs Collection.
ResourceType: Document
ResourceType: Collection
It's not ideal but you can try to regex this information out of error message.
Written the Integration test case for testing Corda Flows through API class as:
val api=ProjectApi(mockNode1.rpcOps)
val message = "some input message"
val resp: Response=api.publishSSI(message)
assertEquals(resp.status, 201,"Failed to publish SSI")
But getting CURRENT_RPC_CONTEXT.get() must not be null exception while starting corda flow inside api.publishSSI() method.
what could be the cause?
You need to use the RPC client API to connect to the node. You can find more details here: https://docs.corda.net/tutorial-clientrpc-api.html
You can't test the API by manually constructing it in this way. When the API is set up, some additional context is passed in, which is lacking here.
When i try to make a remote ejb jndi look up, IBM message Broker throws ClassCastexception for the factory object.
But the same code works fine for a normal local java application and junit.Why this problem occurs when called only from IBM WMB
Context context = new InitialContext(ejbJndiProperties);
Object factoryObj = context.lookup("SampleBeanTAFJ/remote");
return (SampleBeanRemote) factoryObj;
This is often called by loading parts of the interface in a different classloader to the implementation classes.
I would use the env var:
IBM_JAVA_OPTIONS=-Dibm.cl.verbose=*
Then restart the broker, this will dump classloading trace to stdout / console.txt which might give you some clues.
What are the exact classes involved in the error and what jars are they stored in? Deployed to the EG or referenced via a SHARED-CLASSES? The exact details determine which classloaders should be in use here.
Background: I'm calling a Web Service written in ASP.NET that queries an Oracle database. I know the Web Service itself works, because I've used it before other applications. So I have a web application in Visual Studio that I've been switching back and forth to point from a 'DEV' web service to a production configured version of the same web service for testing. Pointing to the 'DEV' configured web service is no problem, but calling the production version I always get an exception calling the service:
SoapException was unhandled by user code
Server was unable to process request. ---> could not execute query
[ SELECT this_.FIELD1 as FIELD1_18_0_, this_.FIELD2 as FIELD12_18_0_ FROM ABC.TABLE_A this_ WHERE this_.FIELD1 like :p0 ORDER BY this_.FIELD1 asc ]
Positional parameters: #0>00073%
[SQL: SELECT this_.FIELD1 as FIELD1_18_0_, this_.FIELD2 as FIELD12_18_0_ FROM ABC.TABLE_A this_ WHERE this_.FIELD1 like :p0 ORDER BY this_.FIELD1] ---> ORA-12571: TNS:packet writer failure
I ran the SQL queries against the appropriate database (cut and pasted straight out of the exception message) and the query came back with the expected data. I've tried updating and re-adding the Web Service reference both as a "Service Reference" (.NET 3.0+ way) and as a "Web Reference" (Older .NET way), and both give the same error.
Question: So, what does a "ORA-12571: TNS:packet writer failure" error mean in the context of a Web Service? Looking up the Oracle Error number gives some very vague possible causes such as "loose cable connection" or "IP address conflict". I'm fairly certain it's neither of these, since a different application is currently successfully using that Web Service. Possibly some kind of configuration error, or maybe something more subtle? Anyone else seen this vexing Oracle error number being attributed to something web-service related?
Your call is going from the ws client to the ws server to the oracle database.
Your error is an ORA error, which is generated by the database. So your problem is probably between the ws server and the database.
When you ran "the SQL queries against the appropriate database", did you do it from the web server? If not could you try that. Make sure that you are using the same connection configuration.
EDIT
As per the comment below, the real problem was a driver mismatch.
I would suggest re-examining your assumptions more carefully, as this is clearly an error in the web-service dialogue with the db and should be completely independent of the w/s caller.
If the w/s call is generating this specific exception, it should be doing so for all other invocations, so your 'other application' that's using the web service successfully is simply not executing the same code or there are outside factors at play.
Either way, it's unrelated to how the service is registered or invoked.