I have a compacted topic and I try to send null value (tombstone) with kafkaTemplate, but getting an exception
Unsupported Avro type. Supported types are null, Boolean, Integer, Long, Float, Double, String, byte[] and IndexedRecord
Here is my sending call
kafkaTemplate.send(topic, GenericMessage<KafkaNull>(KafkaNull.INSTANCE))
and part of kafkaTemplate configuration
#Bean
fun kafkaTemplate(producerFactory: ProducerFactory<String, Any>) =
KafkaTemplate(producerFactory)
According to spring docs I have to use KafkaNull as a null-value.
How to send it without any exceptions ?
The problem was with producer factory configuration.
There was
ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG to KafkaAvroSerializer::class.java
and I added new producer factory with
ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG to StringSerializer::class.java
so there is tombstone sending code
stringKafkaTemplate.send(internalTopic, key, null)
Related
I am handling deserialisation error using the ErrorHandlingDeserialiser sent on my DefaultKafkaConsumerFactory.
I have code a custom
try (ErrorHandlingDeserializer<MyEvent> errorHandlingDeserializer = new ErrorHandlingDeserializer<>(theRealDeserialiser)) {
errorHandlingDeserializer.setFailedDeserializationFunction(myCustomFunction::apply);
return new DefaultKafkaConsumerFactory<>(getConsumerProperties(), consumerKeyDeserializer, errorHandlingDeserializer);
}
My custom function does some processing and publishes to a poison pill topic and returns null.
When a deserialisation error occurs, I would like to log the topic, partition and offset. The only way I can think of doing this is to stop returning null in the function and return a new sub type of MyEvent. My KafkaListener could then interrogate the new sub type.
I have a #KafkaListener component, which listens for the ConsumerRecord as follows:
#KafkaListner(....)
public void onMessage(ConsumerRecord<String, MyEvent> record) {
...
...
// if record.value instance of MyNewSubType
// I have access to the topic, partition and offset here, so I could log it here
// I'd have to check that the instance of MyEvent is actually my sub type representing a failed record.
}
Is this the way to do it? I know null has special meaning Kafka.
The downside of this sub type approach is, I'd have to create a subtype every type using the ErrorHandlingDeserialiser.
Don't use a function; instead, the thrown DeserializationException is passed directly the container's ErrorHandler.
The SeekToCurrentErrorHandler considers these exceptions to be fatal and won't retry them, it passes the record to the recoverer.
There is a provided DeadLetterPublishingRecoverer which sends the record.
See https://docs.spring.io/spring-kafka/docs/current/reference/html/#annotation-error-handling
and
https://docs.spring.io/spring-kafka/docs/current/reference/html/#dead-letters
I had a problem with DES encryption on Java Card: I encrypt the data before sending it and then send it as a response to requests coming from some library.
If in one applet SELECT sends requests for information, then through 28-30 encryptions on the card the session falls with the error 6F00. After that, all encryption calls return 6F00. The problem disappears if you do a second selection of the applet.
All data before encryption, I make a multiple of 8, so the error due to the length of the message I eliminated immediately. There may be a memory problem, but I call JCSystem.requestObjectDeletion(); after each sending of data.
Below is the initialization and encryption function implemented in my Applet.
public static void Init()
rw_des_key = (DESKey) KeyBuilder.buildKey(KeyBuilder.TYPE_DES, KeyBuilder.LENGTH_DES3_3KEY, false);
rw_cipherDes = Cipher.getInstance(Cipher.ALG_DES_CBC_NOPAD, false);
rw_des_key.setKey(rwdeskey, (short) 0);
}
public static short RWEncrypt(byte[] msg, short pos, short len, byte[] encMsg, short encPos) throws ArithmeticException, ArrayIndexOutOfBoundsException, ArrayStoreException, ClassCastException, IndexOutOfBoundsException, NegativeArraySizeException, NullPointerException, RuntimeException, SecurityException {
rw_cipherDes.init(rw_des_key, Cipher.MODE_ENCRYPT);
return rw_cipherDes.doFinal(msg, (short) pos, len, encMsg, (short) encPos);}
If someone can tell what might be the case, then I will be very grateful!
This problem occurs when external access (externalAccess) flag is false in the object but the corresponding object is called by a shareable interface.
For example (from your code): -
rw_cipherDes = Cipher.getInstance(Cipher.ALG_DES_CBC_NOPAD, false);
Here the external access flag is False i.e. you are instructing the Cipher API not to provide the access of cipher object to any external application (which is for the case of calling this API by shareable interface).
So, the following line will generate a security exception (hence SW 6F00) when called from across applications by shareable interface.
return rw_cipherDes.doFinal(msg, (short) pos, len, encMsg, (short)
encPos);}
Obviously,when you call the same API from its own application, then it will work fine.
Solution is allowing externalAccess (true) for the object. This would make it accessible via shareable interface.
rw_cipherDes = Cipher.getInstance(Cipher.ALG_DES_CBC_NOPAD, true);
I Fix my broblem by changing externalAccess flag from False to True in Cipher.getInstance function:
Cipher.getInstance(Cipher.ALG_DES_CBC_NOPAD, true);
It's resolve my problem.
I have the following endpoint
#RequestMapping("missedcall")
fun missedCall(#RequestParam("v") encryptedValue : String, model:
ModelMap): String {
//decrypt encryptedValue here
}
When I execute this endpoint with "http://myurl.com/missedcall?v=this+is+my+encrypted+string", encryptedValue initialized as "this is my encrypted string". I actually want the pluses as they are part of the encryption and I can't decrypt the string without them.
The work around would be URL encode the string back to restore pluses and other special characters, BUT is there a cleaner way? Maybe disabling URL decoding for this particular endpoint?
Note: I can't pass this param in body, it has to be part of the query string. Also this is written in Kotlin, but I am 100% sure Java has similar issues, so don't feel discouraged by Kotlin :).
Any web framework will decrypt the query path for you, as that's expected behavior. If that's not what you want, you will have to define a method argument of type HttpServletRequest and parse the query yourself using HttpServletRequest.getQueryString().
Thanks to #SeanPatrickFloyd, implemented it as
#RequestMapping("missedcall")
fun missedCall(request: HttpServletRequest, model: ModelMap): String {
val encodedValue = request.queryString.split("=")[1]
//decrypt encryptedValue here
}
I might be confused about something, but when I store a custom object from the Java Riak client and then try to read that object using the Python Riak client, I end up with a raw json string instead of a dict.
However, if I store a the object in python, I am able to output a python dictionary when fetching that object.
I could simply use a json library on the python side to resolve this, but the very fact that I am experiencing this discrepancy makes me think that I am doing something wrong.
On the Java side, this is my object:
class DocObject
{
public String status; // FEEDING | PERSISTED | FAILED | DELETING
public List<String> messages = new ArrayList<String>();
}
class PdfObject extends DocObject
{
public String url;
public String base_url;
}
This is how I am storing that object in Riak:
public void feeding(IDocument doc) throws RiakRetryFailedException {
PdfObject pdfObject = new PdfObject();
pdfObject.url = doc.getElement("url").getValue().toString();
pdfObject.base_url = doc.getElement("base_url").getValue().toString();
pdfObject.status = "FEEDING";
String key = hash(pdfObject.url);
pdfBucket.store(key, pdfObject).execute();
}
And this is what I am doing in Python to fetch the data:
# Connect to Riak.
client = riak.RiakClient()
# Choose the bucket to store data in.
bucket = client.bucket('pdfBucket')
doc = bucket.get('7909aa2f84c9e0fded7d1c7bb2526f54')
doc_data = doc.get_data()
print type(doc_data)
The result of the above python is:
<type 'str'>
I am expecting that to be <type 'dict'>, just like how the example here works:
http://basho.github.com/riak-python-client/tutorial.html#getting-single-values-out
I am perplexed as to why when the object is stored from Java it is stored as a JSON string and not as an object.
I would appreciate if anybody could point out an issue with my approach that might be causing this discrepancy.
Thanks!
It would appear you've found a bug in our Python client with the HTTP protocol/transport.
Both the version you're using and the current one in master are not decoding JSON properly. Myself and another dev looked into this this morning and it appears to stem from an issue with charset parameter being returned from Riak with the content-type as Christian noted in his comment ("application/json; charset=UTF-8")
We've opened an issue on github (https://github.com/basho/riak-python-client/issues/227) and will get this corrected.
In the mean time the only suggestion I have is to decode the returned JSON string yourself, or using the 1.5.2 client (latest stable from pypy) and the Protocol Buffers transport:
client = riak.RiakClient(port=8087, transport_class=riak.RiakPbcTransport)
it will return the decoded JSON as a dict as you're expecting.
I have a WCF data services' service operation such as
[WebGet(UriTemplate = "GetData")]
public IQueryable<string> GetData(int code, int deviceid, int groupId, DateTime dateTimeFrom, DateTime dateTimeTo)
{ ...
}
I am calling this service operation by sending a HTTP request to the server in this format:
http://localhost:6000/GetData?code=0&deviceId=1&groupId=0L&dateTimeFrom=datetime'2013-01-31T15:36:50.904'&dateTimeTo=datetime'2012-02-01T15:36:50.904'
and it is working like a charm.
Now I want to pass a NULL value as one of the parameters. How do I do that? Do I need to use nullable types such as "Int?" in the service operation declaration? How do I encode the NULL value in the URI?
Yes - you need to declare the parameter as nullable. Then you can simply omit the parameter (if it's nullable it will be treated as null then).