Configure retention policy of all topics during creation
Trying to configure rentention.ms using spring, as I get an error of:
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.PolicyViolationException: Invalid retention.ms specified. The allowed range is [3600000..2592000000]
From what I've read the new value is -1 (infinity) so is out of that range
Following what was in
How to configure kafka topic retention policy during creation in spring-mvc? ,I added the below code but it seems to have no effect.
Any ideas/hints on how might solve this?
ApplicationConfigurationTest.java
#test
public void kafkaAdmin () {
KafkaAdmin admin = configuration.admin();
assertThat(admin, instanceOf(KafkaAdmin.class));
}
ApplicationConfiguration.java
#Bean
public KafkaAdmin admin() {
Map<String, Object> configs = new HashMap<>();
configs.put(TopicConfig.RETENTION_MS_CONFIG, "1680000");
return new KafkaAdmin(configs);
}
Found the solution by setting the value
spring.kafka.streams.topic.retention.ms: 86400000
in application.yml.
Our application uses spring mvc, hence the spring notation.
topic.retention.ms is the value that needs to be set in the streams config
86400000 is a random value just used as it is in range of [3600000..2592000000]
Related
I'm trying to set the retry.backoff.ms setting for kafka in my producer using the DefaultKafkaProducerFactory from org.springframework.kafka.core. Here's what I got:
public class KafkaProducerFactory extends DefaultKafkaProducerFactory {
public KafkaProducerFactory(Map<String, Object> config) {
super(config);
}
#Configuration
public class MyAppProducerConfig {
#Value("${myApp.delivery-timeout-ms:#{120000}}")
private int deliveryTimeoutMs;
#Value("${myApp.retry-backoff-ms:#{30000}}")
private int retryBackoffMs;
Producer<MyKey, MyValue> myAppProducer() {
Map<String, Object> config = new HashMap<>();
config.put(org.apache.kafka.clients.producer.ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, deliveryTimeoutMs);
config.put(org.apache.kafka.clients.producer.ProducerConfig.RETRY_BACKOFF_MS_CONFIG, retryBackoffMs);
final var factory = new KafkaProducerFactory<MyKey, MyValue>(config);
return factory.createProducer(); // calls DefaultKafkaProducerFactory
}
Now when I add the following to my application.yaml
myApp:
retry-backoff-ms = 50
delivery-timeout-ms = 1000
This is what I see in the logging when I start the application:
o.a.k.clients.producer.ProducerConfig : ProducerConfig values:
delivery.timeout.ms = 1000
retry.backoff.ms = 1000
so the delivery.timeout.ms was set, but the retry.backoff.ms wasn't even though I did the exact same for both.
I did find how to set application properties to default kafka producer template without setting from kafka producer config bean, but I didn't see either property listed under integrated properties.
So hopefully someone can give me some pointers.
After an intense debugging session I found the issue. DefaultKafkaProducerFactory is in a shared library between teams and I'm not super familiar with the class since it's my first time touching it.
Turns out the createProducer() call in DefaultKafkaProducerFactory calls another function that is overriden in KafkaProducerFactory which then creates an AxualProducer.
And the AxualProducerConfig always sets retry.backoff.ms to 1000ms.
We had a rogue producer setting a Kafka Header __TypeId__ to a class that was part of the producer, but not of a consumer implemented within a Spring Cloud Stream application using Kafka Streams binder. It resulted in an exception
java.lang.IllegalArgumentException: The class 'com.bad.MyClass' is not in the trusted packages: [java.util, java.lang, de.datev.pws.loon.dcp.foreignmodels.*]. If you believe this class is safe to deserialize, please provide its name. If the serialization is only done by a trusted source, you can also enable trust all (*).
How can we ensure within the consumer that this TypeId header is ignored?
Some stackoverflow answers point to spring.json.use.type.headers=false, but it seems to be an "old" property, that is no more valid.
application.yaml:
spring:
json.use.type.headers: false
application:
name: dcp-all
kafka:
bootstrap-servers: 'xxxxx.kafka.dev.dvint.de:9093'
cloud:
stream:
kafka:
streams:
binder:
required-acks: -1 # all in-sync-replicas
...
Stack trace:
at org.springframework.kafka.support.mapping.DefaultJackson2JavaTypeMapper.getClassIdType(DefaultJackson2JavaTypeMapper.java:129)
at org.springframework.kafka.support.mapping.DefaultJackson2JavaTypeMapper.toJavaType(DefaultJackson2JavaTypeMapper.java:103)
at org.springframework.kafka.support.serializer.JsonDeserializer.deserialize(JsonDeserializer.java:569)
at org.apache.kafka.streams.processor.internals.SourceNode.deserializeValue(SourceNode.java:58)
at org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:66)
at org.apache.kafka.streams.processor.internals.RecordQueue.updateHead(RecordQueue.java:176)
at org.apache.kafka.streams.processor.internals.RecordQueue.addRawRecords(RecordQueue.java:112)
at org.apache.kafka.streams.processor.internals.PartitionGroup.addRawRecords(PartitionGroup.java:304)
at org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:960)
at org.apache.kafka.streams.processor.internals.TaskManager.addRecordsToTasks(TaskManager.java:1068)
at org.apache.kafka.streams.processor.internals.StreamThread.pollPhase(StreamThread.java:962)
at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:751)
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:604)
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:576)
Here is a unit test
#Test
void consumeWorksEvenWithBadTypesHeader() throws JsonProcessingException, InterruptedException {
Map<String, Object> producerProps = KafkaTestUtils.producerProps(embeddedKafka);
producerProps.put("key.serializer", StringSerializer.class.getName());
DefaultKafkaProducerFactory<String, String> pf = new DefaultKafkaProducerFactory<>(producerProps);
List<Header> headers = Arrays.asList(new RecordHeader("__TypeId__", "com.bad.MyClass".getBytes()));
ProducerRecord<String,String> p = new ProducerRecord(TOPIC1, 0, "any-key",
"{ ... some valid JSON ...}", headers);
try {
KafkaTemplate<String, String> template = new KafkaTemplate<>(pf, true);
template.send(p);
ConsumerRecord<String, String> consumerRecord = KafkaTestUtils.getSingleRecord(consumer, TOPIC2, DEFAULT_CONSUMER_POLL_TIME);
// Assertions ...
} finally {
pf.destroy();
}
}
You have 2 options:
On the producer side set the property to omit adding the type info headers
On the consumer side, set the property to not use the type info headers
https://docs.spring.io/spring-kafka/docs/current/reference/html/#json-serde
It is not an "old" property.
/**
* Kafka config property for using type headers (default true).
* #since 2.2.3
*/
public static final String USE_TYPE_INFO_HEADERS = "spring.json.use.type.headers";
It needs to be set in the consumer properties.
I added https://micrometer.io to our staging server in google cloud. The metric does not show up in "Cloud Run Revision" resource types. It is only visible if I select "Global" as seen here...
The instructions were very simple and very clear (MUCH UNLIKE opencensus which has a way overdesigned api). In fact, unlike opencensus, it worked out of the box except for it is not recording into "Cloud Run Revision".
I can't even choose the service_name in the filter so once I deploy to production, the metric will be recording BOTH prod and staging which is not what we want.
How do I debug micrometer further
If anyone knows offhand as well what the issue might be, that would be great as well? (though I don't mind learning micrometer and debugging it a bit more).
For now the only available monitored-resource types in your custom metrics are:
aws_ec2_instance: Amazon EC2 instance.
dataflow_job: Dataflow job.
gce_instance: Compute Engine instance.
gke_container: GKE container instance.
generic_node: User-specified computing node.
generic_task: User-defined task.
global: Use this resource when no other resource type is suitable. For most use cases, generic_node or generic_task are better choices than global.
k8s_cluster: Kubernetes cluster.
k8s_container: Kubernetes container.
k8s_node: Kubernetes node.
k8s_pod: Kubernetes pod.
So, global is the correct monitored-resource type in this case, since there is not a Cloud Run monitored-resource type yet.
To identify better the metrics, you can create metric descriptors, either Auto-creation or manually
For completeness, I have it recording all the JVM stats now but have a new post on aggregation in google's website here that seems to be a new issue...
Google Cloud Metrics and MicroMeter JVM reporting (is this a Micrometer bug or?)
My code that did the trick was (and using revisionName is CRITICALL for not getting errors!!!)
String projectId = MetadataConfig.getProjectId();
String service = System.getenv("K_SERVICE");
String revisionName = System.getenv("K_REVISION");
String config = System.getenv("K_CONFIGURATION");
String zone = MetadataConfig.getZone();
Map<String, String> map = new HashMap<>();
map.put("namespace", service);
map.put("job", "nothing");
map.put("task_id", revisionName);
map.put("location", zone);
log.info("project="+projectId+" svc="+service+" r="+revisionName+" config="+config+" zone="+zone);
StackdriverConfig stackdriverConfig = new OurGoogleConfig(projectId, map);
//figure out how to put in template better
MeterRegistry googleRegistry = StackdriverMeterRegistry.builder(stackdriverConfig).build();
Metrics.addRegistry(googleRegistry);
//This is what would be used in Development Server
//Metrics.addRegistry(new SimpleMeterRegistry());
//How to expose on #backend perhaps at /#metrics
CompositeMeterRegistry registry = Metrics.globalRegistry;
new ClassLoaderMetrics().bindTo(registry);
new JvmMemoryMetrics().bindTo(registry);
new JvmGcMetrics().bindTo(registry);
new ProcessorMetrics().bindTo(registry);
new JvmThreadMetrics().bindTo(registry);
and then the config is simple...
private static class OurGoogleConfig implements StackdriverConfig {
private String projectId;
private Map<String, String> resourceLabels;
public OurGoogleConfig(String projectId, Map<String, String> resourceLabels) {
this.projectId = projectId;
this.resourceLabels = resourceLabels;
}
#Override
public String projectId() {
return projectId;
}
#Override
public String get(String key) {
return null;
}
#Override
public String resourceType() {
return "generic_task";
}
#Override
public Map<String, String> resourceLabels() {
//they call this EVERY time, so save on memory by only passing the same
//map every time instead of re-creating it...
return resourceLabels;
}
};
The pact-jvm-provider-spring states that for junit5 provider test, it is not required to use the spring library.
However, #PactBroker annotation depends on the system properties. Is there a way to get this working for application properties via the Spring Property Resolver. I tried to create something similar to SpringEnvironmentResolver.kt and used it in the context setup. But that did not work.
#Provider("api-provider-app")
#PactBroker
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
#ActiveProfiles("test")
public class PactVerificationTest {
#LocalServerPort
private int port;
#Autowired
private Environment environment;
#TestTemplate
#ExtendWith(PactVerificationInvocationContextProvider.class)
void testTemplate(Pact pact, Interaction interaction, HttpRequest request,
PactVerificationContext context) {
context.setTarget(new HttpTestTarget("localhost", port));
context.setValueResolver(new SpringResolver(environment));
context.verifyInteraction();
}
}
I get the following error
Invalid pact broker host specified ('${pactbroker.host:}'). Please provide a valid host or specify the system property 'pactbroker.host'.
Update
After some more searching found out that the setTarget was not working and that needs to be moved to #BeforeEach method.
#BeforeEach
void setContext(PactVerificationContext context) {
context.setValueResolver(new SpringResolver(environment));
context.setTarget(new HttpTestTarget("localhost", port));
}
The following snippet helped it work with #PactFolder annotation. But the #PactBroker with properties is still not working
There is a new module added to Pact-JVM that extends the JUnit5 support to allow values to be configured in the Spring Context. See https://github.com/DiUS/pact-jvm/tree/master/provider/pact-jvm-provider-junit5-spring. It will be released with the next version of Pact-JVM, which will be 4.0.7.
I'm writing a kafka consumer using 'org.springframework.kafka.annotation.KafkaListener' (#KafkaListener) annotation. This annotation is expecting the topic to be already at the time of subscribing and trying to create the topic if the topic is not present.
In my case, i don't want the consumer to create a topic with default configuration but it should create a topic with custom configurations (like the no of partitions, clean up policy etc). Is there any option for this in spring-kafka?
See the documentation configuring topics.
If you define a KafkaAdmin bean in your application context, it can automatically add topics to the broker. To do so, you can add a NewTopic #Bean for each topic to the application context. The following example shows how to do so:
#Bean
public KafkaAdmin admin() {
Map<String, Object> configs = new HashMap<>();
configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,
StringUtils.arrayToCommaDelimitedString(embeddedKafka().getBrokerAddresses()));
return new KafkaAdmin(configs);
}
#Bean
public NewTopic topic1() {
return new NewTopic("thing1", 10, (short) 2);
}
#Bean
public NewTopic topic2() {
return new NewTopic("thing2", 10, (short) 2);
}
By default, if the broker is not available, a message is logged, but the context continues to load. You can programmatically invoke the admin’s initialize() method to try again later. If you wish this condition to be considered fatal, set the admin’s fatalIfBrokerNotAvailable property to true. The context then fails to initialize.
If the broker supports it (1.0.0 or higher), the admin increases the number of partitions if it is found that an existing topic has fewer partitions than the NewTopic.numPartitions.
If you are using Spring Boot, you don't need an admin bean because boot will automatically configure one for you.