How do we set acks=all for KAFKA HTTP Topic? I tried sending in the below JSON as "acks":"all" But it throws with an unrecognized property. I tried setting it in the header as well. But in Header when I set to any value like 0,1, all and abcd. It is accepting all the values. So I can't rely on the header. But from Java code, we can set this property as ProducerConfig.ACKS_CONFIG as key-value pair
{
"records": [
{
"value": { "name": "Firstname, lastname" },
"key":"123e4567-e89b-42d3-a456-5566424415123591"
}
]
}
Any Suggestions are helpful.
In case of Kafka REST Proxy, the producer instances are shared between clients i.e. the client(s) will connect to REST proxy instance(s) and those REST proxy instance(s) will in turn connect to the Kafka cluster (brokers).
The json that you have supplied is only going to contain data records.
The REST proxy layer will have the global settings for producers that you're looking for and clients will end up sharing these. So, if you've got access to the REST proxy instances then you could modify these parameters there directly.
Related
we're trying to build an event-based integration between an Apache Camel based system with produces messages in an Azure Service Bus topic and an .NET based consumer of these messages.
The producer used the AMQP interface of the Service Bus, while the .NET based consumer uses the current API from Microsoft in namespace Azure.Messaging.ServiceBus.
When we try to access the body in a received message as follows:
private async Task ProcessMessagesAsync(ProcessMessageEventArgs args)
{
try {
message = Encoding.UTF8.GetString(args.Message.Body);
}
catch( Exception e)
{
_logger.LogError(e, "Body not decoded: Message: {#message}", e.Message);
}
_logger.LogInformation("Body Type: {#bodytype}, Content-Type: {#contenttype}, Message: {#message}, Properties: {#properties}", raw.Body.BodyType, args.Message.ContentType, message, args.Message.ApplicationProperties);
await args.CompleteMessageAsync(args.Message);
}
the following exception is raised:
Value cannot be retrieved using the Body property.Use GetRawAmqpMessage to access the underlying Amqp Message object.
System.NotSupportedException: Value cannot be retrieved using the Body property.Use GetRawAmqpMessage to access the underlying Amqp Message object.
at Azure.Messaging.ServiceBus.Amqp.AmqpMessageExtensions.GetBody(AmqpAnnotatedMessage message)
at Azure.Messaging.ServiceBus.ServiceBusReceivedMessage.get_Body()
When peeking the topic with service bus explorer the message looks strange:
#string3http://schemas.microsoft.com/2003/10/Serialization/�_{"metadata":{"version":"1.0.0","message_id":"AGHcNehoD-hK0pPJCSga9v9sXFwC","message_timestamp":"2022-01-10T13:34:32.778Z"},"data":{"source_timestamp":"2022-01-05T17:20:31.000","material":"101052"}}
When messages are sent to another topic with a .NET producer there's a plaintext JSON body in the topic, as expected.
Did anybody successfully build a solution with Azure Service Bus with components based on the two mentioned frameworks, and what did the trick so that interoperability did work? Who can a Camel AMQP producer create messages with a BodyType of Data so that the body can be decoded by the .NET Service Bus client libraries without need to use GetRawAmqpMessage?
I can't speak to what format Camel is using, but that error message indicates that the Body that you're trying to decode is not an AMQP data body, which is what the Service Bus client library uses and expects.
In order to read a body that is encoded as an AMQP value or sequence, you'll need to work with the data in AMQP format rather than by using the ServiceBusReceivedMessage convenience layer. To do so, you'll want to call GetRawAmqpMessage on the ServiceBusReceivedMessage, which will give you back an AmqpAnnotatedMessage.
The annotated message Body property will return an AmqpMessageBody instance which will allow you to query the BodyType and retrieve the data in its native format using one of the TryGetmethods on the AmqpMessageBody.
On our procuder side a SAP Cloud Integration is used, when the Message Type parapeter of the AMQP Adapter is set to Binary, according to:
https://help.sap.com/viewer/368c481cd6954bdfa5d0435479fd4eaf/Cloud/en-US/d5660c146a93483692335e9d79a8c58f.html.
This seem to correspond to Apache Camel jmsMessageType set to Bytes,
see https://camel.apache.org/components/3.14.x/amqp-component.html for details.
The decoding of the body in the ServiceBusReceivedMessage works as expected and the BodyType is set to Data. If using Text on the producer side, the BodyType will be set to Value as described which led to the problems with the decoding of the body.
I'm trying to figure out a way to disable any health-check related ILogger logging. I am aware of LogLevel filtering, but that will not work here.
As an example, I have a healthcheck that makes an outbound call to a RabbitMQ metrics API. This results in an outbound http call with every inbound call to /health. In general, I want to log all outbound calls made using the HttpClient, but that log is now full of this particular log entry:
[2021.06.15 13:57:04] info: System.Net.Http.HttpClient.Default.LogicalHandler[101] => ConnectionId:0HM9FV5PFFL5K => RequestPath:/health RequestId:0HM9FV5PFFL5K:00000001, SpanId:|6726c52-4217ec92de4df5fb., TraceId:6726c52-4217ec92de4df5fb, ParentId: => Microsoft.Extensions.Diagnostics.HealthChecks.HealthCheckLogScope => HTTP GET http://rabbitmq/api/queues/MyVHost/MyQueue?msg_rates_age=60&msg_rates_incr=60
: End processing HTTP request after 4.6355ms - OK
So, I could apply a warning filter to the HttpClient/LogicalHandler to remove those entries, but then I'd be removing all the info logs of other outbound Http requests, which I don't want.
So, basically, I need a smarter filter that can look at the scopes (or even the text in certain cases), and can filter out based on "Microsoft.Extensions.Diagnostics.HealthChecks.HealthCheckLogScope". That doesn't seem to be possible though, as the filter callback doesn't provide those details.
Does anyone have any idea how to do more specific log filtering for cases like this?
I have looked at .net core log filtering on certain requests, but extending every ILoggerProvider I use isn't possible, since some are not public classes.
You could use Serilog for logging, as it also provides great filtering, enriching, and formatting capabilities with Serilog.Expressions Nuget package.
There is even a simple example provided in the link above for filtering health checks, which fulfilled my needs to fully filter out health check logging based on the request path '/health'. Following the guide, it only required adding appsettings.json configuration, after the serilog was configured as the application logger to make it work:
{
"Serilog": {
"Using": ["Serilog.Expressions"],
"Filter": [
{
"Name": "ByExcluding",
"Args": {
"expression": "RequestPath like '/health%'"
}
}
]
}
}
To filter out those requests you can try using the Log4Net StringMatchFilter filter with the value of the health URL.
Here is the code for the configuration file if health URL is just localhost/health:
<filter type="log4net.Filter.StringMatchFilter">
<stringToMatch value="/health" />
<acceptOnMatch value="false" />
</filter>
First of all i am a newbie at jmeter. I searched a lot of documents but unfortunately i did not find to answer for my problem. I have a web api rest service with following sign. I can't send dictionary format parameter.
[HttpPost,ActionName("DummyService")]
public Dictionary<string,string> DummyService([FromBody] Dictionary<string,string> parameters)
I used Parameters section and BodyData section but parameters are always null.
How can i achieve that?
Thanx in advance.
I believe you need to pass some form of a JSON Array to your web service, in order to do it:
Switch to Body Data tab
Put your JSON payload there like:
{
"Parameters": [
{
"Key": "Key1",
"Value": "Value1"
},
{
"Key": "Key2",
"Value": "value2"
}
]
}
Most likely you will need to add a HTTP Header Manager to your test plan and configure it to send Content-Type header with the value of application/json
(optional) Consider upgrading to JMeter 3.0, looking into screenshot it seems you're sitting on an older version
See Testing SOAP/REST Web Services Using JMeter guide for more information on setting up JMeter for API testing
In Apigee, can fault handling - specifying a FaultRule and a RaiseFault policy be used to handle and provide a custom message for:
{
"fault": {
"faultstring": "Not Found",
"detail": {
"errorcode": "CLASSIFICATION_FAILURE"
}
}
}
If this can be done, should the 'Condition' for the fault rule be 'fault.name = "CLASSIFICATION_FAILURE"'? I tried this and it is not working.
CLASSIFICATION_FAILURE is a system level failure to find an API Proxy for the given URL/URI. The request will not even reach the API proxy(hence the policies) - which is the precise complaint by the system.
So you do not want to handle an error like that.
Another way to approach this case is to have a catch all API proxy with basepath /** which will be invoked when there is no specific URL match. You can generate a custom message in this proxy - this can be the message you wanted to send across in case of classification failure.
Srikanth's answer on 30/05/2014 is only partially correct. Using a basepath /** did not work for us. Instead, we had to create an api proxy with basepath = /
Inside the proxy, we defined a RaiseFault in Preflow and that was it.
I have an Apigee proxy that has two resources (/resource1 and /resource2). If tried to access /resource3. How do I return a 404 error instead of the Apigee default fault?
Apigee displays the below fault string:
{
"fault": {
"faultstring": "The Service is temporarily unavailable",
"detail": {
"errorcode": "messaging.adaptors.http.flow.ServiceUnavailable"
}
}
}
Thanks
Currently the way flows work in apigee this way - It parses through your default.xml (in proxy) and tries to match your request with one of the flow either through the path-suffix like "/resource1, /resource2" or VERB or any other condition you might have. If it does not find any matching condition, it throws the error like above.
You can add a special flow which will be kicked in if the condition matches none of the valid flows you have. You can add a raisefault policy in that flow and add a custom error response through that flow.
A better solution is to:
be sure to define something in the base path of all Proxy APIs
create an additional Proxy API called "catchall" with a base path of "/" and with just a Raise fault throwing a 404
Apigee execute Proxy APIs from longest Base Path to shortest; the catchall will run last and always throw back a 404
I just want to clarify Vinit's answer. Vinit said:
If it does not find any matching condition, it throws the error like above.
Actually, if no matching flow condition is found, the request will still be sent through to the backend. The error you mentioned:
{
"fault": {
"faultstring": "The Service is temporarily unavailable",
"detail": {
"errorcode": "messaging.adaptors.http.flow.ServiceUnavailable"
}
}
}
was returned after attempting to connect to the backend without matching a flow.
Vinit's solution to raise a fault to create the 404 is the best solution for your requirements.
In some cases, however, it is appropriate to pass all traffic through to the backend (for example, if you don't need to modify each resource at the Apigee layer, and you don't want to have to update your Apigee proxy every time you add a new API resource). Not matching any flow condition would work fine for that use case.