I am trying to use Reactive Kafka Receiver in order to process Kafka messages. I expected that consumer will give me new message only in case the last one I polled has been acknowledged. However, even if I do not run ack I get new records. Is it a desired behaviour or am I missing some configs?
void consumeMessages() {
kafkaConsumer(() -> true).subscribe();
}
Flux<Void> kafkaConsumer(BooleanSupplier repeatCondition) {
return reactiveConsumer
.receive()
.doOnError(error -> log.warn("Received error when get kafka message", error))
.doOnNext(receivedRecord -> log.debug("Received event {}", receivedRecord.value()))
.flatMap(this::handleReceivedRecord)
.onErrorResume(error -> Flux.empty())
.repeat(repeatCondition);
}
Flux<Void> handleReceivedRecord(ReceiverRecord<String, CustomEvent> receivedRecord) {
return Mono.just(receivedRecord)
.flatMapMany(
record ->
handleCustomEvent(record.value())
.doOnComplete(() -> {
log.debug("Committing offset for record: {}", record);
//record.receiverOffset().acknowledge();
})
.onErrorMap(error -> new ReceiverRecordException(record, error)))
.doOnError(error -> log.error("Error when processing CustomEvent", error))
.retryWhen(
Retry.backoff(consumerProperties.retries(), consumerProperties.minBackOff())
.transientErrors(true))
.onErrorResume(
error -> {
ReceiverRecordException ex = (ReceiverRecordException) error.getCause();
log.warn("Retries exhausted for " + ex.getRecord().value());
ex.getRecord().receiverOffset().acknowledge();
return Mono.empty();
});
}
Related
I have a custom AX service operation that can take 5+ minutes to complete and I'm trying to figure out how to abort it from my .NET application, but aborting the client doesn't seem to do anything?
The problem is if I call the operation and the service times out, the AX operation continues on until completion, so I lose visibility to the results (success/failure). An example being a long-running posting operation where I don't know if it posted successfully or not.
I've created a simple demo app where I can't seem to get the operation to abort. In the below code I just create a transaction (ttsbegin/ttscommit), insert into a table at start, sleep, insert into table at end.
Sample AX X++ Service Code:
[SysEntryPointAttribute(true)]
public str callAXTimeDelay(int _sleepSeconds)
{
Table1 table1;
ttsBegin;
table1.clear();
table1.SleepData = strFmt("STARTED: %1", DateTimeUtil::utcNow());
table1.insert();
ttsCommit;
sleep(_sleepSeconds * 1000);
ttsBegin;
table1.clear();
table1.SleepData = strFmt("COMPLETED: %1", DateTimeUtil::utcNow());
table1.insert();
ttsCommit;
return strFmt("COMPLETED: %1", DateTimeUtil::utcNow());
}
Then when I call it from .NET, the abort doesn't seem to work? Start/Complete records are still inserted into table1 even though the abort is called before the 15 seconds have completed?
Sample .NET code:
internal class Program
{
static void Main(string[] args)
{
new Program().Run();
Console.WriteLine("Ended, press any key to exit...");
Console.ReadKey();
}
public void Run()
{
AXServicesClient axClient15Sec = new AXServicesClient();
AXServicesClient axClient5sec = new AXServicesClient();
var job15sec = DoLongRunningCall(axClient15Sec, 15);
var job5sec = DoLongRunningCall(axClient5sec, 5);
try
{
var result = Task.Run(() => Task.WhenAny(job15sec, job5sec)).GetAwaiter().GetResult();
if (result == job15sec)
{
Console.WriteLine("job15sec finished first, aborting job5sec");
axClient5sec.Abort();
}
else if (result == job5sec)
{
// This code gets executed because the 5 second job completed and
// it tries to cancel the 15-sec job, but the table ends up with data!
Console.WriteLine("job5sec finished first, aborting job15sec");
axClient15Sec.Abort();
}
}
catch (Exception e)
{
Console.WriteLine("Exception: " + e.Message);
axClient15Sec.Abort();
axClient5sec.Abort();
}
axClient15Sec.Close();
axClient5sec.Close();
}
public async Task<string> DoLongRunningCall(AXServicesClient client, int seconds)
{
var result = await client.callAXTimeDelay(new CallContext
{
Company = "ABCD",
Language = "en-us"
}, seconds);
return result.response;
}
}
During Corda testing with the DriverDSL you can wait for state updates.
e.g.
expectEvents(aliceVaultUpdates, true, () ->
expect(cashVaultUpdateClass, update -> true, update -> {
System.out.println("Alice got vault update of " + update);
Amount<Issued<Currency>> amount = update.getProduced().iterator().next().getState().getData().getAmount();
assertEquals(DOLLARS(1000), Structures.withoutIssuer(amount));
return null;
})
);
How do you add a timeout so the test doesn't wait forever when the expected update doesn't happen?
I think you could just use Junit to do this?
here's an example.
#Test(timeout = 500)
public void testInfiniteTametakingLoop() throws InterruptedException
{
while (true)
{
Thread.currentThread().sleep(1000);
}
}
source: https://howtodoinjava.com/junit/how-to-force-timeout-in-jnuit-testcase-execution/
Problem
I have implemented a function that return an array of missing indexes from packets sent. If 100 packets are sent, then the server will have a vector of indexes with 0 (missing), and 1 (not missing). I don't want to trigger this every time, only when there is a slight delay where no packet is received. I want to change my synchronous function to an asynchronous debouncing function
My attempt to solve debouncing issue
I am looking for a solution to implement a timer (like 300ms) that will have its value constantly overwritten by different threads. Once its value is no longer overwritten, it should trigger a block of code or function. I am using Tokio.
This is pseudo code of what I want to achieve:
// thanks https://stackoverflow.com/questions/26593387/how-can-i-get-the-current-time-in-milliseconds
fn get_epoch() -> u128 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_millis()
}
impl Server {
async fn run(self) -> Result<(), io::Error> {
let Server {
socket,
mut buf,
mut to_send,
} = self;
let mut timer_delay = get_epoch();
loop {
if let Some((size, peer)) = to_send {
timer_delay = get_epoch(); // "reset" the to a closer value
}
futures::join!(
/* execute a block of code if true*/
if get_epoch() - timer_delay > 300,
/* else (default case):*/
to_send = Some(socket.recv_from(&mut buf)
);
}
}
}
I based my project on the following example from Tokio:
impl Server {
async fn run(self) -> Result<(), io::Error> {
let Server {
socket,
mut buf,
mut to_send,
} = self;
loop {
// First we check to see if there's a message we need to echo back.
// If so then we try to send it back to the original source, waiting
// until it's writable and we're able to do so.
if let Some((size, peer)) = to_send {
let amt = socket.send_to(&buf[..size], &peer).await?;
println!("Echoed {}/{} bytes to {}", amt, size, peer);
}
// If we're here then `to_send` is `None`, so we take a look for the
// next message we're going to echo back.
to_send = Some(socket.recv_from(&mut buf).await?);
}
}
}
Spawn another Tokio task for debouncing that will listen to a channel. You can tell when the channel hasn't received anything in a while by using a timeout. When the timeout occurs, that's the signal that you should perform your infrequent action. Don't forget to perform that action when the channel closes as well:
use std::time::Duration;
use tokio::{sync::mpsc, task, time}; // 1.3.0
#[tokio::main]
async fn main() {
let (debounce_tx, mut debounce_rx) = mpsc::channel(10);
let (network_tx, mut network_rx) = mpsc::channel(10);
// Listen for events
let debouncer = task::spawn(async move {
let duration = Duration::from_millis(10);
loop {
match time::timeout(duration, debounce_rx.recv()).await {
Ok(Some(())) => {
eprintln!("Network activity")
}
Ok(None) => {
eprintln!("Debounce finished");
break;
}
Err(_) => {
eprintln!("{:?} since network activity", duration)
}
}
}
});
// Listen for network activity
let server = task::spawn({
let debounce_tx = debounce_tx.clone();
async move {
while let Some(packet) = network_rx.recv().await {
// Received a packet
debounce_tx
.send(())
.await
.expect("Unable to talk to debounce");
eprintln!("Received a packet: {:?}", packet);
}
}
});
// Prevent deadlocks
drop(debounce_tx);
// Drive the network input
network_tx.send(1).await.expect("Unable to talk to network");
network_tx.send(2).await.expect("Unable to talk to network");
network_tx.send(3).await.expect("Unable to talk to network");
time::sleep(Duration::from_millis(20)).await;
network_tx.send(4).await.expect("Unable to talk to network");
network_tx.send(5).await.expect("Unable to talk to network");
network_tx.send(6).await.expect("Unable to talk to network");
time::sleep(Duration::from_millis(20)).await;
// Close the network
drop(network_tx);
// Wait for everything to finish
server.await.expect("Server panicked");
debouncer.await.expect("Debouncer panicked");
}
Received a packet: 1
Received a packet: 2
Received a packet: 3
Network activity
Network activity
Network activity
10ms since network activity
10ms since network activity
Received a packet: 4
Received a packet: 5
Received a packet: 6
Network activity
Network activity
Network activity
10ms since network activity
Debounce finished
In Corda, suppose I am running a flow that creates a transaction. I have signed the transaction, but now the flow is suspended waiting for the counterparty to sign.
Is there any way for me to see a list of transactions that are pending in this way?
As of Corda 3, you cannot see the contents of these transactions.
However, you can use flow progress-tracker steps to find out where each flow is in its lifecyle. For example, you could count the number of flows that are paused at some user-defined Transaction is pending. progress-tracker step as follows:
class Client {
val proxy: CordaRPCOps
init {
val nodeAddress = NetworkHostAndPort.parse("localhost:10006")
val client = CordaRPCClient(nodeAddress)
proxy = client.start("user1", "test").proxy
}
fun currentNumberOfPendingTxs(): Int {
val stateMachineInfos = proxy.stateMachinesSnapshot()
val stateMachinesPendingTxs = stateMachineInfos.filter { info ->
val progressTracker = info.progressTrackerStepAndUpdates
if (progressTracker == null) {
false
} else {
progressTracker.snapshot == "Transaction is pending."
}
}
return stateMachinesPendingTxs.size
}
}
I am using the library RxAndroidBle in order to scan devices and then connect to one specific device and read 4 GATT characteristics.
I can read one characteristic (Battery Level) ith this code :
scanSubscription = rxBleClient.scanBleDevices(
new ScanSettings.Builder()
.build()
)
.observeOn(AndroidSchedulers.mainThread())
.doOnNext(
scanResult -> {
if(scanResult.getBleDevice().getName() != null){
if(scanResult.getBleDevice().getName().equals("NODE 1")){
Log.e("BLE SCAN", "SUCCESS");
Log.e("BLE SCAN", scanResult.getBleDevice().getName());
Log.e("BLE SCAN", scanResult.getBleDevice().getMacAddress());
scanSubscription.unsubscribe();
RxBleDevice device = scanResult.getBleDevice();
subscription = device.establishConnection(false) // <-- autoConnect flag
.flatMap(rxBleConnection -> rxBleConnection.readCharacteristic(UUID.fromString("00002a19-0000-1000-8000-00805f9b34fb")))
.subscribe(
characteristicValue -> {
Log.e("Characteristic", characteristicValue[0]+"");
},
throwable -> {
Log.e("Error", throwable.getMessage());
}
);
}
}
}
)
.subscribe();
I can read two by using :
.flatMap(rxBleConnection -> Observable.combineLatest( // use the same connection and combine latest emissions
rxBleConnection.readCharacteristic(aUUID),
rxBleConnection.readCharacteristic(bUUID),
Pair::new
))
But I don't understand how to do that with 4 characteristics for example.
Thank you
The above example is just fine you would only need some data object that would accept more values than Pair. For instance something like:
class ReadResult {
final byte[] aValue;
final byte[] bValue;
final byte[] cValue;
final byte[] dValue;
ReadResult(byte[] aValue, byte[] bValue, byte[] cValue, byte[] dValue) {
this.aValue = aValue;
this.bValue = bValue;
this.cValue = cValue;
this.dValue = dValue;
}
}
And then the example could look like this:
disposable = rxBleClient.scanBleDevices(
new ScanSettings.Builder().build(),
new ScanFilter.Builder().setDeviceName("NODE 1").build() // one can set filtering by name here
)
.take(1) // take only the first result and then the upstream will get unsubscribed (scan will end)
.flatMap(scanResult -> scanResult.getBleDevice().establishConnection(false)) // connect to the first scanned device that matches the filter
.flatMapSingle(rxBleConnection -> Single.zip( // once connected read all needed values
rxBleConnection.readCharacteristic(aUUID),
rxBleConnection.readCharacteristic(bUUID),
rxBleConnection.readCharacteristic(cUUID),
rxBleConnection.readCharacteristic(dUUID),
ReadResult::new // merge them into a single result
))
.take(1) // once the result of all reads is available unsubscribe from the upstream (connection will end)
.subscribe(
readResult -> Log.d("Characteristics", /* print the readResult */),
throwable -> Log.e("Error", throwable.getMessage())
);
Original/Legacy solution for RxAndroidBle based on RxJava1:
subscription = rxBleClient.scanBleDevices(
new ScanSettings.Builder().build(),
new ScanFilter.Builder().setDeviceName("NODE 1").build() // one can set filtering by name here
)
.take(1) // take only the first result and then the upstream will get unsubscribed (scan will end)
.flatMap(scanResult -> scanResult.getBleDevice().establishConnection(false)) // connect to the first scanned device that matches the filter
.flatMap(rxBleConnection -> Observable.combineLatest( // once connected read all needed values
rxBleConnection.readCharacteristic(aUUID),
rxBleConnection.readCharacteristic(bUUID),
rxBleConnection.readCharacteristic(cUUID),
rxBleConnection.readCharacteristic(dUUID),
ReadResult::new // merge them into a single result
))
.take(1) // once the result of all reads is available unsubscribe from the upstream (connection will end)
.subscribe(
readResult -> Log.d("Characteristics", /* print the readResult */),
throwable -> Log.e("Error", throwable.getMessage())
);