I am looking for a way to capture when a call is completed on the server and no errors were thrown.
I understand that SimpleForwardingServerCallListener exists, however,onComplete is called when an exception is thrown.
My use case is for transaction management.
Yes, it currently only triggers onComplete(). There is a bug filed for that. If it's fixed, you can probably get onCancel() instead.
For now, you can wrap the ServerCall with SimpleForwardingServerCall in your ServerInterceptor, and override close(). If the RPC ends successfully, OK will be passed to close().
Related
Note: Help with the immediate problem would be great, but mostly I'm looking for advice on troubleshooting gRPC timing issues in general (this isn't my first such issue).
I am adding a new server streaming service to a C++ module which has an existing server streaming service, and the two appear to be conflicting. Specifically, the completion queue Next() call on the server is crashing intermittently after the C# client calls Cancel() on the cancellation token for one of the services. This doesn't happen if I run each service independently.
On the client, I get this at the response stream MoveNext() call:
System.InvalidOperationException
HResult=0x80131509
Message=Shutdown has already been called
Source=Grpc.Core
StackTrace:
at Grpc.Core.Internal.CompletionQueueSafeHandle.BeginOp()
at Grpc.Core.Internal.CallSafeHandle.StartReceiveMessage(IReceivedMessageCallback callback)
at Grpc.Core.Internal.AsyncCallBase`2.ReadMessageInternalAsync()
at Grpc.Core.Internal.ClientResponseStream`2.<MoveNext>d__5.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at MyModule.Connection.<DoSubscriptionReceives>d__7.MoveNext() in C:\snip\Connection.cs:line 67
On the server, I get this at the completion queue next() call:
Exception thrown: read access violation.
core_cq_tag->**** was 0xDDDDDDDD.
The stack trace:
MyModule.exe!grpc_impl::CompletionQueue::AsyncNextInternal(void * * tag, bool * ok, gpr_timespec deadline) Line 59 C++
> MyModule.exe!grpc_impl::CompletionQueue::Next(void * * tag, bool * ok) Line 176 C++
...snip...
It appears something is being added to the queue after shutdown. The difficulty is I have little visibility into what is being added into the queue and in what order.
I'm trying to write a server-side interceptor to log all requests & responses, but there seems to be no documentation. So far, poking through the API hasn't gotten me very far. Is there any documentation available on wiring up an interceptor in C++? Or, are there other approaches for troubleshooting timing conflicts between services?
Windows 11, Grpc.Core 1.27
What I've tried:
I first played with the GRPC_TRACE & GRPC_VERBOSITY environment variables. I was able to get some unhelpful output from the client, but nothing from the server. Of course, there's been lots of debugging, stripping the client & server down to barebones, disabling keep alives, ensuring we aren't using deadlines, having the services share a cancellation token, etc.
Update: I have found that the crash only happens when the client is run from an NUnit test. In that environment, the completion queue is getting more hits on Next(), but I'm still trying to figure out where they are coming from.
Is 1.27 the version you are using? That seems pretty old.. There might have been fixes since then.
For using the C++ server interception API, I think you would find this very useful - https://github.com/grpc/grpc/blob/0f2a0f5fc9b9e9b9c98d227d16575d106f1e8d43/test/cpp/end2end/server_interceptors_end2end_test.cc#L48
One suggestion I have is to run the code another sanitizers https://github.com/google/sanitizers to make sure that we don't have a heap-use-after-free type bug.
I would also check for API misuse issues. (If you had posted the code, I could've given a look to see if anything seems weird..)
I am using C++/winRT UWP to discover and connect to Bluetooth Low Energy devices. I am using the advertisment watcher to look for advertisements from devices I can support. This works.
Then I pick one to connect to. The connection procedure is a little weird by my way of thinking but according to the microsoft docs one Calls this FromBluetoothAddressAsync() with the BluetoothAddress and two things happen; one gets the BluetoothLEDevice AND a connection attempt is made. One needs to register a handler for the connection status changed event BUT you can't do that until you get the BluetoothLEDevice.
Is there a timing issue causing the exception? Has the connection already happened BEFORE I get the BluetoothLEDevice object? Below is the code and below that is the log:
void BtleHandler::connectToDevice(BluetoothLEAdvertisementReceivedEventArgs eventArgs)
{
OutputDebugStringA("Connect to device called\n");
// My God this async stuff took me a while to figure out! See https://msdn.microsoft.com/en-us/magazine/mt846728.aspx
IAsyncOperation<Windows::Devices::Bluetooth::BluetoothLEDevice> async = // assuming the address type is how I am to behave ..
BluetoothLEDevice::FromBluetoothAddressAsync(eventArgs.BluetoothAddress(), BluetoothAddressType::Random);
bluetoothLEDevice = async.get();
OutputDebugStringA("BluetoothLEDevice returned\n");
bluetoothLEDevice.ConnectionStatusChanged({ this, &BtleHandler::onConnectionStatusChanged });
// This method not only gives you the device but it also initiates a connection
}
The above code generates the following log:
New advertisment/scanResponse with UUID 00001809-0000-1000-8000-00805F9B34FB
New ad/scanResponse with name Philips ear thermometer and UUID 00001809-0000-1000-8000-00805F9B34FB
Connect to device called
ERROR here--> onecoreuap\drivers\wdm\bluetooth\user\winrt\common\bluetoothutilities.cpp(509)\Windows.Devices.Bluetooth.dll!03BEFDD6: (caller: 03BFB977) ReturnHr(1) tid(144) 80070490 Element not found.
ERROR here--> onecoreuap\drivers\wdm\bluetooth\user\winrt\device\bluetoothledevice.cpp(428)\Windows.Devices.Bluetooth.dll!03BFB9B7: (caller: 03BFAF01) ReturnHr(2) tid(144) 80070490 Element not found.
BluetoothLEDevice returned
Exception thrown at 0x0F5CDF2F (WindowsBluetoothAdapter.dll) in BtleScannerTest.exe: 0xC0000005: Access violation reading location 0x00000000.
It sure looks like there is a timing issue. But if it is, I have no idea how to resolve it. I cannot register for the event if I don't have a BluetoothLEDevice object! I cannot figure out a way to get the BluetoothLEDevice object without invoking a connection.
================================ UPDATE =============================
Changed the methods to IAsyncAction and used co_await as suggested by #IInspectable. No difference. The problem is clearly that the registered handler is out of scope or something is wrong with it. I tried a get_strong() instead of a 'this' in the registration, but the compiler would not accept it (said identifier 'get_strong()' is undefined). However, if I commented out the registration, no exception is thrown but I still get these log messages
onecoreuap\drivers\wdm\bluetooth\user\winrt\common\bluetoothutilities.cpp(509)\Windows.Devices.Bluetooth.dll!0F27FDD6: (caller: 0F28B977) ReturnHr(3) tid(253c) 80070490 Element not found.
onecoreuap\drivers\wdm\bluetooth\user\winrt\device\bluetoothledevice.cpp(428)\Windows.Devices.Bluetooth.dll!0F28B9B7: (caller: 0F28AF01) ReturnHr(4) tid(253c) 80070490 Element not found.
But the program continues to run an I continue to discover and connect. But since I can't get the connection event it is kind of useless at this stage.
I hate my answer. But after asynching and co-routining everything under the sun, the problem is unsolvable by me:
This method
bluetoothLEDevice = co_await BluetoothLEDevice::FromBluetoothAddressAsync(eventArgs.BluetoothAddress(), BluetoothAddressType::Random);
returns NULL. That should not happen and there is not much I can do about it. I read that as a broken BLE API.
A BTLE Central should be able to do as follows
Discover a device if new then:
If user selects connect, connect to
the device
perform service discovery
read/write/enable
characteristics as needed
handle indications/notifications
If at any time the peripheral sends a security request or insufficient authentication error, start pairing
repeat the action that caused the insufficient authentication.
On disconnect, save the paired and bonded state if the device is pairable.
On rediscovery of the device, if unpaired (not a pairable device)
repeat above
If paired and bonded
start encryption
work with the device; no need to re-enable or do service discovery
========================= MORE INFO ===================================
This is what the log shows when the method is called
Connect to device called
onecoreuap\drivers\wdm\bluetooth\user\winrt\common\bluetoothutilities.cpp(509)\Windows.Devices.Bluetooth.dll!0496FDD6: (caller: 0497B977) ReturnHr(1) tid(3b1c) 80070490 Element not found.
onecoreuap\drivers\wdm\bluetooth\user\winrt\device\bluetoothledevice.cpp(428)\Windows.Devices.Bluetooth.dll!0497B9B7: (caller: 0497AF01) ReturnHr(2) tid(3b1c) 80070490 Element not found.
BluetoothLEDevice returned is NULL. Can't register
Since the BluetoothLEDevice is NULL, I do not attempt to register.
================= MORE INFO ===================
I should also add that taking an over-the-air sniff reveals that there is never a connection event. Though the method is supposed to initiate a connection as well as return the BluetoothLEDevice object, it ends up doing neither. My guess is that the method requires more pre-use setup of the system that only the DeviceWatcher does. The AdvertisementWatcher probably does not.
In BLE you always have to wait for every operation to complete.
I am not an expert in C++, but in C# the async connection procedure returns a bool if it was successful.
In C++ the IAsyncOperation does not have a return type, so there is no way to know if the connection procedure was successful or completed.
You will have to await the IAsyncOperation and make sure that you have a BluetoothLEDevice object, before you attach the event handler.
To await an IAsyncOperation there is a question/answer on how to await anIAsyncOperation:
How to wait for an IAsyncAction? How to wait for an IAsyncAction?
Is there anyway to log message contents when an exception occurs?
I looked at various logging extensions but they are just logging CorrelationId. And message contents are not available.
There is a CurrentMessge property in MessageContext, but that is not available at the time logger writes the exception.
I tried to handle PoisonMessage Event, which allows me to log the message contents.
public static void OnPoisonMessage(IBus bus, ReceivedTransportMessage receivedTransportMessage, Rebus.Bus.PoisonMessageInfo poisonMessageInfo) {
var message = new JsonMessageSerializer().Deserialize(receivedTransportMessage);
Log.Error("{#messageType} failed {#message}", message.Messages[0].GetType(), message);
}
This works great, but now I have two errors in the log one coming from my handler and the other coming from logger.
I am wondering if there is a better way to handle this requirement.
If your requirement is to simply log the message contents as JSON, I think you've found the right way to do it - at least that's the way I would have done it.
I'm curious though as to what problem you're solving by logging the message contents - you are aware of the fact that the failing message will end up in an error queue where you can inspect it?
If you're using MSMQ, you can inspect JSON-serialized messages using Rebus' Snoop-tool, which is a simple MSMQ inspector. It will also allow you to move the message back into the input queue where it failed ("return to source queue")
A good way to monitor your Rebus installation is to set up some kind of alerts when something arrives in an error queue, and then you can look at the message (which event includes the caught exceptions in a special header) and then resolve the situation from there.
If, as part of a single Meteor.call, I make two calls to the database on the server, will these happen synchronously or do I need to use a callback?
Meteor.methods({
reset: function(id) {
Players.remove(_id:id);
// Will the remove definitely have finished before the find?
Players.find();
...
}
From the docs:
In Meteor, your server code runs in a single thread per request, not in the asynchronous callback style typical of Node. We find the linear execution model a better fit for the typical server code in a Meteor application.
If you read the docs on docs.meteor.com/#remove
you can find this :
Blockquote
On the server, if you don't provide a callback, then remove blocks until the database acknowledges the write and then returns the number of removed documents, or throws an exception if something went wrong. If you do provide a callback, remove returns immediately. Once the remove completes, the callback is called with a single error argument in the case of failure, or a second argument indicating the number of removed documents if the remove was successful.
Blockquote
On the client, remove never blocks. If you do not provide a callback and the remove fails on the server, then Meteor will log a warning to the console. If you provide a callback, Meteor will call that function with an error argument if there was an error, or a second argument indicating the number of removed documents if the remove was successful.
So on server side you choose if you want it to run in a sync or async way, it depends if you send a callback or not.
For a user application (not a driver) using WinUSB, I use WinUsb_ControlTransfer in combination with overlapped I/O to asynchronously send a control message. Is it possible to cancel the asynchronous operation? WinUsb_AbortPipe works for all other endpoints but gives an 'invalid parameter' error when the control endpoint is passed (0x00 or 0x80 as the pipe address). I also tried CancelIo and CancelIoEx but both give an 'invalid handle' error on the WinUSB handle. The only related information I could find is on http://www.winvistatips.com/winusb-bugchecks-t335323.html, but offers no solution. Is this just impossible?
Probably not useful to the original asker any more, but in case anyone else comes across this: you can use CancelIo() or CancelIoEx() with the file handle that you originally passed in to WinUsb_Initialize().
This is similar to how the documentation of WinUsb_GetOverlappedResult says:
This function is like the Win32 API routine, GetOverlappedResult, with one difference—instead of passing a file handle that is returned from CreateFile, the caller passes an interface handle that is returned from WinUsb_Initialize. The caller can use either API routine, if the appropriate handle is passed. The WinUsb_GetOverlappedResult function extracts the file handle from the interface handle and then calls GetOverlappedResult.