I've got a AWS lambda function that takes a large array of email addresses and submits them to SES for sending. The function is invoked via a SNS subscription. It works nicely when the message's email array size is small, however when the message's email array size is large the lambda function is NOT invoked. No logging occurs....
I've confirmed that the payload is below the SNS message size required, and I've subscribed to the SNS topic via my email...this works. I get emailed the expected message payload.
Any idea why this fails silently? Suggestions on how to work around this?
For posterity:
Configure the 'Delivery Status' for the SNS topic. It will then log success/failures to CloudWatch.
Expect to see:
167542 byte payload is too large for the Event invocation type (limit 131072 bytes)
So...even though SNS can send a larger payload, that doesn't mean lambda's can be invoked with this size payload. Yuck!
A couple workarounds for this:
1) Post the data to an S3 bucket. Have the Lambda function listen for Object CREATE on that bucket.
2) Post the data to S3, and publish an SNS event with just the bucket and key where the data can be found in S3. Have Lambda subscribe to the SNS topic.
In either case you're going to end up GET-ing the data from an S3 bucket inside your Lambda function. Now you have no size limits. :-)
Related
Working on an Android app to send data to a peripheral device via BLE. Question about Android BluetoothGATTCallback onCharacteristicWrite function - how does it know that the write transaction was successful? Is success assumed so long as no error occurs? Or does it record a success response of some sort sent from the peripheral device characteristic to which data is written?
I would say matdev's comments in his answer is not correct, at least according to my experience.
Assuming you use "Write With Response" (WRITE_TYPE_DEFAULT) rather than "Write Without Response" (WRITE_TYPE_NO_RESPONSE), the following applies:
First you call writeCharacteristic. This method performs some sanity checks, such as the characteristic you are trying to write to has the "write" property, there are no other pending GATT operations on this BluetoothGatt object, and that the object is not closed. If those sanity checks are passed, this method returns true.
The write request is then transferred over the air to the remote device's GATT server. Here the request is processed and the GATT server returns a Write Response or an Error Response with an error code, which is sent back to the Android device. When Android receives a Write Response, it will call onCharacteristicWrite with GATT_SUCCESS (0) as status. If Android instead receives an Error Response, it will call onCharacteristicWrite and set the status to the code included in the Error Response. If the GATT server e.g. returns 0x80 (Application Error) or 0x13 (Value Not Allowed), this is the status code you will receive in your callback.
If the connection drops before the response is received, Android will call the callback with some currently undocumented non-success status code.
If you instead use "Write Without Response" (WRITE_TYPE_NO_RESPONSE), the behaviour of onCharacteristicWrite changes. Now, this callback is instead used for flow control. When the local Bluetooth stack's buffers are ready to accept another packet, this is when you will get the callback. This can happen even before the packet is sent over the air.
You can assume the BLE write transaction is successful if the status param of BluetoothGATTCallback's onCharacteristicWrite() equals BluetoothGatt.GATT_SUCCESS i.e. 0
If an error occurs, the status value will indicate the type of error, such as
/** GATT write operation is not permitted */
public static final int GATT_WRITE_NOT_PERMITTED = 0x3;
Here is an extract of the doc about onCharacteristicWrite():
Params:
...
status – The result of the write operation BluetoothGatt.GATT_SUCCESS if the operation succeeds.
Really bizarre that Firebase doesn't seem to work quite like typical Express app. Whatever I write in Express and copy-paste to Firebase Functions I typically get error. There is one that I can't figure out on my own though.
This endpoint is designed to start a function and live long enough to finish even longer task. That request is a webhook (send docs, we will transform them and ping you when it's done to specified another webhook). Very simplified example below:
router.post('/', (req, res) => {
try {
generateZipWithDocuments(data) // on purpose it's not async so request can return freely
res.sendStatus(201)
} catch (error) {
res.send({ error })
}
})
On my local machine it works (both pure Express app and locally emulated Firebase Functions), but in the cloud it has problems and even though I put a cavalcade of console.log() I don't get much information. No error from Firebase.
If generateZipWithDocuments() is not asynchronous res.sendStatus() will be immediately executed after it, and the Cloud Function will be terminated (and the job done by generateZipWithDocuments() will not be completed). See the doc here for more details.
You have two possibilities:
You make it asynchronous and you wait its job is completed before sending the response. You would typically use async/await for that. Note that the maximum execution time for a Cloud Function is 9 minutes.
You delegate the long time execution job to another Cloud Function and, then, you send the response. For delegating the job to another Cloud Function, you should use Pub/Sub. See Pub/Sub triggers, the sample quickstart, and this SO thread for more details on how to implement that. In the Pub/Sub triggered Function, when the job is done you can inform the user via an email, a notification, the update of a Firestore document on which you have set a listener, etc... If generateZipWithDocuments() takes a long time, it is clearly the most user friendly option.
I need to get the number of messages that are ready. A queue has three types of messages: 1. Total 2. Unack'd 3. Ready
Ready is the ones' that are in the queue but haven't been consumed yet.
Currently I use requests
url = "http://<RABBITHOST>:15672/api/queues/%2f/{}".format(q)
res = requests.get(url, auth=("<user>","<password>")).json()
messages_in_queue = res.get("messages_ready")
The problem here is that I have to pass in the username and password. Using Pika I believe you can get the "total" messages. Is there any way to get the other two types (unack'd and ready) using Pika?
No, the AMQP protocol doesn't support getting unacknowledged messages. You will still have to use the HTTP API for that. If you do a passive queue declaration, the message count returned is the number of ready messages.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
I have a Firebase Cloud Function that is triggered by a PubSub message. The function should either consume the message or wait to consume it at a later time.
Is there a way to return from this function without acknowledging the message, so that it will be re-delivered at a later time?
For example, can I return a Message from the cloud function? The docs seem to indicate this is possible, if I'm reading them right:
Returns
non-null functions.CloudFunction containing non-null functions.pubsub.Message A Cloud Function which you can export.
When PubSub trigger a function (Firebase or Cloud Function), if the function end correctly, the message is acknowledge. But if the function crash or raise an exception (in summary, an abnormal terminaison) the message not ack and resend immediately.
This retry loop is performed until the message is ack or the message expired (default and maximum TTL of 7 days, minimum is 10 minutes. You can customize the messageRetentionDuration in the subscription)
I am trying to use Skype's DBus API in order to retrieve the list of messages (message IDs) I've exchanged with a contact. However, both the SEARCH CHATMESSAGES <target> (protocol >= 3) and the SEARCH MESSAGES <target> (protocol < 3) commands return unexpectedly empty results.
Here is the trace of a few exchanges I had with the API. I used d-feet to send my requests, but the result is exactly the same when I send the request from my own program.
Bus name: com.Skype.API
Object: /com/Skype
Interface: com.Skype.API
Method used: Invoke(String request)
Trace:
-> NAME dfeet
<- OK
-> PROTOCOL 8
<- PROTOCOL 8
-> SEARCH CHATMESSAGES mycontact
<-
The same thing happens with two other SEARCH commands:
SEARCH MESSAGES <target> (with PROTOCOL 2).
SEARCH CHATS
Additionally, I also get an empty result when I try to request a message list based on a chat ID: GET CHAT <chat_id> GETMESSAGES.
However, commands such as SEARCH FRIENDS, SEARCH CALLS, or SEARCH ACTIVECHATS work just fine, and return their lists of IDs (contacts IDs, calls IDs, or chat IDs) as expected.
It might also be worth noting that this happens for all contacts, regardless of how many messages I've exchanged with them (I thought at first that there might be too many messages involved, but the result is the same, whether I've sent 3, or thousands of messages to the contact).
Is there anything that would explain why I get these empty responses through DBus, for these requests?
Skype will not use Invoke's return value when its reply is too heavy. As it so happens, when Skype has too much data to prepare and transfer after a request, it automatically returns an empty string to the Invoke call. The true, heavy reply is then prepared asynchrously by Skype, and the client program must be ready to receive it when it eventually arrives.
Whenever you are communicating with Skype over DBus, your application must act as both a client (calling Invoke), and a server (providing a DBus object for Skype to reach). This design was a little unexpected (I guess we could argue on its quality), but here is what it requires you to do:
Make your program a DBus "server" (providing objects to reach). Through your bus name to Skype, register an object path called /com/Skype/Client implementing the com.Skype.API.Client interface.
Prepare a message handler for the only method of this interface: Notify(s). This is the method Skype will try to call to send you the heavy reply to one of your previous requests.
Program your own mechanism to match your Invoke request with the asynchronous Notify message coming in as an answer later on.
The creation of an object can be done through dbus_connection_register_object_path, the parameters for which are:
The DBusConnection structure representing your bus name.
The object path you are registering, here /com/Skype/Client.
A table of message handlers (DBusObjectPathVTable) used to process all incoming requests.
Data to be sent to these handlers when they are called. This is additional data, not the actual message being received since you're just setting up the handler here.
For instance...
DBusHandlerResult notify_handler(DBusConnection *connection,
DBusMessage *message,
void *user_data){
return DBUS_HANDLER_RESULT_HANDLED;
}
void unregister_handler(DBusConnection *connection,
void *user_data){}
DBusObjectPathVTable vtable = {
unregister_handler,
message_handler,
NULL
};
if(!dbus_connection_register_object_path(connection,
"/com/Skype/Client",
&vtable, NULL)){
// Error...
}
Note that this is just an object's definition. In order to actually hook on the Notify calls, you'll have to select() on a DBusWatch file descriptor, and dispatch the incoming DBusMessage in order to have your message handler called.
If you are working with other bindings, you'll probably find much faster ways to setup objects and start working as a client application. See:
GLib's g_dbus_connection_register_object
Exporting objects with dbus-python
QtDBus's QDBusConnection::registerObject
... (other bindings)