While using google cloudVision services we have a need to track individual requests made and the response. Is there a way I can generate an ID as a memo field to every request I make to the CloudVision API & read it in the usage logs/ billing statement?
I can get close to this with timestamps- but with concurrent usages and latencies, mapping can get challenging. Along with reconciliation, this will also help track error codes and reasons
I am wondering whether you can log every request which has error yourself?
You may sending request in parallel in multiple thread, but in each thread you can just write the request and its result to a GCS bucket to record the log.
Related
So suppose I have an API to create a cloud instance asynchronously. So after I made an API call it will just return the success response, but the cloud instance will not been initialized yet. It will take 1-2 minutes to create cloud instance and after that it will save the cloud instance information (ex. ip, hostname, os) to db which mean I have to wait 1-2 minutes so I can fetch the data again to show cloud information. At first I try making a loading component, but the problem is that I don't know when the cloud instance is initialized (each instance has different time duration for creating). I'm considering using websocket or using cron or should I redesign my API? Has anyone design asynchronous system before how do you handle such a case.
If the API that you call gives you no information on when it's done with its asynchronous processing, it seems to me that you'll have to check at intervals until you find that the resource is ready; i.e. to poll it.
This seems to me to roughly fit the description and intent of the Polling Consumer pattern. In general, for asynchronous systems design, I can't recommend Enterprise Integration Patterns enough.
As other noted you can either have a notification channel using WebSockets or poll the backend. Personally I'd probably go with the latter for this case and would actually create several APIs, one for initiating the work and get back a URL with "job id" in it where the status of the job can be polled.
RESTfully that would look something like POST /instances to initiate a job GET /instances see all the instances that are running/created/stopped and GET /instances/<id> to see the status of a current instance (initiating , failed , running or whatever)
WebSockets would work, but might be an overkill for this use case. I would probably display a status of 'creating' or something similar after receiving the success response from the API call, and then start polling the API to see if the creation process has finished.
Can anyone clarify what is the purpose of using queue ?
What i understand is that a webhook is just a URL , you do a POST request to that URL and then do some stuff based on the body/data of the request. So why i need to queue the data and store it in a database then loop through the database again and perform the stuff.
The short answer is, you don't have to use a queue. A webhook is just an HTTP request (typically POST) notifying your application of some type of event. The reason you might want to consider a queue is because of typical issues you could run into.
One of these is because of response time back to the webhook requester (source). Many sources want a response (HTTP status 200) as quickly as possible so they can dequeue the request from their webhook system. If processing the webhook takes some time, a source will typically advise you to use a queue to defer the lengthier process asynchronous to the 200 response to the webhook.
Another possible reason could be for removing duplicate requests. There is no guarantee with webhooks that you will only receive a single request per event. A queue can be used to de-dupe these requests.
I would recommend you stick with a simple request handler if possible, then evolve a more sophisticated handler if you run into issues. Consider queues as a potential design approach if you run into issues like those above.
You need some way to prevent a conflict if the webhook is invoked multiple times very close together.
It doesn't necessarily have to be a queue, though. If the webhook performs database queries and updates, you can use a transaction to ensure that this is atomic for each invocation.
In this respect, it's little different from any other web utility. You should do something similar in scripts that process web forms.
Currently, I am using Measurement Protocol to push the data to GA. The problem is I didn't get any response back for Success or Error on Production, If yes Please suggest?
Due to this, I am looking if there is any other options available for the same like can we achieve it using analytics 360?
The google analytics production data collection endpoint does not return a request status back (always 200 OK) by design to ensure ultra-light processing speed.
What I usually recommend to clients using Measurement Protocol server-side is to
To log a reasonable amount (or all of them) of requests somewhere. Storage is extremely cheap nowadays and knowing the data format if an emergency happens you will be able to manually extract the data
Every once in a while (one on thousand or one on a million or even more oftne depending on the importance of the data randomly) validate request on GA debug endpoint and parse the returned json. If there are any warnings or error send a notification for further investigation. This way, if anything went wrong you will be on top of the problem by the time BI & Marketing Team would be affected
We have a REST API implemented as a Cloud Service, that sends telemetry to Application Insights. And we use commands like
POST /api/groups/GRP_75e0b852-ee21-45fb-b943-13aa465c62da/members.
POST /api/groups/GRP_75e0b852-ee21-45fb-b943-13aa465c62da/folders/FLD_080af364-ad37-4351-837e-4fb1d5f02e50/discussions
The sections of the command preceded by GRP_ and FLD_ are parameters.
This makes looking at the breakdown of requests in Application Insights difficult since those requests show up individually.
I’ve implemented an ITelemetryInitializer that “normalizes” the Context.Operation.Name (and the Request URL) in our requests. But I see that those request are showing up bucketed as “Other Values”.
Requests with Other Values
Is there any way to reset the "bucketing" of the top-level list, or do I need to get a new AppInsights instance?
Standard dimentions like request name should be reset after a week. So if you stopped collecting parameters in the names it should clear up after a week. Current limit is 1000.
We are building service that will synchronize with user Google Tasks data, so if user add/edit/delete task in GTask, so it will be added/edited/deleted in our service.
And there is a big problem with synchronization: as I see GTasks API does not provide any onUpdate/onChange event listeners. I mean, the perfect solution can be if there will be Google Tasks API method that can be used to set some callback URL that will be requested when user add/edit/delete tasks.
But I can't find such method in Google Tasks API, so now there is only one very bad way to sync with Google Tasks API - request all users tasks and compare them with service tasks. This is very bad way to sync, because if we have 10k users and want their tasks list be synchronizaed up to 1 minute, so we will need to make > 10k GTasks API requests per minute :(
I hope that I'm wrong and there is some way to set onChange/onUpdate callback for user tasks. Or may be there is some another way to receive actual notification of user GTasks changes(by email & etc).
Does anybody know it?
Thank you.
You could use updatedMin parameter to only get Tasks that have been updated since a given timestamp, as described in the documentation.
You should be able to rely on ETag and If-None-Match headers when querying user tasks lists to get a 304 Not Modified if the no tasks in the list have changed. (Not that should also works when polling individual tasks)
This way you can effectively poll for the tasks that have changed since the last time you synced.