I am trying to associate multiple log statements (which are log statements for the same HTTP request but created by different classes) based on the operation id. The log statements which are to be grouped have the same operation id.
I am using jsonPayload for the log statements.
I am using latest version for google-fluentd for logging.
Can anyone please let me know if there is anything else that is required for grouping the log statements in stackdriver?
Documentation for LogEntryOperation
Related
I am new to google-cloud and to stack driver logging. deployed a python script as cloud function also enabled in-built logging for python. I had set multiples logs based on the execution.
logging.info("Cloud fucntion was Triggerred on time {}".format(datetime.utcnow()))
While viewing in cloud function logs, i see many, like even the statement above, for which I have set INFO as the log level is displayed with !! icon which according to the legends should be displayed for only error types.
As I am new to this stack-driver logging, I am not sure of the reason, can anyone please explain? Thanks.
There's multiple executions happening interleaved/concurrently in firebase. The log id changes when new execution happens and the old execution id is forgotten. So the execution id moves forward only. When the old function resumes, it uses new execution id. Is there a way to achieve old execution id for old function & new execution id for new function.
Workflow:
Lets say Function1 & Function2 are diffrent triggers of same function.
1. Function1 does some db reads and do http requests. This returns an http promise - This takes some time(maybe some ms). Lets assume its execution-id from log is 154690519665944.
2. Function2 get triggered while function1 was waiting. function2 gets execution-id 154690574405903. function2 also does same thing and waits for http response.
3. Function1 resumes and it got http response and while logging it uses another execution-id 154694739233261 in log.
What happened to execution-id 154690519665944 ?
Since there's multiple triggers happening simultaneously, the only way to find whether a function completed successfully is to check logs. So by using execution-id as the filter, I could have find whether the function executed successfully or not. But because firebase changes execution-id randomly, I guess I have to find another solution.
PS: There's an update call which will trigger the same function. Does that change the parent function execution-id ?
Without seeing your code or complete logs, I don't think a definitive answer can be provided.
However, it sounds like the question you're asking is how can you keep data in memory across asynchronous transactions. Regardless of Firebase or not, you basically have two options:
commit that data to the database during the first part of the
transaction and then retrieve it in the second part
pass the data
from the first part to the second part so that it already has it.
You seem to be relying on the execution-id, so I would recommend taking the latter approach and passing that id along as part the input to your httprequest and having the server your calling return it in its response.
I need to run query with importing modules from pod.
Without importing modules if I run simple query with Database Id using below, it is working.
let $queryParam := fn:concat("?query=",xdmp:url-encode($query),"&eval=",$dataBaseId,":123")
let $url := fn:concat($hostcqport,"/eval.xqy",$queryParam)
let $response := xdmp:http-post($url, $options)[2]
If I have import modules statements then it is throwing Error(File Not Found).
So I tried getting the app-server id and tried passing that instead of database-id as below,
let $queryParam := fn:concat("?query=",xdmp:url-encode($query),"&eval=",$serverId,":123")
let $url := fn:concat($hostcqport,"/eval.xqy",$queryParam)
let $response := xdmp:http-post($url, $options)[2]
How to pass the server-id to make the query executing against particular app-server.
Is this MarkLogic 8 or earlier (I ask because rewrite options on 8 allow for dynamic switching of module databases before execution (among lots of other amazing goodies). This may be what you want because you can look at the query parameters at this point and build logic into the rewite rules.
Otherwise, Can you explain in more detail what you are trying to accomplish in the end. By the time your code ran, it was already executed in the context of a particular App server - so asking to execute against a another app server by analysing the query parameters is a bit too late (because you are already using the app server).
[edit] The following is in response to the comments since provided. This is a messy response because the actual ticket and comments are still not a completely clear picture. But if you stitch them together, then a problem statement does now exist for which I can respond.
The original author of the question confirmed via comments that they are "trying to hit an app server on a different node than the one that you actually posted to"
OK.. This is the response to that clarification:
That is not possible. Your request is already being processed by a thread on the node that you hit with your http request. Marklogic is a cluster, but it does not share threads (or anything else for that matter). Choices are:
a redirect to the proper node
possibly use the current node to make the request on your behalf.
But that ties up the first thread and the thread on the other node and has the HTTP communication overhead - and you need to have an app server listening for this purpose.
If this is a fire-and-forget type of situation, then you can hit any node and save the data/request in a document in the DB using a URI naming convention that indicates what app server it is for, and by way of insert triggers (with a URI-prefix for their server id), pick up the request from the DB and process it.
How to get the user who initiated the process in IBM BPM 8.5. I want to reassign my task to the user who actually initiated the process. How it can be achieved in IBM BPM?
There are several ways to get that who initiated a Task , But who initiated a process Instance is somewhat different.
You can perform one out of the following :
Add a private variable and assign it tw.system.user_loginName at the POST of start. you can access that variable for user who initiated the process.(It will be null or undefined for the scenario if task is initiated by some REST API or UCA.)
Place a Tracking group after Start event . Add a input variable to it as username , assign it a value same as tw.system.user_loginName. So whenever Process is started entry will be inserted to DB Table.You can retrieve this value from that view in PerformanceDB.
Also there might be some table ,logging the process Instances details , where you can find the user_id directly.
I suggest you to look in getStarter() method of ProcessInstanceData API.
Official Documentation on API
This link on IBM Developerworks should help you too: Process Starter
Unfortunately there's not an Out Of The Box way to do this - nothing is recorded in the Process Instance that indicates "who" started a process. I presume this is because there are many ways to launch a process instance - from the Portal, via a Message Event, from an API call, etc.
Perhaps the best way to handle this is to add a required Input parameter to your BPD, and supply "who" started the process when you launch it. Unfortunately you can't supply any inputs from the OOTB Portal "New", but you can easilty build your own "launcher".
If you want to route the first task in process to the user that started the process the easiest approach is to simply put the start point in the lane, and on the activity select routing to "Last User In Lane". This will take care of the use case for you without requiring that you do the book keeping to track the user.
Its been a while since I've implemented this, so I can't remember if it will work elegantly if you have system steps before the first task, but this can easily be handled by moving the system steps into the human service to be executed as part of that call, rather than as a separate step in the BPD.
Define variable as string type and using script task to define the login user that use this task and assign it to your defined variable to keep to you in all of the process as initiator of the task.
You can use this line of code to achieve the same:
tw.system.user_loginName
We receive many large data files daily in a variety of formats (i.e. CSV, Excel, XML, etc.). In order to process these large files we transform the incoming data into one of our standard 'collection' message classes (using XSLT and a pipeline component - either built-in or custom), disassemble the large transformed message into individual 'object' messages and then call a series of SOAP web service methods to handle business logic and database operations.
Unlike other files received, the latest file will contain all data rows each day and therefore, we have to handle the differences to prevent identical records from being re-processed each day.
I have a suitable mechanism for handling inserts and updates but am currently struggling with the deletes (where the record exists in the database but not in the latest file).
My current thought process is to flag the deleted records in the database using a 'cleanup' task at the end of the entire process but this would require a method to be called once all 'object' messages from the disassembled file have completed.
Is it possible to monitor individual messages from a multi-record file and call a method on completion of the whole file? Currently, all research is pointing to an orchestration with some sort of 'wait' but is this the only option?
Example: File contains 100 vehicle records. This is disassembled into 100 individual XML messages which are processed using 100 calls to a web service method. Wish to call cleanup operation when all 100 messages are complete.
The best way I've found to handle the 'all rows every day' scenario is to pre-stage the data in SQL Server where it's easier to compare the 'current' set to the 'previous' set. The INTERSECT and EXCEPT operators make it pretty easy in most cases.
Then drain the records with a Polling statement.
The component that does the de-batching would need to publish a start of batch message with the number of individual records and a correlation key.
The components that do the insert & update would need to publish a completion message with the same correlation key when it is completed processing.
The start of batch message would have spun up an Orchestration that would would listen for the completion messages with that correlation key and count the number, and either after it has received the correct number or after a timeout period it would call the cleanup or raise an exception.