I am very new to stack driver and trying to implement some charts which i have implemented in splunk for a different product.
We have the api name in the logs under textPaylod field and i want to extract the api name from the field and create a chart based on the counts of API names.
ex below is the sample log.
type: "k8s_container"
}
severity: "INFO"
textPayload: "19-04-29T04:30:51.058+0000 INFO PostFilter: POST response to http://<endpoint>/abc/def/users/getNames
"
timestamp: "2019-04-29T04:30:51.059143860Z"
}
type: "k8s_container"
}
severity: "INFO"
textPayload: "19-04-29T04:30:51.058+0000 INFO PostFilter: POST response to http://<endpoint>/abc/def/users/getPhoneNumbers
"
timestamp: "2019-04-29T04:30:51.059143860Z"
}
I've create a custom metric and extracted the text after "/abc/def" into API_NAME label expecting to use it as group function in the metric.
Crating Custom Metric
When i tried to explore the metric and see the counts in a stacked bar i am not able to find the counts by apiname
Metric Explorer
When asking for help debugging a specific issue you've encountered following existing instructions, you may get a better response by emailing google-stackdriver-discussion#googlegroups.com .
As outlined in Logs-based Metric Labels, you should specify the appropriate capture group to extract the value of the label.
You can then see the time series for the logs-based metric you've created (see https://cloud.google.com/monitoring/api/troubleshooting for how to query the raw data). It's likely that your regular expression is not matching exactly what you think it's matching, and you are always getting an empty value for the API_Name label. One suspect is the escaped \? in your pattern — according to the RE2 syntax, ? should not be escaped.
Related
Hi, I'm new to kibana visualize.
Background
I want to visualize cache hit ratio of api response.
when it's a hit I log ...: “<url…> <some info…> USING CACHE”
what it's a miss I log...: “<url …> <some info…> NO HIT”
Query
I made two Y-Axis (picture below) but two graphs are same...
Question
How do I show cache hits and misses in one graph?
Do I have to add a new filter? or X-Axies?
I remember wrongly the options about the Y-axis in the aggregation based visualizations. You can build those kind of visualizations both with TSVB and Aggregation Based viz.
With TSVB you just select the index pattern in the panel options (you can also restrict the documents searched with a Kibana query in the "panel filter")
In the data section you can build two filter metrics: in the example "error" and "warn", but in your case you can produce a "hit" metric with query string content: "USING CACHE" and similarly with a "miss" metric with content: "NO HIT"
If you instead want to use an aggregation based visualization like in your screenshot, you can split the series like in the following example
I've set up a device on my Thingsboard server. Data is being received from my device.
When I add a chart to the device's dashboard and set the timewindow to any value to "last" and the data aggregation function to anything but "none", nothing shows in the graph.
Any data aggregation function (besides "none") causes no data to be displayed in the graph.
Is there something I'm missing?
Thingsboard supports different data types. If you are submitting your telemetry as String, then it is saved as the string. And in this case, 'SUM' and 'AVG' functions will return empty dataset because it is not possible to calculate the sum and average value on String data. But 'MIN/MAX/COUNT' aggregation functions will work.
For example:
{"key1":"10"} - value is saved as string ("10") - SUM and AVG will not work
{"key1":10} - value is saved as integer (10) - all aggregation functions will work
You can enable debug mode in 'save telemetry' node to find original messages that are published to the Thingsboard.
You can also find more details on GitHub issues:
#1149
#817
I am having a field named rpc in my elasticsearch database and I am displaying it using Kibana. When I search in search bar of kibana like:
rpc:*
It display all the values of rpc field but I want to have only those value to be displayed which are unique.
I have been playing around with Kibana4 since a couple of weeks now. I find it intuitive and simple and the experience has been great till now. Following your question, I tried getting unique results via a Data Table visualization. Why? Because I personally find it easier to understand. Following are the steps:
1. Get unique count
Create the visualization (Visualize -> Data Table). First lets get
the count of how many unique entries we have for a particular field
(We will use this in the later part for verification). I'm using
clientip.raw but as I see, it will work just fine with any friendly
field name too.
2. Set the aggregation right
Set you aggregation back to count and have a Split Rows as follows. Not doing this will give you count 1 for each field value (since it is looking for unique counts) when you populate the table. Noteworthy part is setting the Top field to 0. Because Kibana won't let you enter anything else than a digit (Obviously!). This was the tricky part. Hit Apply and you'll get the results. Unique field values and the count of each of them.
3. Verification:
Going to the last page of the table, we see there are exactly 543 results. This is how I know it works.
What Next?
You save this visualization and add it to a Dashboard. There you can always check the request, query, response and other stats.
Just an addition to the above mathakoot answer.
For the user of newer version (which do not allow bucket size of 0 anymore) just set a value greater than the maximum number of result
And report the value in the Options>Per Page field
I am using Kibana 6 so the UI looks a bit different than the older answers here.
Here is what worked for me
Create a visualization from your query, I used a line graph type (don't think it matters)
Under Data, set metrics aggregation = "Unique Count" and set field to your field.
Set x-axis aggregation = "Terms" and set field to your field.
Set Size > your number of records
Under Metrics and Axes, disable drawing of the graph, circles, and labels (this really helps the UI not lag)
Run query and then click "Inspect" and download CSV
Data
Metrics & Axes
I wanted to achieve something similar but I'm stuck with Kibana 3.1.
I simply added a panel of type "TERMS" and configured its Field = User-agent and left everything else on default values. This gave me a nice bar chart with one bar for each User-agent.
I curled the sample bank data to elasticsearch. On Kibana I see under the Indices bank and on visualize all the fields (age, balance, etc.) on the left but cannot see charts on the right. It always says No Results Found. On the left I selected Y-Axis - Agg as Average and Field as Balance and X-Axis Agg as Terms and Field as Age, all from drop downs. Then I used the APPLY action. No error is displayed, but neither is a chart.
What am I doing wrong here? Using curl I can check the elastic search and see that the data exists. No matter what I do, I do not see any logging in the elasticsearch logs under /usr/share/elasticsearch/logs.
I don't know if you figured this out or not but I had the exact same problem. I was using a mapping type that did not contain the field I was trying to visualize. Make sure the _type field contains a mapping with the correct indexed fields.
When setting up my initial index patterns, I glossed over specifying the 'Default timestamp' field, and the UI didn't insist on having one. The fix was to delete my index patterns and re-create them them, with the specified timestamp field. Everything snapped into place.
I am using custom variables to track order ids. In order to aggregate analytics data into out data warehouse, I want to select a number of metrics with the custom variable as a dimension. However, if I do so, I will not get the entries where the variable is not set (E.g. sessions that didn't result in a sale). I need to get these as well.
Can I write a filter or segment that selects only the entries that doesn't have a particular custom variable? I have tried:
segment=dynamic::ga:customVarValue1==
But that doesn't seem to work (It gives no results back).
Basically I'm looking for the equivalent to where ga:customVarValue1 is null in sql.
In short, it's not possible to get the nullset data, as explained by a Google rep:
For some dimensions, GA uses the default value of (not set).
Custom Variable do not have a default value, so if a hit does not have a
custom variable associated with it, all the other dimensions in the query
are not added to the reports.
The original answer is a little confusing, but when you read between the lines it suggests that they throw out these "empty" values when they run their aggregates.
The "correct" approach, as he explains, is to set a default value for any row you want reported:
If you need to see the (not set) value, you could try sending a default
value for custom variables.
For example if you use visitor level custom vars to track member vs
non-member, you should always set non-member as a default for everybody;
then modify to member once they register.
Details are here: http://groups.google.com/group/google-analytics-data-export-api/browse_thread/thread/cd078ddb26ca18d5?pli=1
I've just had some success solving this by using a Regex to capture users or sessions where the custom dimension has no value. In my case I want to separate logged in and logged out users.
The Regex .+ will capture any non-empty value, so can be used to get the job done.
The filters for my Returning users segment is matches regex: .+ like this:
For the Customer prospects I used: does not match regex: .+ like this:
It's early days, but this appears to be working: