As a user of a logstash + Open Search setup I would like to create a dashboard that visualizes, in a simplified way, that certain log entries have occurred.
The scenario is that we have a business process that spans over different components, all of which produces log outputs.
Let's say that this process consists of these steps:
Component A notifies Component B that there is a data Set D1 to be fetched.
This generates log entries L1a and L1b.
Component B then fetches D1 from Component A.
This generates log entries L2a and L2b.
I have no problem picking out these log entries in the "discover" view. But I would like to visualize it in a simpler way, for non technical viewers. For example like labels on a timeline, where each each label corresponds to L1a, L1b, L2a and L2b.
I have sat with the visualization feature for a while but not found anything that suits this need.
If you could point my in the right direction to achieve this, it would be much appreciated.
Related
I'm logging some custom metrics in Application insights using the TelemetryClient.TrackMetric method in .NET, and I've noticed that occasionally some of the events are duplicated when I view them in the Azure portal.
I've drilled into the data, and the duplicate events have the same itemId and timestamp, but if I show the ingestion time by adding | extend ingestionTime = ingestion_time() to the query then I can see that the ingestion times are different.
This GitHub issue indicates that this behavior is expected, as AI uses at-least-once delivery.
I plot these metrics in charts in the Azure portal using a sum aggregation, however these duplicates are creating trust issues with the charts as the duplicates are simply treated as two separate events.
Is there a way to de-dupe the events based on itemId before plotting the data in the Azure portal?
Update
A more specific example:
I'm running an algorithm, triggered by an event, which results in a reward. The algorithm may be triggered several dozen times a day, and the reward is a positive or negative floating point value. It logs the reward each time to Application Insights as a custom metric (called say custom-reward), along with some additional properties for data splitting.
In the Azure portal I'm creating a simple chart by going to Application Insights -> Metrics and customising the chart. I select my custom-reward metric in the Metric dropdown, and select Sum as the aggregation. I may or may not apply splitting. I save the chart to my dashboard.
This simple chart gives me a nice way of monitoring the system to make sure nothing unexpected is happening, and the Sum value in the bottom left of the chart allows me to quickly see whether the sum of the rewards is positive or negative over the chart's range, and by how much.
However, on occasion I've been surprised by the result (say over the last 12 hours the sum of the rewards was surprisingly negative), and on closer inspection I discovered that a few large negative results have been duplicated. Further investigation shows this has been happening with other events, but with smaller results I tend not to notice.
I'm not that familiar with the advanced querying bit of Application Insights, I actually just used it for the first time today to dig into the events. But it does sound like there might be something I can do there to create a query that I can then plot, with the results deduped?
Update 2
I've managed to make progress with this thanks to the tips by #JohnGardner, so I'll mark that as the answer. I've deduped and plotted the results by adding the following line to the query:
| summarize timestamp=any(timestamp), value=any(value), name=any(name), customDimensions=any(customDimensions) by itemId
Update 3
Adding the following line to the query allowed me to split on custom data (in this case splitting by algorithm ID):
| extend algorithmId = tostring(customDimensions.["algorithm-id"])
With that line added, when you select "Chart" in the query results, algorithmId now shows up as an option in the split dropdown. After that you can click "Pin to dashboard". You lose the handy "sum over the time period" indicator in the bottom left of the chart which you get via the simple "Metrics" chart, however I'm sure I'll be able to recreate that in other ways.
if you are doing your own queries, you would generally be using something like summarize or makeseries to do this deduping for a chart. you wouldn't generally plot individual items unless you are looking at a very small time range?
so instead of something like
summarize count() ...
you could do
summarize dcount(itemId) ...
or you might add a "fake" summarize to a query that didn't need it before with by itemId to coalesce multiple rows into just one, using any(x) to grab any individual row's value for each column for each itemId.
but it really depends on what you are doing in your specific query. if you were using something like sum(itemCount) to also deal with sampling, you have other odd cases now, where the at-least-once delivery might have duplicated sampled items? (updating your question to add a specific query and hypothetical result would possibly lead to a more specific answer).
I want to calculate and display the average scroll depth in Data Studio from analytics.
I’m looking to get an average scroll depth in Studio. I’ve got the 10%,25%, etc scroll depth data coming in, but I now need to be able to calculate the average scroll % from this data.
To calculate the average scroll depth:
multiply the scrolled threshold by the number of events (10x500) + (20x400) + (30x475) +(40x300) + (50x200) + (60x100) +(70x75) +(80x60) + (90x20) + (100x10)
Then, take that total divided by the total number of events. 500 + 400 + 475... etc
Because I can’t reference cells in Studio I can’t get it to work. I’ve also tried Google Sheets, which does work to do the calculation, but then I can’t use Data Studios filter to provide a specific page path?
I'm thinking that perhaps the calculation will need to be done at data source, but I am not sure how to reference a 'cell'?
Data Studio doesn't work based on a concept of "cells", it works based on a concept of "fields"—which are basically properties of the data source. Similarly, you don't have "formulas" per se, but rather "calculated fields". These fields can be created either at the chart-level (single-use, but doesn't require permissions to modify the data source) or in the data source (reusable across many charts, requires permissions to modify the data source). Most fields also have an aggregation type, which tells the report how to aggregate it in charts by default (e.g. Sum or Average).
When you either edit your data source and hit "Add Field" or the option with the same name under the "Add metric" or "Add dimension" menus on a chart, you'll be presented with a box to input the formula. To access a field, just type its name (of if you're in the data source, select it from the list on the left). The editor will also typically give you an auto-complete list below your cursor based on what you're typing. Once your entry matches a field, it will get a highlight box around it (the color is based on the type; green = dimension/string,blue = metric/number). The functions available are sort of a mash-up of something between what you'd expect in Google Sheets and in a SQL query, but with more constraints on when you're allowed to use certain functions.
The documentation for calculated fields is pretty simple, so I'd recommend starting there before you try to do too much heavy-lifting in Data Studio. Because of constraints in Data Studio's data model, you'll often find that you need to create separate calculated fields for different parts of the formula, and then combine them in a new calculated field. I'll warn you that the error messages in the field editor aren't super helpful sometimes, so you may need to re-read the documentation for the functions and field types you're working with to ensure you get a valid result.
If you're running into problems, including the field names and values that you need in your calculation may help, including the source of the data (are these GA events?). The more details you give, including what you've already tried, the more helpful we can be. Also, make sure to read the docs first to make sure you have a good handle on the product you're using and the terminology the community is most likely to understand.
I'm trying to understand the workflow and how to think when working with Power BI.
I'm trying to create the visual side of a logging framework using azure application insights and log analytics in the background.
What I got stuck on right now is using a visualization for different table structures.
Scenario:
So lets say I have an integration workflow that is compiled of different components (starts at one endpoint, being processed and sent to destination endpoint), and this workflow is used in parallel depending on the endpoints.
Visualization:
A ----> (integration process) -----> B
C ----> (integration process) -----> D
A <---- (integration process) <----- B
C <---- (integration process) <----- D
In this process I'm logging traces and exceptions for each workflow for example.
And lets say I have extracted the power BI query from log analytics to import the data into Power BI.
Now as you can see above I have an "Send" and "Receive" flow for both sets of endpoints per parallel integration so to speak and I want to be able to pair these per report in Power BI.
Problem:
So then to my problem for now;
Let's say I want to put a doughnut chart with total amount sent AND received messages.
Also when clicking on the different parts of the chart I want to change the containing data and also the structure of the single table in the same report.
What I'm struggling with is using a table to display the content of two workflows that are related to each other in the sense of being a "Sent and received" visualization of the integration, but don't have any hard drawn relations to each other in code.
I was reading something about a switch statement but I thought i'd ask the question here to learn "the way of thinking" when using Power BI.
Hope I made my problem clear by the description above, if I forgot anything please ask and I'll do my best to try to further explain what I'm asking for.
EDIT:
Ok so regarding the feedback i got of the question being to broad, does it help i i would say i want to be able to switch everything in a Table (that is: all columns and values), by a clickable slicer for example.
why i'm trying to achieve is something similar to this, but for a table view:
Youtube Link:
Using SWITCH True Logic In Power BI - DAX Concepts
Do i select the content of a table with a measure-query or do i have to do any other special fix to get this to work?
Hope this clarify the problem i'm working on.
If you are looking to switch between two entirely different tables, I'd recommend using buttons to toggle between different bookmarks which show/hide different visuals.
Check out the documentation for bookmarks and buttons for more detailed information and come back if you get stuck and have more specific questions.
I've got metrics in Graphite showing response time for various organizations. The list of organizations can change on the fly. I want panels in Grafana to appear for any origanization who's response time is over a certain threshold. Was thinking the Singlestat panels was the right panel to use. Question is how to make them appear dynamically? Is a scripted dashboard the right approach?
If a scripted dashboard is the correct solution, can anyone recommend a Grafana cloud/service provider that supports scripted dashboards? The current one I have been testing out does not support scripts. Note that I am not really tied to Graphite as the backend since this project is in proof of concept phase. Just need the backend to also be a service. Don't want to roll the backend myself. Thanks.
As far as I know, it is not possible right now.
We had a similar use case in my organisation, and here is what we did.
You can define a template variable for your organizations, and then use SingleStat panel with “Repeat Panel” on this variable, but that will display panels for all of your organizations. Filtering based on a criteria is a requested feature.
Alternatively, you can use the Table panel for your use case.
Choose Table panel
In “Metrics”, enter your metric organizations.*.response_time (or whatever more complicated you need, applyByNode can be handy for such cases)
In “Options”
“To Table Transform”: choose “Time Series aggregations”
“Columns”: Avg, or Current (depending on your needs)
“Coloring”: use thresholds to paint in red or something anything above your desired response-time threshold.
Sort the Table per the Number column.
Ta-da! Your organisations needing attention will be at the top of the table and highlighted.
In the lack of true filtering, this worked for us. Hope it will work for you too :)
I have a scenario as given in the below workflow diagram using core service.
First step is manual step author will create/edit content and finish the activity and it is assigned to everyone.
In the second step, I want to do filter the users based on their groups.
In the third step, couple of scenario
a. If the user belongs to group "A" I want to list out all the next 3 available activities, User A can choose any one of these 3 activities.
b. If the user belongs to group "B", I want to list out "Reviewer" and "Approver" activity as next activity, user b can select any one of these 2.
Can any one give some idea to achieve this?
You should turn your problem around. Yes, it is possible, but the amount of knowledge required will be a huge barrier for you to implement.
So, instead think about it differently.
Change your "Filter Next activity" automated activity to an automatic decision, and have your code that determines the group membership also determine which activity should be next
have 2 forward paths from there - 1 for the users of Group A, other for the users of group B
on each forward path, have different selection options. You may try having links to the same activity from 2 different manual decisions, I don't know if that works though.
Worst case scenario you'll have repeated activities (not in name, but in function), and you saved yourself about 2 months of headaches trying to cope with the learning curve of doing CME extensions.
The simple answer to your initial question is: "no, this functionality is not out of the box".
But of course many things are possible by implementing critical parts yourself.
You'd probably need a GUI extension of some sorts to allow the user to pick from a list.
You'd need to store the information between phases of your custom solution.
And you'd then need some code in an automated workflow activity to implement the picking of the next activity.
Most of these steps have been covered in other questions already. Did you have a look for each of these? If so, you might want to share how far you already got and where you are stuck.