When exporting, for instance to Power BI, Azure Data Explorer time series, I receive an strange structure. What's the best practice to export these data? (For example, exporting a time series prediction).
Adding details:
This is my time series chart:
And this is the time series in table format:
This table format is exactly what I get when importing from Power BI (from Kusto Explorer -> Share -> Power BI):
As Yoni L said, using mv-expand solved the issue:
Related
Grafana 6.x/7.x versions, especially version 7.x
If I have the following data in an Elasticsearch Index (<--Grafana: which it refers) or flat file (csv)
Sprint Velocity: 10 20 15 35 22
--------------------------------------
Sprint Name: S1 S2 S3 S4 S5
Sprint Number: 1 2 3 4 5
Then, in Grafana, is it possible to show Velocity field on Y axis and either Sprint Name/Number on X axis (which are non-date-time based fields).
In ELK (Kibana GUI), I have a visualization which points to an Elasticsearch Index via Index-Pattern (for the above data) and I can easily render/show a LINE visualization having X axis (Terms: using Sprint Name or Sprint Number).
I'm trying to achieve the same in Grafana but having hard time.
https://grafana.com/grafana/plugins/natel-plotly-panel didn't help either as it's supported only up to version Grafana 6.
Also, if I use the "series" option, it doesn't allow the points to be connected in the graph since Grafana thinks they all separate series. For ex: If I use this approach: https://github.com/gipong/grafana-groupedbarchart-panel I can't get one metric (Velocity) on Y axis as LINE chart and another metric (TotalSprintPoints) on Y axis as a BAR chart.
#AKS your example is really information did you tired with line chart ? I can follow your steps for bar chart etc when I move to line its not working. Also I like to show 4 different line in the chart but I am getting only one.
In Grafana, there are few possibilities (but it requires some leg work). A user can use MySQL (as the data source) to visualize non-time series data on X axis. One can use PostgreSQL as well. For more info, read the blog below. Thanks for Sean Bradley.
https://medium.com/grafana-tutorials/graphing-non-time-series-sql-data-in-grafana-8a0ea8c55ee3
Grafana can graph Time Series data from many different types of data sources extremely well. But sometimes you just want to graph, simple non time series data. i.e. Data without timestamps, flat tables with regularly updated statistics or just simple lookup tables.
Example Non Time Series data as a flat table.
And you want to see this simple data, as graphs similar to below.
Grafana needs Time Series data to create graphs, so this is not immediately possible from Non Time Series data, but there is a way, and it's shown below, how you can do it.
To test this yourself, you should have a MySQL database and create a Data Source connection to it in Grafana. If you don’t have a MySQL database, but you do have a Grafana Server, then you can follow the instructions at https://sbcode.net/grafana/create-mysql-data-source/ to quickly install yourself a MySQL server and configure a MySQL data source inside Grafana.
Now to organize a simple flat table with some non time series data it.
On my MySQL server, we can have a schema named exampledb. In it let's create a table called flat_table_example
Open a MySQL prompt on your server, and create a table with this script
CREATE TABLE `exampledb`.`flat_table_example` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`username` varchar(45) DEFAULT NULL,
`total` decimal(10,0) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=10 DEFAULT CHARSET=latin1;
Now, let's insert some data:
INSERT INTO `exampledb`.`flat_table_example`
(`username`,
`total`)
VALUES
('Cat',56),
('Dog',5),
('Lizard',4),
('Crocodile',2),
('Koala',50),
('Cassowary',2),
('Peacock',1),
('Emu',1),
('Kangaroo',1);
To check if the data exists, run the following:
SELECT * FROM ``exampledb`.`flat_table_example`;
**Now**, Open up your **Grafana UI**, **ensure** your **MySQL Data Source** has been **configured and connects**, then go to the Explore Tab.
Choose your MySQL data source, press the Edit SQL button
Replace the default SQL with this below,
SELECT username AS metric, total as value FROM flat_table_example ORDER BY id
And select Format As = Table to see your data as a table inside Grafana.
Now, this data is not Time Series data, it has no timestamp columns, so Grafana will not know how to graph it. So here is the trick,
Modify the SQL statement to be,
SELECT NOW() AS "time", username AS metric, total as value FROM flat_table_example ORDER BY id
And now the data has been converted to Time Series. Now, all rows have time stamps, are identical and update to show now every time the query is executed. You may think this is wrong, but it’s not, it’s perfect.
Grafana can now graph this data as a pretty graph.
Leave the Explore page now, go to Manage Dashboards, create a New Dashboard and Add Query
Select your MySQL data source, press the Edit SQL button just like before, paste the new SQL script into the text area. But this time, leave Format As = Time Series instead of Table
Now go to visualisation, select Bar Gauge. Straight away, you have this below.
You can select Orientation = Horizontal, and Field Unit = Percent (0-100)
You can select the basic Gauge visualisation to get this result below, and add yourself an extra Threshold
And even select the default Graph visualisation, but ensure you also select X-Axis Mode = Series
Video Tutorial
To see this video tutorial on Viewing Non Time Series Data in Grafana then visit,
https://youtu.be/Q6aw9oInsyw
Special Thanks to Sean B.
One other way is mentioned here for grouped charts:
https://github.com/gipong/grafana-groupedbarchart-panel
I'm using R package "cluster" in Power BI to cluster customers from their historical transactions.
My data looks like this:
Using Run R-Script command in Power Query Editor, I have the following code:
library(cluster)
k.means.fit_log <- kmeans(dataset[2:4], 3)
output<-dataset
output$cluster <- k.means.fit_log$cluster
After this code is executed, I get an additional column with the number of the cluster and all looks good.
However, k.means.fit_log is a list with 9 rows, that contains the cluster centers, size, etc. so I'd like to be able to create another table or tables in Power BI with the contents of that object.
How can I achieve this?
Thanks in advance for your help!
If it is a list, and delimited, you can use the spilt function in Power Query. It would be good to share the R outputed table in your question
I am trying to integrate Time series Model of R in Tableau and I am new to integration. Please help me in resolving below mentioned Error. Below is my code in tableau for integration with R. Calculation is Valid bur getting an error.
SCRIPT_REAL(
"library(forecast);
cln_count_ts <- ts(.arg1,frequency = 7);
arima.fit <- auto.arima(log10(cln_count_ts));
forecast_ts <- forecast(arima.fit, h =10);",
SUM([Count]))
Error : Error in auto.arima(log10(cln_count_ts)) : No suitable ARIMA model found
When Tableau calls R, Python, or another tool, it does so as a "table calc". That means it sends the external system one or more vectors as arguments and expects a single vector in response.
Depending on your data and calculation, you may want to send all your data to R in a single call, passing a very large vector, or call it several times with different vectors - say forecasting each region separately. Or even call R multiple times with many vectors of size one (aka scalars).
So with table calcs, you have other decisions to make beyond just choosing the function to invoke. Chiefly, you have to decide how to partition your data for analysis. And in some cases, you also need to determine the order that the data appears in the vectors you send to R - say if the order implies a time series.
The Tableau terms for specifying how to divide and order data for table calculations are "partitioning and addressing". See the section on that topic in the online help. You can change those settings by using the "Edit Table Calc" menu item.
Tearing my hair out on this one. Took me hours just to get rJava up and running (because mac OS X el capitan was not wanting to play nice with Java) in order to load excel-specific data importing packages etc. But in the end this hasn't helped my problem, and I'm just about at my wits end. Please help.
Basic situation is this:
Have simple excel data of time durations, over a span of a couple of years. So the two columns I'm importing are the time(duration) and year(2016,2017 etc).
In Excel the data is formatted as [h]:mm:ss so it displays correctly (data is related to number of hours worked in a month, so typically something like 80:xx:xx ~ 120:xx:xx). I'm aware that in excel, despite the cells being formatted as above, and only showing the relevant period of hours, that in reality excel has appended an (irrelevant, arbitrary) date to this hours data. I have searched and searched and found no way around this limitation in the way excel handles dates/times/durations.
I import this data into R via the "import data -> import from excel data set" menu item in R commander GUI, not the console.
However when importing the data into R, the data displays as a single number e.g. approx. 110 hrs is converted to 4.xxxxx, not as hh:mm:ss. So when running analyses and generating plots etc, instead of the actual (meaningful) 110:xx:xx type data being displayed, a completely meaningless 4.xxxxxx is displayed.
If I change the formatting of the excel cells to display the date as well as the time rather than use the [h]:mm:ss cell formatting, R erroneously interprets the data to something equally useless, like 1901/02/04 05:23 am
I have installed and loaded a variety of packages such as xlsx, XLConnect, lubridate etc but it hasn't made any difference to how R interprets the excel data on import, from the GUI at least.
Please tell me how do I either
a) edit the raw data to a format that R will understand as a time duration (and nothing but a time duration) in hh:mm:ss format, or
b) format the current data from within R after importing, so that it displays the data in the correct way rather than a useless number or arbitrary date/time?
[Please note: I can use the console, when given the commands etc needed to be executed. But I need to find a solution that ultimately will allow the data to be imported and/or manipulated from within the GUI, not from typing a bunch of commands into the console, as the end user (not me) has zero programming ability and cannot use a console, and will only ever be using R via the GUI.]
Is your code importing the data from excel as seconds?
library(lubridate)
duration <- lubridate::as.duration(400000)
as.numeric(duration, "hours")
111.1111
as.numeric(duration, "days")
4.62963
seconds_to_period(400000)
"4d 15H 6M 40S"
I have a bunch of sales opportunities in various excel files- broken down by region, type, etc.- that are one column each and simply list the dollar amounts of each opportunity. In R I have run a simulation to determine the likelihood of each opportunity closing with a sale or not, and repeated the simulation 100,000 times. I know that I can't pass the full results table back to Tableau because it has 100,000 rows- one total for each simulation- and the data I'm pulling into Tableau would just have the $ value of each opportunity so would only have a length of the number of opportunities of that type.
What I have in R is basically this first block of code; repeated a number of times with varying inputs and changing probabilities; then ultimately combine the totals vectors to get a quarter total vector.
APN<-ncol(APACPipelineNew)
APNSales<-matrix(rbinom(APN, 1, 0.033), 100000, APN)
APNSales<-sweep(APNSales,2,APACPipelineNew,'*')
APNTotals<-rowSums(APNSales)
...
Q1APACN<-APNTotals+ABNTotals+AFNTotals
...
Q1Total<-Q1APACT+Q1EMEAT+Q1NAMT
What I'd like to do is set this up as a dashboard in Tableau so that it can automatically update each week, but I'm not sure how to pass the simulation back into Tableau given the difference in length of the data.
Some suggestions:
For R you can use a windows scheduler to run a job at any given interval (or use the package taskscheduleR).
After you save the R data you can manually update your dashboard if it is on a desktop version (I do not know if you can schedule an extract refresh with a desktop dashboard).
However, if your dashboard lives on a tableau server you can schedule an extract refresh every week. Obviously, I would schedule the r update before the tableau extract refresh.
If you only wanted the data to update if there was a differing number of rows from the previous weekly run you can build that logic into R. Although saving the r data and refreshing the extract with the same data and number of rows should not cause any problems.