Hello I have an issue of matching two different datetime columns.
I need to compare the two of them (and their data), but at the moment of putting them in the same table (using a datetime relation) I do not get the match I need:
What I need:
| Datetime_1 | Datetime_2 |
| ---------- | ---------- |
| 01/01/2023 08:00:00 AM | |
... ...
| 01/11/2023 12:00:00 AM | 01/11/2023 12:00:00 AM |
| 01/11/2023 01:00:00 AM | 01/11/2023 01:00:00 AM |
... ...
| 31/01/2023 12:00:00 PM | 31/01/2023 12:00:00 PM |
What I get:
Datetime_1 goes from 01/01/2023 12:00:00AM to 01/31/2023 11:00:00PM (with steps of 1h) and Datetime_2 goes from 01/11/2023 8:15:00 PM to 02/06/2023 7:45:00 PM (with steps of 30min).
I did a relation with the two of them and I didn't receive any error:
I already put both lists in Date/Time format in Power Query and Data panel.
However, I noticed my main datetime list doesn't have the hierarchy icon on fields panel, while the secondary datetime lists have it, (but not the hour section):
Also, as I mentioned before, my list have a range between Jan and Feb. I do not understand why this range continues and match some dates on the on my main datetime list:
Troubleshooting
Part of the difficulty troubleshooting this is the two columns are formatted differently. Just for now, make sure both are formatted as Long Date Time. When comparing the relationship, do not drag the hierarchy (for the one that has it) into the table but rather just the date itself. When you do, you will see the full timestamp for both columns and the issue will become more clear.
Power BI & Relationships on DateTime
Power BI will only match related rows if the date and time match exactly, so 4/15/2023 12:00:00 AM will not match 4/15/2023/12:00:01 AM. You mentioned one side of the relationship has 30 minute steps while the other has 1 hour steps. Power BI is not going to match up a 1:30am and 1:00am value for you. If you want that 1:30 value to match up to 1:00, create another column truncating the :30 minutes and build your relationship on the truncated column.
Time Dimension
I'm not sure of your application so don't know if this will work, but when dealing with time, I try to separate Date and Time into separate columns and have both a Date and Time dimension. Below is my time dimension DAX. You can generate any minute-precise interval with it. Notice the last defined column "timekey". I create a column in my fact table to relate to this key.
DimTime =
var every_n_minutes = 15 /* between 0 and 60; remainders in last hourly slice */
/* DO NOT CHANGE BELOW THIS LINE */
var slice_per_hour = trunc(DIVIDE(60,every_n_minutes),0)
var rtn =
ADDCOLUMNS(
SELECTCOLUMNS(
GENERATESERIES(0, 24*slice_per_hour - 1, 1),
"hour24", TRUNC(DIVIDE([Value],slice_per_hour),0),
"mins", MOD([Value],slice_per_hour) * every_n_minutes
),
"hour12", MOD([hour24] + 11,12) + 1,
"asTime", TIME([hour24],[mins],0),
"timekey", [hour24] * 100 + [mins]
)
return rtn
As requested, turning this into an answer. The reason you're getting these results is that your time stamps will never line up. Yes, it let you create the join, but my guess is that is only because both fields have the same formatting. Also, it is best practices to separate your dates and time in separate date and time dimensions, then join them via a fact table. See also here.
Related
It seems MySql 8 differenciates TIMESTAMP from DATETIME more than previous versions (at least more than 5.7) and java drivers understand a Timestamp is universal (senconds since epoch) while DATETIME is like a textual representation of date and time without timezone information, it is not an instant in time.
I need to work with timestamps but I need to add 1 second to one of them in a SELECT query. Unfortunately I can't do it. If I try to do it then it gets converted to DATETIME with it's side effects, as they are not the same.
This code can be using for testing without other programming language:
CREATE TABLE z (
`id` bigint unsigned NOT NULL PRIMARY KEY AUTO_INCREMENT,
`myTs` timestamp NULL DEFAULT NULL
) ENGINE=InnoDB;
insert into z (myTs) values (TIMESTAMP'2023-01-01 00:00:00+00:00');
create table x as select myTs + interval 1 second from z limit 1;
desc x;
drop table x;
drop table z;
You will get myTs + interval 1 second | datetime | ...
But if you remove + interval 1 second then you get myTs | timestamp | ...
I have read the documentation and tried different functions and I couldn't avoid or reverse this conversion.
Specifically I tried at least timestamp(myTs + interval 1 second), timestampadd(second, 1, myTs), date_add(myTs, interval 1 second) and timestamp(myTs + 1).
I need to either sum 1 second to a timestamp without it being converted to datetime or a way to convert a datetime to a timestamp type (providing a time zone if needed) in MySql 8.
I'm using MySql 8.0.30.
Thank you.
For a 2 minute time window this vector has the following results (I am using Grafana Explore with a picked 2 minute time):
instana_metrics{aggregation="max", endpoint="mutation addProduct"}
t1 - 3051
t2 - 5347
t3 - 5347
t4 - 4224
t5 - 4224
I need something equivalent to
SELECT Count(*)
FROM instana_metrics
with a result of 5.
The best I was able to come up with is this
count( instana_metrics{aggregation="max", endpoint="mutation addProduct"} )
t1 | 1
t2 | 1
t3 | 1
t4 | 1
t5 | 1
My interpretation is that every point in time has a count of 1 sample value. But the result itself is a time series, and I am expecting one scalar.
Btw: I understand that I can use Grafana transformation for this, but unfortunately I need a PromQL only solution.
Just use count_over_time function. For example, the following query returns the number of raw samples over the last 2 minutes per each time series with the name instana_metrics:
count_over_time(instana_metrics[2m])
Note that Prometheus calculates the provided query independently per each point on the graph, e.g. each value on the graph shows the number of raw samples per each matching time series over 2 minutes lookbehind window ending at the given point.
If you need just a single value per each matching series over the selected time range in Grafana, then use the following instant query:
count_over_time(instana_metrics[$__range])
See these docs about $__range variable.
See these docs about instant query type in Grafana.
I have been able to read the blogs and understand somewhat how the calculations are taking place, but need further clarity.
Case: Ingestion_Time_Logged will output the CreationTime event on the 7th min or the 37th min regardless of what time the actual CreationTime event occurred. Example:
If CreationTime Event occured at "2021-03-06 07:38:59.000" Then the Ingestion_Time_Logged will be the closest 7th min or 37th min. In this case the Ingestion_Time_Logged will be
"2021-03-06 08:07:59.000". The code works fine. I am just trying to understand how it is calculating it.
What I understand so far:
% is the modulo operator which is a way to determine the remainder of a division operation. Instead of returning the result of the division, the modulo operation returns the whole number remainder.
latestComposed_min%30 will divide the minutes by 30 and result in the reminder. I used it to find if the minute is between 0 - 7 or 30 - 37. It is the same as below but much easier and efficient.
if(latestComposed_min < 7 OR (latestComposed_min>30 AND latestComposed_min < 37))
Excluded 37 and 7 to keep them as they are since they are already ok, if we do not exclude them, they will be added 37 minutes. Since both are NOT comparisons AND should be used between them otherwise with OR the result will be always true (which is wrong)
The piece I am struggling with:
| eval Ingestion_Time_Logged=strftime(case(latestCreated_min%30 < 7, CreationTime_epoch-CreationTime_epoch%1800+420+latestCreated_sec, latestCreated_min!=37 AND latestCreated_min!=7, CreationTime_epoch-CreationTime_epoch%1800+2220+latestCreated_sec,1=1,CreationTime_epoch),"%Y-%m-%d %H:%M:%S.%6N")
Full Code:
| makeresults
| eval CreationTime="2021-03-06 07:38:59.000"
| eval CreationTime_epoch=strptime(CreationTime, "%Y-%m-%d %H:%M:%S.%6N")
| eval latestCreated_hour=tonumber(strftime(CreationTime_epoch, "%H"))
| eval latestCreated_min=tonumber(strftime(CreationTime_epoch, "%M"))
| eval latestCreated_sec=round(CreationTime_epoch%60,6)
| eval Ingestion_Time_Logged=strftime(case(latestCreated_min%30 < 7, CreationTime_epoch-CreationTime_epoch%1800+420+latestCreated_sec, latestCreated_min!=37 AND latestCreated_min!=7, CreationTime_epoch-CreationTime_epoch%1800+2220+latestCreated_sec,1=1,CreationTime_epoch),"%Y-%m-%d %H:%M:%S.%6N")
| table Ingestion_Time_Logged, CreationTime, CreationTime_epoch, latestCreated_hour, latestCreated_min
Let's break it down
Here, we're setting the value of the Ingestion_Time_logged field to the result of the strftime function. That is, we're converting a epoch time into a string.
| eval Ingestion_Time_Logged=strftime(
The epoch to convert is determined by a case statement.
case(
If the created minute (38 in the example) is 0-6 or 30-36
latestCreated_min%30 < 7,
then round down. The %1800 is the same as %30 above, only in seconds rather than minutes. Subtracting that from the epoch time shifts the epoch time to the top or bottom of the hour. The 420 adds in 7 minutes then we add in seconds.
CreationTime_epoch-CreationTime_epoch%1800+420+latestCreated_sec,
If the created minute is neither 7 nor 37 (our case)
latestCreated_min!=37 AND latestCreated_min!=7,
then we do the same as above, except using 37 minutes, advancing the time to the next :07 or :37.
CreationTime_epoch-CreationTime_epoch%1800+2220+latestCreated_sec,
This is the default (catch-all) case, which takes the CreationTime_epoch field as-is.
1=1,CreationTime_epoch),
This is how the timestamp string will be formatted.
"%Y-%m-%d %H:%M:%S.%6N")
I have two cross tables on a single page.
The first cross table is a summary that has Components on the horizontal axis, and Facilities on the vertical axis. The cell values shows colors "RED", "YELLOW", or "NA". The second cross table is a drilldown of the marked row on the summary table, with the horizontal axis Components and Type on the vertical axis. The cell values are a count function.
What I need is to have the color of what I marked show below each component in the drilldown.
Summary
+----------+--------+-------+--------+
| Facility | COMP1 | COMP2 | COMP3 |
+----------+--------+-------+--------+
| FAC1 | NA | RED | RED |
| FAC2 | YELLOW | NA | RED |
| FAC3 | RED | RED | YELLOW |
+----------+--------+-------+--------+
Drilldown (If I mark the FAC2 row)
+-------+--------+-------+
| Type | COMP1 | COMP3 |
+ + YELLOW + RED +
|-------|--------|-------|
| TYPE1 | 12 | |
| TYPE2 | 11 | 4 |
+-------+--------+-------+
Does anyone know if this is possible with cross tables? Any tips on how to do it? I appreciate the help.
Thanks,
John
Edit: I'm doing this to go around not being able to color column headers of a cross table, so if anyone has an alternative, I would appreciate it.
Currently using Spotfire 7.11
Okay. Bear with me here as I have hacked together a solution. I will say, I made some assumptions about your data structure. Depending on the structure of your data, the answer may need slightly modified.
Here is the structure of my data:
Step 1: Create two document properties to hold the values of the title. I created two document properties named "tableTitle1" and "tableTitle2" (one for each column in the details cross table). Create one document property to hold a DateTime value that an r script will pass us (will discuss later). I named mine "time".
Step 2: Create the cross tables as you have them. Ensure the first cross table is using Marking "Marking" and the second is limited by the marking "Marking". In the second cross table, ensure that the titles look something like this: Count([Comp1]) as [Comp1 ${tableTitle1}], Count([Comp3]) as [Comp2 ${tableTitle2}]. You need to use the document properties created in Step 1.
Step 3: Create the python script. The code is as follows:
from System.Collections.Generic import List
from Spotfire.Dxp.Data import *
# Create a cursor for the table column to get the values from.
# Add a reference to the data table in the script.
dataTable = Document.Data.Tables["SOTest"]
cursor = DataValueCursor.CreateFormatted(dataTable.Columns["Comp1"])
# Retrieve the marking selection
markings = Document.Data.Markings["Marking"].GetSelection(dataTable).AsIndexSet()
# Create a List object to store the retrieved data marking selection
markedata = List [str]();
# Iterate through the data table rows to retrieve the marked rows
for row in dataTable.GetRows(markings, cursor):
value = cursor.CurrentValue
if value <> str.Empty:
markedata.Add(value)
# Get only unique values
valData = List [str](set(markedata))
# Store in a document property
Document.Properties["tableTitle1"] = ', '.join(valData)
####DO IT AGAIN FOR THE SECOND COLUMN#####
# Create a cursor for the table column to get the values from.
# Add a reference to the data table in the script.
cursor = DataValueCursor.CreateFormatted(dataTable.Columns["Comp2"])
# Create a List object to store the retrieved data marking selection
markedata = List [str]();
# Iterate through the data table rows to retrieve the marked rows
for row in dataTable.GetRows(markings, cursor):
value = cursor.CurrentValue
if value <> str.Empty:
markedata.Add(value)
# Get only unique values
valData = List [str](set(markedata))
# Store in a document property
Document.Properties["tableTitle2"] = ', '.join(valData)
Step 4: Create an R Script to kick off the python script when data is marked. This is going to be a very simple R Script. The code is as follows:
markedTable <- inputTable
time <- Sys.time()
The check box for allow caching should be unchecked. The output parameter time should go to the document property time. the input parameter inputTable should be your datatable, all columns, and should be limited by Marking. Ensure that the refresh function automatically checkbox is checked.
Step 5: Map the python script to the time document property. In the Edit > Document Properties dialogue box, under Properties, assign the python script we created to the document property. The R script will change the current datetime each time the marking on the table changes, thus running our python script for us.
Step 6: Watch the magic happen.
Here's my question.
I have a table (tblCourses) that has three columns: Course, StartDate, and EndDate
I want to create a query that gives me all the dates that are between StartDate and EndDate.
So if I have
Course: A
StartTime: 1/10/15
EndTime: 1/15/15
I want the query to show:
Course; NewDate
A _ 1/10/15
A _ 1/11/15
A _ 1/12/15
A _ 1/13/15
A _ 1/14/15
A _ 1/15/15
I've done a lot of google searching, and I've looked through stackoverlow forums, but I can't seem to find a solution. Any help would be GREATLY appreciated!
As Yermo states in his answer, one approach would be to use a [Dates] table that contains every valid date covering a period of time. Another approach would be to use a [Numbers] table containing integer values from zero to (some sufficiently large number), e.g.,
n
--
1
2
3
...
19
20
Once that table has been created you could use a query like
SELECT tblCourses.Course, DateAdd("d", Numbers.n, tblCourses.StartDate) AS NewDate
FROM tblCourses, Numbers
WHERE DateAdd("d", Numbers.n, tblCourses.StartDate) <= tblCourses.EndDate
which would return
Course NewDate
------ ----------
A 2015-01-10
A 2015-01-11
A 2015-01-12
A 2015-01-13
A 2015-01-14
A 2015-01-15
This is actually better solved in code than in the SQL query.
The reason it's not straight forward is you are trying to generate rows in a result set that are not in the table.
As a quick hack you could create another of dates and fill it to cover a few years. Then you could do a straight join which would get you the result you are looking for.