In the R package SPARQL, xsd:date datatypes are by default converted into Unix time. This is a problem because this involves two date transformations - the first taking place within the function SPARQL() - which are determined by the local system time zone. This is a problem if you say, let's say in Sydney, Australia (Sys.timezone() == "Australia/Sydney") because the following query, requesting the date of the 2016 US presidential election
query <- "SELECT ?date WHERE {wd:Q699872 wdt:P585 ?date}"
res <- SPARQL('https://query.wikidata.org/sparql', query)
as.POSIXct(res$results$date, origin = '1970-01-01')
will return "2016-11-07" instead of "2016-11-08" (the correct date), which is instead returned if
Sys.setenv(TZ='GMT')
res <- SPARQL('https://query.wikidata.org/sparql', query)
as.Date(as.POSIXct(res$results$date, origin = '1970-01-01'))
Is there any way to ask SPARQL to return date datatypes as characters?
I'm not sure how the R SPARQL package determines it's a date, but assuming it looks at the assigned datatype, you can coerce to string by retrieving only the lexical value:
SELECT (STR(?date) as ?dateString) ....
Of course this only works if the Unix time conversion happens on the result processing side, not during query evalation. If the latter is the case: get a better SPARQL engine.
Related
I'm trying to convert a datetime that looks like this: 2017-09-19T07:00:00-07:00 into EST, but i keep getting Null values when using the hive built in UTC conversion.
I've tried using a regular expression to parse the date:
date_format(from_unixtime(unix_timestamp(2017-09-19T07:00:00-07:00, "yyyy-MM-dd'T'HH:mm:ss")
- (cast(regexp_extract(regexp_extract(2017-09-19T07:00:00-07:00, '(-[0-9][0-9]:[0-9][0-9])$', 1),'(-[0-9][0-9])',1) as int)*3600) -18000),'YYYY-MM-dd HH:mm')
but that's not good, since there's an hourly difference based on the time of year.
I've also tried:
FROM_UTC_TIMESTAMP(UNIX_TIMESTAMP(2017-09-19T07:00:00-07:00, "yyyy-MM-dd'T'hh:mm:ss:SSS'ZZZZZ'") * 1000, 'EST')
and
FROM_UTC_TIMESTAMP(UNIX_TIMESTAMP(2017-09-19T07:00:00-07:00, "yyyy-MM-dd'T'hh:mm:ss:SSS'Z'") * 1000, 'EST')
but that appears to not work either. What am I doing wrong?
I think that this method needs the date as a string like this:
date_format(from_unixtime(unix_timestamp('2017-09-19T07:00:00-07:00', "yyyy-MM-dd'T'HH:mm:ss")
Normally, the date formats are for strings, not for integers or numbers.
I found the answer on my own by combining the two ways of running the query.
date_format(
FROM_UTC_TIMESTAMP(
(unix_timestamp('2017-09-19T07:00:00-07:00', "yyyy-MM-dd'T'HH:mm:ss")
+ (cast(
regexp_extract(
regexp_extract('2017-09-19T07:00:00-07:00', '(-[0-9][0-9]:[0-9][0-9])$',1),'(-[0-9][0-9])',1) as int)
*-3600)
)*1000 ,'America/New York')
,'YYYY-MM-dd HH:mm:ss')
You are getting NULL because the pattern (format of the date and time) you have provided is not matching with the actual date time value. Correcting the date time format in your query would resolve this issue:
select from_unixtime(UNIX_TIMESTAMP("2017-09-19T07:00:00-07:00", "yyyy-MM-dd'T'HH:mm:ssXXX"), "yyyy-MM-dd HH:mm:ss");
Check out this link to know more about the date time patterns: https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html
I'm using Joi package to validate a timestamp field but How can I set a max() value on it, I want the input timestamp to be less than current time stamp
var schema = Joi.object().keys({
t: Joi.date().timestamp('unix').max(moment().unix()),
})
but I get the error that:
child "t" fails because ["t" must be less than or equal to "Sun Jan 18
1970 07:35:17 GMT+0330 (IRST)"]
I'm sure that the moment().unix() returns the current timestamp, but here it is casted to string.
It seems that max() and min() functions can do the trick but they only work if the threshold is specified in milliseconds.
t: Joi.date().timestamp('unix')
.max(moment().unix() * 1000)
.min(moment().subtract('42', 'weeks').unix() * 1000),
It doesn't look like Joi.date().max() accepts unix timestamps properly despite being able to specify in your schema that a unix timestamp is expected for incoming values.
If you need to use the current date in your schema you can pass the string 'now' instead of using the date. Or just make sure you enter the current date in format that .max() expects. I tried this using milliseconds and it seems to work as expected. I think Joi is using the default Date constructor under the hood to construct dates to compare which expects milliseconds.
var schema = Joi.object().keys({
t: Joi.date().timestamp('unix').max(moment().unix() * 1000)
});
From the docs on date.max()
Notes: 'now' can be passed in lieu of date so as to always compare relatively to the current date, allowing to explicitly ensure a date is either in the past or in the future.
I am reading a big csv (>1GB big for me!). It contains a timestamp field.
I read it (100 rows to start with ) with fread from the excellent data.table package.
ddfr <- fread(input="~/file1.csv",nrows=100, header=T)
Problem 1 (RESOLVED): the timestamp fields (called "ts" and "update"), e.g. "02/12/2014 04:40:00 AM" is converted to string. I convert the fields back to timestamp with lubridate package mdh_hms. Splendid.
ddfr$ts <- data.frame( mdy_hms(ddfr$ts))
Problem 2 (NOT RESOLVED): The timestamp is created with time zone as per POSIXlt.
How do I create in R a timestamp with NO TIME ZONE? is it possible??
Now I use another (new) great package, PivotalR to write the dataframe to PostGreSQL 9.3 using as.db.data.frame. It works as a charm.
x <- as.db.data.frame(ddfr, table.name= "tbl1", conn.id = 1)
Problem 3 (NOT RESOLVED): As the original dataframe timestamp fields had time zones, a table is created with the fields "timestamp with time zone". Ultimately the data needs to be stored in a table with fields configured as "timestamp without time zone".
But in my table in Postgres the data is stored as "2014-02-12 04:40:00.0", where the .0 at the end is the UTC offset. I think I need to have "2014-02-12 04:40:00".
I tried
ALTER TABLE tbl ALTER COLUMN ts type timestamp without time zone;
Then I copied across. While Postgres accepts the ALTER COLUMN command, when I try to copy (using INSERT INTO tbls SELECT ...) I get an error:
"column "ts" is of type timestamp without time zone but expression is of type text
Hint: You will need to rewrite or cast the expression."
Clearly the .0 at the end is not liked (but why then Postgres accepts the ALTER COLUMN? boh!).
I tried to do what the error suggested using CAST in the INSERT INTO query:
INSERT INTO tbl2 SELECT CAST(ts as timestamp without time zone) FROM tbl1
But I get the same error (including the suggestion to use CAST aargh!)
The table directly created by PivotalR (based on the dataframe) has this CREATE script:
CREATE TABLE tbl2
(
businessid integer,
caseno text,
ts timestamp with time zone
)
WITH (
OIDS=FALSE
);
ALTER TABLE tbl1
OWNER TO mydb;
The table I'm inserting into has this CREATE script:
CREATE TABLE tbl1
(
id integer NOT NULL DEFAULT nextval('bus_seq'::regclass),
businessid character varying,
caseno character varying,
ts timestamp without time zone,
updated timestamp without time zone,
CONSTRAINT busid_pkey PRIMARY KEY (id)
)
WITH (
OIDS=FALSE
);
ALTER TABLE tbl1
OWNER TO postgres;
My apologies for the convoluted explanation, but potentially a solution could be found at any step in the chain, so I preferred to put all my steps in one question. I am sure there has to be a simpler method...
I think you're confused about copying data between tables.
INSERT INTO ... SELECT without a column list expects the columns from source and destination to be the same. It doesn't magically match up columns by name, it'll just assign columns from the SELECT to the INSERT from left to right until it runs out of columns, at which point any remaining cols are assumed to be null. So your query:
INSERT INTO tbl2 SELECT ts FROM tbl1;
isn't doing this:
INSERT INTO tbl2(ts) SELECT ts FROM tbl1;
it's actually picking the first column of tbl2, which is businessid, so it's really attempting to do:
INSERT INTO tbl2(businessid) SELECT ts FROM tbl1;
which is clearly nonsense, and no casting will fix that.
(Your error in the original question doesn't match your tables and queries, so the details might be different as you've clearly made a mistake in mangling/obfuscating your tables or posted a newer version of the tables than the error. The principle remains.)
It's generally a really bad idea to assume your table definitions won't change and column order won't change anyway. So always be explicit about columns. In this case I think your intention might have actually been:
INSERT INTO tbl2(businessid, caseno, ts)
SELECT CAST(businessid AS integer), caseno, ts
FROM tbl1;
Note the cast, because the type of businessid is different between the two tables.
I´m new to MDX and I have a simple question. I work with the TFS Cube it is named as Team System. My problem:
I have an IIF expression where I want to check additional my expression with an AND operator. There I want to compare two DateTime objects. The report should only show me the data from the actual date. Here my code:
IIF(ISEMPTY(SUM(YTD(
[Work Item].[PlannedWeek__HierarchyByWeek].CurrentMember),
[Measures].[EffectivelyValue]))
AND[Work Item].[PlannedWeek__HierarchyByWeek].CurrentMember < Now()
, [Measures].[EffectivelyValue]
, SUM(YTD(
[Work Item].[PlannedWeek__HierarchyByWeek].CurrentMember),
[Measures].[EffectivelyValue]) )
Planned Week is a self created field which has the DateTime datatype. The Now() function has also a DateTime datatype so the comparision should be right but it happens nothing.
Thanking you in anticipation
Eugen
Hierarchy members in MDX have a data type of 'member', and do not have a 'primitive' data type like datetime, string, or integer. Only member properties have 'primitive' data types. You could either define a property like datetime of your week attribute. Assuming you are SQL Server Analysis Services, this would be done via relationships.
Or you could use string operations to extract the date information from the UniqueName property which avoids having to change the cube. The UniqueName contains the data that you defined as the key in your cube design. Assuming your week hierarchy members have a key from which you can extract something like 20130820 for August 20, 3013 via string functions (I just will use Mid(, 30, 8) as an example below), you could do something like
CLng(Mid([Work Item].[PlannedWeek__HierarchyByWeek].CurrentMember.UniqueName, 30, 8))
<
CLng(Format(Now(), "yyyymmdd"))
You will have to check what exactly the CurrentMember.UniqueName shows in your cube to adapt the above code.
And finally, you could of course also use string methods to extract the relevant parts from the UniqueName and then the CDate function on that to compare to an unchanged Now(), i. e. do all operations on the left side of the <.
Hello thank you very much for you answer. I tried:
Mid([Work Item].[xxxx_PlannedWeek__HierarchyByWeek].CurrentMember.UniqueName,58,10)
shows e.g. 2013-07-21, 2013-07-28 (it shows the week endings)
so I tried this:
AND CLng(Mid([Work Item].[xxxx_PlannedWeek__HierarchyByWeek].CurrentMember.UniqueName,58,10))
<CLng(Format(Now(), "yyyy-mm-dd"))
But it happens nothing. If I execute it in a single way it shows everywhere "true". But I have datasets with dates which are e.g. > 2013-08-23. So there should be false values too.
EDIT: OK I solved the problem. The
Format(Now(), "yyyy-mm-dd")
must be
Format(Now(), "yyyy-MM-dd")
I'm trying to do a query like this on a table with a DATETIME column.
SELECT * FROM table WHERE the_date =
2011-03-06T15:53:34.890-05:00
I have the following as an string input from an external source:
2011-03-06T15:53:34.890-05:00
I need to perform a query on my database table and extract the row which contains this same date. In my database it gets stored as a DATETIME and looks like the following:
2011-03-06 15:53:34.89
I can probably manipulate the outside input slightly ( like strip off the -5:00 ). But I can't figure out how to do a simple select with the datetime column.
I found the convert function, and style 123 seems to match my needs but I can't get it to work. Here is the link to reference about style 123
http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.help.ase_15.0.blocks/html/blocks/blocks125.htm
I think that convert's slightly wrongly documented in that version of the docs.
Because this format always has century I think you only need use 23. Normally the 100 range for convert adds the century to the year format.
That format only goes down to seconds what's more.
If you want more you'll need to past together 2 x converts. That is, past a ymd part onto a convert(varchar, datetime-column, 14) and compare with your trimmed string. milliseconds comparison is likely to be a problem depending on where you got your big time string though because the Sybase binary stored form has a granularity of 300ms I think, so if your source string is from somewhere else it's not likely to compare. In other words - strip the milliseconds and compare as strings.
So maybe:
SELECT * FROM table WHERE convert(varchar,the_date,23) =
'2011-03-06T15:53:34'
But the convert on the column would prevent the use of an index, if that's a problem.
If you compare as datetimes then the convert is on the rhs - but you have to know what your milliseconds are in the_date. Then an index can be used.