I want to import a huge csv-file with datetime-column in format YYYY.MM.DD hh:mm:ss.nnn (2015.09.28 00:00:02.721). It is the supported string literal format for datetime:
datetime (Transact-SQL).
I'm using DT_DBTIMESTAMP as the Integration Services data type:
Integration Services Data Types.
But the import does not work due to convertion error. I can only import in format YYYY-MM-DD hh:mm:ss.nnn. My OS, SQL 2014 and DB are all in german. How can I execute this task without search and replace in csv with regular expression?
I'm not sure I understand what you mean about Regular Expressions - but here is a solution for you:
You can import the file into a temporary holding table and then convert it to a datetime using the DataConversion task.
You can also apply the following SQL to your table - which is probably going to be quicker:
UPDATE tempTable SET NewDateCol = CAST(OldDateCol AS DATETIME)
where OldDateCol is the imported text field containing the "invalid" date format.
Because we know that this will work:
SELECT CAST('2012.05.06 11:25:33.123' AS DATETIME)
Pure SSIS dataflow solution - Derived Column transformation with the following formula for NewDateCol
(DT_DBTIMESTAMP2)(REPLACE(SUBSTRING(OldDateCol,1,10),".","-")+SUBSTRING(OldDateCol, 11, LEN(OldDateCol)))
Basically, this expression replaces . with - on the first 10 symbols and then appends the rest.
My workaround:
1. CREATE TABLE with datetime (it makes milliseconds rounding error - 2015.09.28 00:00:01.159 is 2015-09-28 00:00:01.160 etc., for whatever reason) or datetime2(3) (2015.09.28 00:00:01.159 is 2015-09-28 00:00:01.1590000 - correct but oversized)
2. SET DATEFORMAT ymd
3. BULK INSERT from csv with TABLOCK and CODEPAGE='RAW' to increase perfomance
Related
I am working with an S3 bucket and one of the fields is a datetime group and the values appear to be 128-bit integers. An example: 45376204963002810833065984. Has anyone seen this before? If so, how should this be parsed? This should work out to a date in 2022.
This is giving me problems when I try to get the data into a pandas (or polars) dataframe. The data outputs as a JSON string, and then when I try to put it into a pandas dataframe, I get an error: Value too big!. So I'm hoping to use SQL to cast this integer as a datetime before outputting as a JSON file.
My source is a file and loading into SQL Server table. I'm working on a scenario where i have to convert a string '2019-04-02T21:24:00.065' to informatica datetime format.
I tried below expression but some times its failing due to we are not receiving milliseconds from our source file in few occasions.
IIF(NOT ISNULL(DATEFIELD),TO_DATE(SUBSTR (DATEFIELD, 0, 10) || ' ' || SUBSTR(DATEFIELD, 12, 12), 'YYYY-MM-DD HH24.MI.SS.US'),NULL)
I'm looking for a permanent fix to handle all types of datetime formats regardless of what we receive in the file.
Well... I'm sorry to say, but there is no magic component that will recognize all possible date and time formats (including e.g. verbal in swahili).
You will need to detect the format for yourself. You can use a DECODE function, like e.g.:
DECODE(True,
IS_DATE(your_input_port, 'DD/MM/YYYY'), TO_DATE(your_input_port, 'DD/MM/YYYY'),
...)
If you are completely sure that only seconds/milliseconds are the missing part, you can check for length, if less than 12, use RPAD to the second part of your SUBSTR with missing format, or you can use decode as suggested by #maciejg and write code for all possible date formats.
Thanks for your inputs guys. Since we are not sure which date format we are receiving , we decided to go with a simple fix for this. I have changed the target field to varchar and simply replacing the T with ' ' value since this a staging mapping.
Again thanks for your time and inputs.
I want to format dateTime with in the cts:element-values itself. Can anyone help me around this?
I have a dateTime format string -
let $date-format := "[Y0001]-[M01]-[D01]T[h01]:[m01]:[s01].[f1]"
and I want to use it in a query like this -
cts:element-values(
xs:QName($field),
(),
($direction),
cts:and-query((cts:collection-query("urn:iddn:collections:searchable"), cts:query($cts-query)))
)
Provided $field is of type dateTime.
You can accomplish this by writing a User-Defined Function. UDFs are run as map/reduce, so they are very fast even with a large data set. I wrote an example UDF to create a day-of-the-week facet based on dateTime data. That example is based on MarkLogic 6, but should still work in MarkLogic 8.
The good thing is that UDFs are very fast. The tricky part is that you'll have to write it in C++. Full documentation in the User-Defined Functions section of the MarkLogic documentation.
I insert my date/time data into a CHAR column in the format: '6/4/2015 2:08:00 PM'.
I want that this should get automatically converted to format:
'2015-06-04 14:08:00' so that it can be used in a query because the format of DATETIME is YYYY-MM-DD hh:mm:ss.fffff.
How to convert it?
Given that you've stored the data in a string format (CHAR or VARCHAR), you have to decide how to make it work as a DATETIME YEAR TO SECOND value. For computational efficiency, and for storage efficiency, it would be better to store the value as a DATETIME YEAR TO SECOND value, converting it on input and (if necessary) reconverting on output. However, if you will frequently display the value without doing computations (including comparisons or sorting) it, then maybe a rococo locale-dependent string notation is OK.
The key function for converting the string to a DATETIME value is TO_DATE. You also need to look at the TO_CHAR function because that documents the format codes that you need to use, and because you'll use that to convert a DATETIME value to your original format.
Assuming the column name is time_string, then you need to use:
TO_DATE(time_string, '%m/%d/%Y %I:%M %x') -- What goes in place of x?
to convert to a DATETIME YEAR TO SECOND — or maybe DATETIME YEAR TO MINUTE — value (which will be further manipulated as if by EXTEND as necessary).
I would personally almost certainly convert the database column to DATETIME YEAR TO SECOND and, when necessary, convert to the string format on output with TO_CHAR. The column name would now be time_value (for sake of concreteness):
TO_CHAR(time_value, '%m/%d/%Y %I:%M %x') -- What goes in place of x?
The manual pages referenced do not immediately lead to a complete specification of the format strings. I think a relevant reference is GL_DATETIME environment variable, but finding that requires more knowledge of the arcana of the Informix product set than is desirable (it is not the first thing that should spring to anyone's mind — not even mine!). If that's correct (it probably is), then one of %p and %r should be used in place of %x in my examples. I have to get Informix (re)configured on my machine to be able to test it.
I'm trying to do a query like this on a table with a DATETIME column.
SELECT * FROM table WHERE the_date =
2011-03-06T15:53:34.890-05:00
I have the following as an string input from an external source:
2011-03-06T15:53:34.890-05:00
I need to perform a query on my database table and extract the row which contains this same date. In my database it gets stored as a DATETIME and looks like the following:
2011-03-06 15:53:34.89
I can probably manipulate the outside input slightly ( like strip off the -5:00 ). But I can't figure out how to do a simple select with the datetime column.
I found the convert function, and style 123 seems to match my needs but I can't get it to work. Here is the link to reference about style 123
http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.help.ase_15.0.blocks/html/blocks/blocks125.htm
I think that convert's slightly wrongly documented in that version of the docs.
Because this format always has century I think you only need use 23. Normally the 100 range for convert adds the century to the year format.
That format only goes down to seconds what's more.
If you want more you'll need to past together 2 x converts. That is, past a ymd part onto a convert(varchar, datetime-column, 14) and compare with your trimmed string. milliseconds comparison is likely to be a problem depending on where you got your big time string though because the Sybase binary stored form has a granularity of 300ms I think, so if your source string is from somewhere else it's not likely to compare. In other words - strip the milliseconds and compare as strings.
So maybe:
SELECT * FROM table WHERE convert(varchar,the_date,23) =
'2011-03-06T15:53:34'
But the convert on the column would prevent the use of an index, if that's a problem.
If you compare as datetimes then the convert is on the rhs - but you have to know what your milliseconds are in the_date. Then an index can be used.