When importing a CSV file containing live market data to DolphinDB, how to convert the time columns to the DolphinDB TIME type time in DolphinDB? For example, the csv file contains a time value 093000000 which will be parsed as a string if I import the file with loadText. How can I convert it to 09:30:00.000, the TIME data type?
If you have imported the data into DolphinDB, use temporalParse to convert strings to temporal types in DolphinDB.
temporalParse( "093000000", "HHmmssSSS")
If you haven't yet imported the CSV file, use loadText. Use the schema parameter to specify the time format of the temporal column.
Check the code sample below:
schema = extractTextSchema("yourFilePath");
update schema set type="TIME" where name = "yourTimeColumnName" //modify the data type of a specified column
update schema set format="" // Add a format column to the variable schema
update schema set format="HHmmssSSS" where name = "yourTimeColumnName" // Modify the time format of the specified TIME column
loadText("yourFilePath",,schema)
Related
I have a .txt file contains time stamp column like 20180607093059000 and some other columns, but while importing into teradata SQL assistant getting invalid time stamp error. please help me the way
as I need to import file data into volatile and has to do minus operation between these file table and actual table.
The default format for a timestamp is 'YYYY-MM-DD HH:MI:SS', you can apply TO_TIMESTAMP
To_Timestamp(ts, 'yyyymmddhhmissff3')
which results in a Timestamp(6)
To get Timestamp(3) you need to CAST using a FORMAT after adding the fractional period:
Cast(Substring(ts From 1 FOR 14) || '.' || Substring(ts From 15) AS TIMESTAMP(3) FORMAT 'yyyymmddhhmiss.s(3)')
I'm using a SQLite3 database, and I have a table that looks like this:
The database is quite big and running queries is very slow. I'm trying to speed up the process by indexing some of the columns. One of the columns that I want to index is the QUOTE_DATETIME column.
Problem: I want to index by date (YYYY-MM-DD) only, not by date and time (YYYY-MM-DD HH:MM:SS), which is the data I currently have in QUOTE_DATETIME.
Question: How can I use CREATE INDEX to create an index that uses only dates in the format YYYY-MM-DD? Should I split QUOTE_DATETIME into 2 columns: QUOTE_DATE and QUOTE_TIME? If so, how can I do that? Is there an easier solution?
Thanks for helping! :D
Attempt 1: I tried running CREATE INDEX id ON DATA (date(QUOTE_DATETIME)) but I got the error Error: non-deterministic functions prohibited in index expressions.
Attempt 2: I ran ALTER TABLE data ADD COLUMN QUOTE_DATE TEXT to create a new column to hold the date only. And then INSERT INTO data(QUOTE_DATE) SELECT date(QUOTE_DATETIME) FROM data. The date(QUOTE_DATETIME) should convert the date + time to only date, and the INSERT INTO should add the new values to QUOTE_DATE. However, it doesn't work and I don't know why. The new column ends up not having anything added to it.
Expression indexes must not use functions that might change their return value based on data not mentioned in the function call itself. The date() function is such a function because it might use the current time zone setting.
However, in SQLite 3.20 or later, you can use date() in indexes as long as you are not using any time zone modifiers.
INSERT adds new rows. To modify existing rows, use UPDATE:
UPDATE Data SET Quote_Date = date(Quote_DateTime);
I have a comma delimited data set whose first 2 rows is like this:
1/13/2010 21:09,3.3
11/30/2010 7:33,7.2
....
In trying to read the data in SAS, I have done this data step below:
data myDataSet;
infile 'sampleData.csv' dlm=',';
input timestamp :mmddyy16. value;
run;
Now that the data is now a SAS data set, I try to view it by doing:
data viewData;
set myDataSet;
format timestamp date9.;
run;
proc print data=viewData;
run;
I observed that the timestamp column output only contains the date and not with the time. I want the timestamp to be read and displayed in a format like "dd-mm-yyyy HH:MM:SS". How do I ensure that in reading the file, the informat is correctly specified and no component of the timestamp is lost?
MMDDYYw is a date informat, not a datetime informat - it only reads days and larger and ignores the time. It has a practical maximum length of 10 (although it allows up to 32) as a result.
You can use MDYAMPM. to read those two dates in, among other informats. That might be the ultimately correct informat, or it might not, depending on the totality of your data; several NLS specific informats might also work. See the SAS informat documentation for more details.
I am using PHPExcel library to read spread sheet data. This library is giving me trouble reading the 'DATE' values.
Question:
How to instruct the PHPExcel to read the 'DATE' values properly even if 'setReadDataOnly' is set as 'false'?
Description:
PHPExcel version: 2.1, Environment: Ubuntu 12
Here is the code block:
$objReader = \PhpExcel\PHPExcel_IOFactory::createReaderForFile($spreadSheetFullFilePathString);
$objReader->setReadDataOnly(true);
$objPHPExcel = $objReader->load($spreadSheetFullFilePathString);
$objWorksheetObject = $objPHPExcel->getActiveSheet();
Am getting 'integer' values for the 'DATE' column values by default.
I found that, the line '$objReader->setReadDataOnly(true)' is causing the trouble.
So, Changed it as '$objReader->setReadDataOnly(false)'.
I am getting the Date column value properly after this change. But now the reader is reading ALL the ROWS + COLUMNS found in the excel.
Any help would be greatly appreciated.
If $objReader->setReadDataOnly(true);, then the only reader that can identify dates is the Gunumeric Reader. A date value can only be identified by the number format mask, and setReadDataOnly(true) tells the Reader not to read the formatting, so dates cannot subsequently be identified (Gnumeric is the exception, because it was written after the problem was identified, but none of the other Readers have yet been modified to read this information regardless of the setReadDataOnly value. If you need to identify dates, then the only option at this point is $objReader->setReadDataOnly(false);
However, for date cells, you shouldn't get an integer value; you should get a float: it's the decimal that identifies the time part of the Excel datetime serialized value.
If you know which cells contain dates, then you can convert them to unix timestamps or PHPExcel DateTime objects using the helper functions defined in the PHPExcel_Shared_DateTime class (and can then use all the standard PHP date handling functions).
You say that the reader is now reading ALL the ROWS + COLUMNS found in the excel: it should, irrespective of the setReadDataOnly value. If you don't want to read all the rows and columns, then you need to set a Read Filter.
Good Morning,
A program in SAS is about to select/merge/sort dates/times in alphanumeric value (ex : 14-Jan-2013 07:00:00.479) inside a lot of tables and to create a single table.
This program use the instruction "format E8601" and several lines after "format type $10.;
informat type $10.;", what transforms the dates in a numeric value (ex : 2013-01-14T07:02:03.647).
When this table is exported in Excel 2007, the value becomes " " and can't be modified in a traditional date/time format.
How to do it ? Is there any other format (instead of E8601) which can be used to keep the date in text or in a alphanumeric value ?
Thanks for your help.
SAS datetime values are internally represented as floating point values equal to the number of seconds since January 1, 1960. FORMATS are used to control how those numeric values are externally represented. For example, consider this:
data have;
myDateTime1 = '14-Jan-2013 07:00:00.479'dt;
myDateTime2 = '14-Jan-2013 07:00:00.479'dt;
myDateTime3 = '14-Jan-2013 07:00:00.479'dt;
format myDateTime2 datetime23.3
myDateTime3 E8601DT23.3;
put myDateTime1= 'as a number'
/ myDateTime2= 'as a normal SAS datetime'
/ myDateTime3= 'as an ISO 8601 datetime'
;
run;
When run, this is shown in the SAS log:
myDateTime1=1673766000.5 as a number
myDateTime2=14JAN2013:07:00:00.479 as a normal SAS datetime
myDateTime3=2013-01-14T07:00:00.479 as an ISO 8601 datetime
Note the three myDateTime variables have the same value but are displayed differently based on the format specified.
Assuming you have SAS Access to PC File Formats licensed, you can just use PROC EXPORT to create an Excel workbook:
proc export data=have
outfile='c:\temp\test_dates.xlsx'
replace;
run;
The data values in the Excel workbook for the two variables formatted as "datetime" values will appear correctly as Excel columns. However, the default formatting in Excel only shows the "date" portion; to display the complete value in Excel you will need to change the Excel column formats.