I have a HIVE table which stores timestamp in yyyy-MM-dd HH:mm::ss.S format. But I have noticed that The value in such columns keeps changing, e.g. the value that I sqooped was '2016-01-25 00:00:00.0' but sometimes (not always) this value is shown as '2016-01-24 19:00:00.0'.
To make matters worse, this does not happen in all tables at the same time. Sometimes table1 would have the correct format and table2, the incorrect one (2016-01-24 19:00:00.0) and vice versa
I don't know if this is relevant, but we recently moved from ORACLE to HIVE. In the oracle table the date column was of type 'DATE' which stored datda as 25-JAN-16 but in HIVE the column is of type 'TIMESTAMP' and stores data as 2016-01-25 00:00:00.0.
The timestamp is correct when I sqoop the data but is sometimes incorrect when I check it out later.
Could someone please tell me how I could fix or work around this problem?
I believe this is because of the timezone problem. Try using timezone UDFs in hive and check whether you are getting it right.
eg. to_utc_timestamp
Related
I have a sqlite db (it is the spiceworks db) and I am mapping the tables to grails domain classes. There is a table in particular that is in a datetime format (yyyy-MM-dd HH:mm:ss). An example of such a date in the db for anyone who wants to verify would be: 2015-06-26 15:32:39
I created the domain class and mapped my properties to the columns. Let grails generate the views so that they are default. When I try to get to the index page I get:
URI
/spiceworks/weeklyReportItem
Class
java.text.ParseException
Message
Unparseable date: "2015-06-26 15:32:39" does not match (\p{Nd}++)\Q-\E(\p{Nd}++)\Q-\E(\p{Nd}++)\Q \E(\p{Nd}++)\Q:\E(\p{Nd}++)\Q:\E(\p{Nd}++)\Q.\E(\p{Nd}++)
I've used MSSQL datetimes in the past and have never ran into this issue. In the database, the data type for the problematic column is datetime. Anyone know what's going on?
EDIT: I only have Read permissions on the db and the spiceworks source code isn't open source.
If you look at that regular expression, you'll see it's looking for a ISO8601 timestamp formatted like YYYY-MM-DD HH:mm:SS.SSS - in other words, it needs a decimal seconds, and you're just providing whole seconds.
An easy fix would be to update every existing value in the relevant column by appending '.000' to it, and update the insertion routines to do the same.
If you refer to the sqlite date time functions, there's a %f format specifier for strftime() that can be used to produce fractional seconds that might be helpful if you're building the timestamp directly in an insert query.
In the database, the data type for the problematic column is datetime
Sqlite3 doesn't have a datetime type. The timestamps you're storing are strings. More information.
I am trying to load a .csv file using SQL loader. I get this error message on a column that should store time column:
column TIME.ORA-01843: not a valid month
In my .csv file time has the format HH:MM:SS but I can't understand why Oracle does not recognise it.
Using the command select * from nls_session_parameters; I see that the default time format is HH24.MI.SSXFF. I tried to change the time separator in my csv file but I got the same error result.
I have a very simple control file, that looks like below.
LOAD DATA
INFILE Cycling_Accidents0512.csv
INTO TABLE CYCLING_ACCIDENTS
FIELDS TERMINATED BY ','
(ID,
ACC_ID,
OSGB_EASTING,
OSGB_NORTHING,
WGS_LONG,
WGS_LAT,
POLICE_FORCE,
ACC_SEVERITY,
NUM_VEI,
NUM_CAS,
ACC_DATE,
DAY_WEEK,
TIME,
LOC_AUTH_DIS,
LOC_AUTH_HIGH,
FST_ROAD_CLASS,
FST_ROAD_NUMBER,
ROAD_TYPE,
SPEED_LIMIT,
JUNCT_DETAIL,
JUNCT_CONTROL,
SND_ROAD_CLASS,
SND_ROAD_NUM,
PED_HUM,
PED_PHY,
LIGHT_COND,
WEATH_COND,
ROAD_SUR_COND,
SPEC_COND,
CARR_HAZARDS,
URB_RUR,
POLICE_ATT,
LSOA_ACC_LOC,
VEI_TYPE)
If someone of you can help me to modify the control file in order to make the time format acceptable to Oracle, that would be appreciated. I tried to look up on other web resources but I haven't found anything that could help.
Thanks!
You mentioned querying the session NLS parameters, but the value you showed appears to be NLS_TIME_FORMAT, which is only used internally.
Oracle doesn't have a time-only type, so your field is presumably actually a DATE type (or possibly TIMESTAMP). The values in the column will have a date part, even if you ignore it.
SQL*Loader will use NLS_DATE_FORMAT to interpret the data file value for a DATE field. If that is DD/MM/YYYY then it would liberally interpret a value like 22:41:17 as the 22nd day of the 41st month - hence your error - in the year 17.
You can specify the date format model in the control file:
TIME DATE 'HH24:MI:SS',
The value in the table would have that time, on the first day of the current month.
The SQL*Loader documentation does refer to a TIME data type which I have never seen used, and I can't find any references to it anywhere, including MOS. A quick bit of experimentation hasn't helped. If I make the control file entry:
TIME TIME,
... then the record is rejected with ORA-00933: SQL command not properly ended. The log file also shows the data type as DATETIME HH24.MI.SSXFF, which looks related to the NLS_TIME_FORMAT value. I haven't found a way to make it accept that. If I change the column definition from DATE to TIMESTAMP then I get a different error, ORA-00904: "TO_TIME": invalid identifier, which is even stranger. It almost looks like these data types are defined in SQL*Loader for future use. (This discussion suggests they thought about adding TIME as a database type in 10g, but obviously can't verify that. And this is in the SQL*Loader reference at least back to 9i).
When i open up toad sql editor and run a select statement on a column that has datatype of timestamp, then toad hangs every time. But when i do a trunc on the timestamp column, the data is returned quickly.
What is it about timestamp that makes toad hang? Also is there a way without using trunc?
Guess based on the information you have provided:
It is possible that the table that you are accessing has a function based index on TRUNC(timestamp_column), which causes Oracle to retrieve the data faster.
You can use the indexes tab in toad or use all_indexes data dictionary view in Oracle and see if that is the case with your specific table and column.
I really doubt if this is a problem with Toad and Oracle time stamps in general.
My database table have a Timestamp column named as inTime and i am using LINQ for insert, update data in SQL server. Now i want to add the timestamp value in my database but i don't know how to insert?
spaBL.attendance obj = new spaBL.attendance();
obj.FK_employeeId = 1;
obj.inTime =[what to write here??]
inTime is of timestamp type.
Timestamp as in SQL Server Timestamp data type?
;) If I got a cent every time someone did not read the documentation and thought that is a TIME STAMP I would be more rich than bill gates.
Timestamp data time has NO TIME INFORMATION IN IT. It is a running version number, legacy to Sybase SQL Server where SQL Server from Microsoft originated and a totally borked design.
So, no, you CAN NOT SET THAT FIELD, sorry, and it has no usable value except to see whether it changed (then the row was updated).
The documentation is explicitly clear on that, even if some people think reading is maybe a lost art:
http://msdn.microsoft.com/en-us/library/ms182776(v=sql.90).aspx
Is a data type that exposes automatically generated, unique binary
numbers within a database. timestamp is generally used as a mechanism
for version-stamping table rows.
There is no way for you to set it. CHange your LINQ setup to not update this column.
Timestamp values are auto generated by the server on inserts. You should insert rows to the table without supplying a value for the timestamp column. Sql server will then generate the value for the column.
Note the columns of type timestamp does NOT contain values that can be parsed as a DateTime. The idea behind timestamp columns is that they can be used to check if a row has been updated between fetching the row and trying to update the row with new values.
You do not provide values for columns with a Timestamp (Rowversion) data type. SQL Server will provide a value for you automatically (and it won't be a datetime value). Therefore you do not need to be concerned with having Linq To SQL insert a Timestamp value for you. In fact, you cannot do it. SQL Server will do it for you. You can however, retrieve the value of a Timestamp column. I believe the corresponding C# type will be System.Data.Linq.Binary.
How to add Timestamp value in SQL database by using LINQ
hope it helps
You cannot update a timestamp. See here. And indeed, as TomTom indicated, it doesn't contain actual time information.
I have never used Informix before and I'm trying to write a query that will return records over the last 365 days.
Here is the query I have been trying to use:
Select * from Visit where vis_mod_dt between today-365 and today;
That returns no records even though I know that there is data for the last 365 days. I am guessing that the vis_mod_dt in not a true date column, although it displays as '12/31/1899' I have tried to cast this column using:
select * from visit where date(vis_mod_dt) between today-365 and today;
This still returns no data.
Any ideas?
Informix DATE format
Be aware that the date 1899-12-31 corresponds to the internal date representation of zero (0). That is, internally, Informix stores DATE values in a 4-byte integer, and counts the number of days since 1899-12-31, so Day 1 was 1900-01-01 (and since it knows that 1900 was not a leap year, Day 60 was 1900-03-01).
That fact makes me worry about what is in your table. However, if the data in your table cannot be converted to a DATE upon request, normally you would get an error.
What is your table schema?
It would be sensible for you to establish the schema either using DB-Access and the Info/Tables option, or use DB-Schema:
dbschema -d dbase -t visit
The DB-Schema output is more suitable for adding to your question.
The query expressions using 'TODAY-365' and 'TODAY' should work fine - if there is data to select.
DBDATE environment variable
There is an environment variable, DBDATE, that you may need to set to get things to work - to convert from a string representation to dates. Since you are probably based in the UK (from your icon), then you may want and need to set the value of DBDATE to:
export DBDATE=DMY4/
This says that dates consist of the day, the month, a 4-digit year and the '/' is used as the preferred separator. You won't be surprised to learn that the presumed default value is usually 'MDY4/', for US format; I use 'Y4MD-' habitually, so I see DATE value the same as DATETIME YEAR TO DAY, which is the ISO 8601:2004 notation for a date. (It has many benefits: it is unambiguous, and naive sorting algorithms sort such dates into date order.) There's actually a lot of mechanism in the background in IDS (IBM Informix Dynamic Server - which, I assume, is the DBMS that you are using; there are some alternatives that are also Informix DBMS) such that strings with 2-digit dates will usually be converted correctly (but they are ambiguous and undesirable), and separators other than '/' will be recognized on input, but the slash will be used on 'output' (when converting DATE to string).
Information needed to improve the answer to this question - 1st Edition.
If what is here does not help, then I recommend editing your question to include:
The table schema.
A few (2-4) rows of data that you think should be selected but aren't.
Platform and version information. It can help to have the version down to the level of detail of IDS 11.50.FC4W1; occasionally it matters. Most usually, the first three digits are what affect things, of course.
If your table is big (many columns), try to select the key columns (vis_mod_dt is by far the most important one). Ideally, you won't need any scroll bars in the display.
Make sure you don't include any sensitive information.
Information needed to improve the answer to this question - 2nd Edition
I will help you if you pay attention to the questions I ask you. I cannot help you if you do not pay attention to the questions I ask. And please edit your question rather than adding information as an 'answer'.
What is the table schema? What is the output from:
SELECT t.tabid, t.tabname, c.colno, c.colname, c.coltype, c.collength
FROM "informix".systables AS t, "informix".syscolumns AS c
WHERE t.tabid = c.tabid
AND t.tabname = "visit"
ORDER BY t.tabid, c.colno;
What do you get from:
SELECT TODAY, TODAY-365 FROM "informix".systables WHERE tabid = 1;
Do you have the environment variable DBDATE set? If so, what is its value?
Do you have the environment variables CLIENT_LOCALE or DB_LOCALE set? If so, what are their values?
Which version of Informix are you using?
Which platform are you using it on?
Which language or tool are you using to run the query.
Note: if you cannot copy'n'paste the queries above, then you probably do not need to include the quoted '"informix".' attributes on the system catalog; however, as written, the queries will work on any extant Informix database - OnLine 5.x, SE 5.x or 7.x, IDS 7.x, XPS 8.x, IDS 9.x or 10.x or 11.x - and any mode of database (unlogged, logged, MODE ANSI). I'd use the JOIN notation except that some of the older versions don't support it - though you have to be on very old versions for that to be a problem.
This is a little confusing, because when I run the following, I get data:
select count(*) from visit where vis_mod_dt between "10/01/2008" and "10/01/2009"
how about unloading the table to ascii file, examine the unloaded vis_mod_dt values to see if they conform to DBDATE=MDY4 (mmddyyyy) format?.. if they do, ALTER vis_mod_dt to TYPE DATE if it's not a DATE column, then LOAD the unloaded table back in.
the: "BETWEEN today-365 AND today" part of your SELECT statement works for me in my apps.