I'm using nodejs with cloudera impala driver and trying to figure out how to map data types returned as integers to the actual types of the data.
Examples that I can kind of figure out from some of the data
93 = datetime
12 = text
3 = some type of number (unfortunately value came back null)
I could not find a good comprehensive reference for this anywhere. I could find things close to it, but nothing that listed everything out.
Related
Suppose you are storing a dataframe that contains dates in a sqlite database using the following code:
df <- data.frame(d = lubridate::as_date(1:30))
mydb <- DBI::dbConnect(RSQLite::SQLite(), "database.db")
DBI::dbWriteTable(mydb, "mytable", df)
DBI::dbDisconnect(mydb)
my understanding is that a conversion such as
as.numeric(df$d)
is performed before saving the data into the sqlite database. Is this correct? Why is the numeric conversion instead of the string conversion preferred?
Briefly:
There is no such thing as lubridate date object. There is a base R type Date you can create, _inter alias, with as.Date(). If you feel you must use another package so be it.
The Date type is stored internally as a (floating point) number of (fractional) days since a start date (of 1970-01-01). So storing as a number is the most efficient way.
Databases sometimes do, or do not, have native date types. MySQL didn't for many years; I can't recall where RSQLite is. It does not matter because ...
Database access packages need converters from Date and Datetime along with native support in the database.
We built this into RPostgreSQL a decade ago as PostgreSQL has it natively.
If you want textual representation but either one (or both) of the interface or database layer cannot do Date types, then call format() first and store the character string. Every database knows how to do that. But you just lost the type information...
I already used the search here (and other forums as well) but haven't found an answer exacty to what I'm trying to do.
I know that it can easily be done in some other way, and this is just a small sandbox-framework I'm coding for a University course... in a real environment I'd just take Spring, Hibernate etc.
So what I did was coding myself a small generic Data Access Layer with POJOs, working with generic methods to retrieve, check or insert data to the database (Oracle). Most of this is done through PreparedStatements.
This is working as long as I don't have joins... is it possible to put in a Column as parameter?
Example:
Table A has Attribute X + others
Table B has Attribute Y + others
PreparedStatement with query SELECT * FROM A,B WHERE "A"."X" = ?
And then fill in "B"."Y" as the parameter...
The database doesn't throw me an error or exception, but the ResultSet returned after executing the statement is empty. Is it just not possible to do, or am I just missing some escaping?
I'm using PreparedStatement.setString(int index, String value) to fill in the parameter... in lack of ideas which other setX method I could use...
Again, in a real project I'd never code that myself, but rather use something like Spring or Hibernate and not re-invent the wheel, but I see it as an interesting exercise to code such a generic small data access layer myself.
No, JDBC does not allow this. Only column values can be set. If you want to make dynamic changes to the sql statement you will have to do it before you create the PreparedStatement.
So I'm transferring an old filemaker database to MySQL and some repeating fields are causing me some problems. I've read that the ODBC standard support those fields, only when their types is "Text" and that each repetition is concatenated with a certain delimiter (see page 47 (PDF)). However, I just can't reproduce this. All I get is the first repetition.
If I export the database to the .csv format, the fields are correctly concatenated, so I'm not completely stuck, but if possible, I'd like to be able to obtain the same result with the ODBC connection. Thanks!
With JDBC and Filemaker 12 I can access the repeating field using brackets as it was table beginning with index 1.
It should be the same in ODBC.
Of course I recommend to normalize but it can help to know there is other options.
In my experience the documentation about repeating fields is a lie. :)
If you can get it to work, please, please post an answer. But I imagine you'll have to do the workaround using the csv export.
My recommendation regarding this would be to normalize the repeating fields to a separate table within FileMaker and then perform the transfer of the data. You can create a related table in FileMaker and then use a script to populate the table with the repeating field values. Let me know if you need assistance writing such a script.
I want to provide details about #Signix answer above. I was able to fetch repeating fields from JDBC but it's tricky. At page 30 of FileMaker "ODBC and JDBC Guide", it states:
Note FileMaker repeating fields are supported like arrays.
Example
INSERT INTO mytable(repField[3]) VALUES (‘this is rep 3’)
SELECT repField[1], repField[2] FROM mytable
This is the only documentation! So in theory you could use this query:
ResultSet resultSet = fmStatement.executeQuery("SELECT id, repField[1], repField[2] FROM mytable");
But the tricky part is getting the results. The only way seems to use the column index.
System.out.println(resultSet.getString("repField[1]")); // fails, throws FMSQLException
System.out.println(resultSet.getString("repField[2]")); // fails, throws FMSQLException
System.out.println(resultSet.getString("repField")); // returns repField[1]
System.out.println(resultSet.getString(2)); // returns repField[1]
System.out.println(resultSet.getString(3)); // returns repField[2]
I think the reason is because fields are being named without their bracket parts.
System.out.println(resultSet.getMetaData().getColumnName(1)); // returns "id"
System.out.println(resultSet.getMetaData().getColumnName(2)); // returns "repField"
System.out.println(resultSet.getMetaData().getColumnName(3)); // returns "repField"
So when using resultSet.getString("repField") it returns the first column value with that name. It's stupid but it works.
I have never used Informix before and I'm trying to write a query that will return records over the last 365 days.
Here is the query I have been trying to use:
Select * from Visit where vis_mod_dt between today-365 and today;
That returns no records even though I know that there is data for the last 365 days. I am guessing that the vis_mod_dt in not a true date column, although it displays as '12/31/1899' I have tried to cast this column using:
select * from visit where date(vis_mod_dt) between today-365 and today;
This still returns no data.
Any ideas?
Informix DATE format
Be aware that the date 1899-12-31 corresponds to the internal date representation of zero (0). That is, internally, Informix stores DATE values in a 4-byte integer, and counts the number of days since 1899-12-31, so Day 1 was 1900-01-01 (and since it knows that 1900 was not a leap year, Day 60 was 1900-03-01).
That fact makes me worry about what is in your table. However, if the data in your table cannot be converted to a DATE upon request, normally you would get an error.
What is your table schema?
It would be sensible for you to establish the schema either using DB-Access and the Info/Tables option, or use DB-Schema:
dbschema -d dbase -t visit
The DB-Schema output is more suitable for adding to your question.
The query expressions using 'TODAY-365' and 'TODAY' should work fine - if there is data to select.
DBDATE environment variable
There is an environment variable, DBDATE, that you may need to set to get things to work - to convert from a string representation to dates. Since you are probably based in the UK (from your icon), then you may want and need to set the value of DBDATE to:
export DBDATE=DMY4/
This says that dates consist of the day, the month, a 4-digit year and the '/' is used as the preferred separator. You won't be surprised to learn that the presumed default value is usually 'MDY4/', for US format; I use 'Y4MD-' habitually, so I see DATE value the same as DATETIME YEAR TO DAY, which is the ISO 8601:2004 notation for a date. (It has many benefits: it is unambiguous, and naive sorting algorithms sort such dates into date order.) There's actually a lot of mechanism in the background in IDS (IBM Informix Dynamic Server - which, I assume, is the DBMS that you are using; there are some alternatives that are also Informix DBMS) such that strings with 2-digit dates will usually be converted correctly (but they are ambiguous and undesirable), and separators other than '/' will be recognized on input, but the slash will be used on 'output' (when converting DATE to string).
Information needed to improve the answer to this question - 1st Edition.
If what is here does not help, then I recommend editing your question to include:
The table schema.
A few (2-4) rows of data that you think should be selected but aren't.
Platform and version information. It can help to have the version down to the level of detail of IDS 11.50.FC4W1; occasionally it matters. Most usually, the first three digits are what affect things, of course.
If your table is big (many columns), try to select the key columns (vis_mod_dt is by far the most important one). Ideally, you won't need any scroll bars in the display.
Make sure you don't include any sensitive information.
Information needed to improve the answer to this question - 2nd Edition
I will help you if you pay attention to the questions I ask you. I cannot help you if you do not pay attention to the questions I ask. And please edit your question rather than adding information as an 'answer'.
What is the table schema? What is the output from:
SELECT t.tabid, t.tabname, c.colno, c.colname, c.coltype, c.collength
FROM "informix".systables AS t, "informix".syscolumns AS c
WHERE t.tabid = c.tabid
AND t.tabname = "visit"
ORDER BY t.tabid, c.colno;
What do you get from:
SELECT TODAY, TODAY-365 FROM "informix".systables WHERE tabid = 1;
Do you have the environment variable DBDATE set? If so, what is its value?
Do you have the environment variables CLIENT_LOCALE or DB_LOCALE set? If so, what are their values?
Which version of Informix are you using?
Which platform are you using it on?
Which language or tool are you using to run the query.
Note: if you cannot copy'n'paste the queries above, then you probably do not need to include the quoted '"informix".' attributes on the system catalog; however, as written, the queries will work on any extant Informix database - OnLine 5.x, SE 5.x or 7.x, IDS 7.x, XPS 8.x, IDS 9.x or 10.x or 11.x - and any mode of database (unlogged, logged, MODE ANSI). I'd use the JOIN notation except that some of the older versions don't support it - though you have to be on very old versions for that to be a problem.
This is a little confusing, because when I run the following, I get data:
select count(*) from visit where vis_mod_dt between "10/01/2008" and "10/01/2009"
how about unloading the table to ascii file, examine the unloaded vis_mod_dt values to see if they conform to DBDATE=MDY4 (mmddyyyy) format?.. if they do, ALTER vis_mod_dt to TYPE DATE if it's not a DATE column, then LOAD the unloaded table back in.
the: "BETWEEN today-365 AND today" part of your SELECT statement works for me in my apps.
Is there a way to further restrict the lookup performed by a database lookup functoid to include another column?
I have a table containing four columns.
Id (identity not important for this)
MapId int
Ident1 varchar
Ident2 varchar
I'm trying to get Ident2 for a match on Ident1 but wish it to only lookup where MapId = 1.
The functoid only allows the four inputs any ideas?
UPDATE
It appears there is a technique if you are interested in searching across columns that are string data types. For those interested I found this out here...
Google Books: BizTalk 2006 Recipes
Seeing as I wish to restrict on a numberic column this doesn't work for me. If anyone has any ideas I'd appreciate it. Otherwwise I may need to think about my MapId column becoming a string.
I changed the MapId to MapCode of type char(3) and used the technique described in the book I linked to in the update to the original question.
The only issue I faced was that my column collations where not in line so I was getting an error from the SQL when they where concatenated in the statement generated by the map.
exec sp_executesql N'SELECT * FROM IdentMap WHERE MapCode+Ident1= #P1',N'#P1 nvarchar(17)',N'<MapCode><Ident2>'
Sniffed this using the SQL Profiler