How can I stop the conversion of number into byte formate while loading Oracle table into Kafka using confluent 5.0.0.? The conversion to byte is creating a lot of issues while reusing the data in broker. Is there a way to convert byte format to decimal in vertica db directly?
More details:
Everytime I send a table from oracle to Kafa the number type converts into byte and date converts into long. When I load the data into vertica using KafkaSource and KafkaAvroParser, the data doesn't get decoded back.
Is there a way I can edit the schema formate in schema-registry in confluent 5.0.0
Details:
curl -X GETÂ http://localhost:8081/subjects/test-oracle-jdbc-EMP1-value/versions/1
{"subject":"test-oracle-jdbc-EMP1-value","version":1,"id":2,"schema":"{\"type\":\"record\",\"name\":\"EMP1\",\"fields\":[{\"name\":\"EMPNO\",\"type\":{\"type\":\"bytes\",\"scale\":0,\"precision\":64,\"connect.version\":1,\"connect.parameters\":{\"scale\":\"0\"},\"connect.name\":\"org.apache.kafka.connect.data.Decimal\",\"logicalType\":\"decimal\"}},{\"name\":\"ENAME\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"JOB\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"MGR\",\"type\":[\"null\",{\"type\":\"bytes\",\"scale\":0,\"precision\":64,\"connect.version\":1,\"connect.parameters\":{\"scale\":\"0\"},\"connect.name\":\"org.apache.kafka.connect.data.Decimal\",\"logicalType\":\"decimal\"}],\"default\":null},{\"name\":\"HIREDATE\",\"type\":[\"null\",{\"type\":\"long\",\"connect.version\":1,\"connect.name\":\"org.apache.kafka.connect.data.Timestamp\",\"logicalType\":\"timestamp-millis\"}],\"default\":null},{\"name\":\"SAL\",\"type\":[\"null\",{\"type\":\"bytes\",\"scale\":2,\"precision\":64,\"connect.version\":1,\"connect.parameters\":{\"scale\":\"2\"},\"connect.name\":\"org.apache.kafka.connect.data.Decimal\",\"logicalType\":\"decimal\"}],\"default\":null},{\"name\":\"COMM\",\"type\":[\"null\",{\"type\":\"bytes\",\"scale\":2,\"precision\":64,\"connect.version\":1,\"connect.parameters\":{\"scale\":\"2\"},\"connect.name\":\"org.apache.kafka.connect.data.Decimal\",\"logicalType\":\"decimal\"}],\"default\":null},{\"name\":\"DEPTNO\",\"type\":[\"null\",{\"type\":\"bytes\",\"scale\":0,\"precision\":64,\"connect.version\":1,\"connect.parameters\":{\"scale\":\"0\"},\"connect.name\":\"org.apache.kafka.connect.data.Decimal\",\"logicalType\":\"decimal\"}],\"default\":null}],\"connect.name\":\"EMP1\"}"}
Assuming that you're using the JDBC source connector, you can set
"numeric.mapping":"best_fit"
in your connector configuration.
See docs and this blog for more details.
Robin is correct about the numeric mapping. As for the date showing up in as long, that is a timestamp in milliseconds. To convert that value to a Vertica TIMESTAMP, divide the number by 1,000 and then cast it as a timestamp.
To use HIREDATE as an example, you would do the following:
SELECT TO_TIMESTAMP(HIREDATE / 1000) ...
Related
The use case is that, there is an Informatica Cloud mapping which loads from SQL Server to Teradata database. If there any failures during the run time of the mapping then that mappings writes all the failed rows to a table in Teradata database. The key column in this error table is HOSTDATA which I assume. I am trying to decode the HOSTDATA column so that if a similar ETL failure happens in the production then it would be helpful in identifying the root cause much quickly. By default HOSTDATA is a column of type VARBYTES.
To decode the HOSTDATA column, converted the column to ASCII and Base 16 format. None of them made any use.
Then tried the below from the Teradata forum.
Then tried to extract the data from the error table using a BTEQ script. For that the data is being exported into a .err file and it is being loaded back into the Teradata database using a fastload script. Fastload is unable to load the data because there is no specific delimiter for the data. There data in the .err file looks gibberish. Snapshot of the data from the .err file:
My end goal is to interpret the Hostdata column in a more human readable way. Any suggestions in this direction are also welcome.
The Error Table Extractor command twbertbl which is part of "Teradata Parallel Transporter Base" software is designed to extract and format HOSTDATA from the error table VARBYTE column.
Based on the screenshot in your question, I suspect you will need to specify FORMATTED as the record format option for twbertbl (default is DELIMITED).
I've received a SQLite file with .db extension.
Opening it, I have a column with serial numbers: i.e. 1600414704594 (that should correspond to 2020-10-09 and whatsover time)
The db comes from outside and I don't know how that date field has been built.
Could you suggest me a query to get a valid date/time from that db column ?
I've tried reading this post but none of the given solution returned me a valid (and actual) date, please help me.
It looks like an Unix time in milliseconds. SQLite's unixepoch modifier expects it in seconds. The conversion is fairly easy :
SELECT DATETIME(1600414704594 / 1000, 'unixepoch')
2020-09-18 07:38:24
I have a sqlite db (it is the spiceworks db) and I am mapping the tables to grails domain classes. There is a table in particular that is in a datetime format (yyyy-MM-dd HH:mm:ss). An example of such a date in the db for anyone who wants to verify would be: 2015-06-26 15:32:39
I created the domain class and mapped my properties to the columns. Let grails generate the views so that they are default. When I try to get to the index page I get:
URI
/spiceworks/weeklyReportItem
Class
java.text.ParseException
Message
Unparseable date: "2015-06-26 15:32:39" does not match (\p{Nd}++)\Q-\E(\p{Nd}++)\Q-\E(\p{Nd}++)\Q \E(\p{Nd}++)\Q:\E(\p{Nd}++)\Q:\E(\p{Nd}++)\Q.\E(\p{Nd}++)
I've used MSSQL datetimes in the past and have never ran into this issue. In the database, the data type for the problematic column is datetime. Anyone know what's going on?
EDIT: I only have Read permissions on the db and the spiceworks source code isn't open source.
If you look at that regular expression, you'll see it's looking for a ISO8601 timestamp formatted like YYYY-MM-DD HH:mm:SS.SSS - in other words, it needs a decimal seconds, and you're just providing whole seconds.
An easy fix would be to update every existing value in the relevant column by appending '.000' to it, and update the insertion routines to do the same.
If you refer to the sqlite date time functions, there's a %f format specifier for strftime() that can be used to produce fractional seconds that might be helpful if you're building the timestamp directly in an insert query.
In the database, the data type for the problematic column is datetime
Sqlite3 doesn't have a datetime type. The timestamps you're storing are strings. More information.
I'm using ROracle to query a database with a VARCHAR2 field containing some Unicode characters. When I access the database directly or via RJDBC, I have no issues with pulling this data.
When I pull the data with ROracle, I get ????? instead of the text.
In OCI you have use env. variable NLS_LANG. For example:
NLS_LANG=AMERICAN_AMERICA.AL32UTF8
will make OCI client return all strings returned in UTF8. This should work, if internal string representation in R also uses UTF8. Then ROracle can make simple binary copy from one buffer into another buffer.
Oracle uses question marks in case when it can not translate char into target code page.
In one of my application i was trying to save one file (Size of the file is 2MB) in to the teradata database table as a BLOB data by using JDBC connection. But everytime i was getting one exception which is as follows
com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata JDBC Driver] [TeraJDBC 14.00.00.01] [Error 1186] [SQLState HY000] Parameter 2 length is 2215538 bytes, which is greater than the maximum 64000 bytes that can be set.
Please help me out resolving this issue.
Thanks,
Sourav
Try to set the blob column larger ,or just use the system default value ,which is 2097088000 btw.
This is from Teradata User Documentations : SQL data types and literals Chapter 7
"the number of bytes to allocate for the BLOB column. The maximum number
of bytes is 2097088000, which is the default if n is not specified."
Assuming you have data in String format we can use setCharacterStream to bypass the restriction of 64000 bytes.
Using preparedStatement to insert large amount of data clob column as
pStmt.setCharacterStream(paramterIndex, new StringReader(StringToBeInserted), StringToBeInserted.length());