How do you bind parameters in the Julia sqlite library? - sqlite

I'm trying to use the Julia SQLite.jl library, but I can't figure out how to bind variables.
using SQLite
# Create DB and table
db = SQLite.DB("mydb.sqlite")
SQLite.createtable!(db, "Student", Tables.Schema((:Name, :Grade), (String, Int64)); temp=false, ifnotexists=true)
# Add vals
SQLite.execute(db, "INSERT INTO Student VALUES('Harry', 1)")
# Prepared statement: Can use: ?, ?NNN, :AAA, $AAA, #AAA
insert_stmt = SQLite.Stmt(db, "INSERT INTO Student VALUES(:N, :G)")
SQLite.bind!(insert_stmt, Dict("N"=>"George", "G"=>"4"))
# This fails, with error: SQLiteException("`:N` not found in values keyword arguments to bind to sql statement")
insert_stmt = SQLite.Stmt(db, "INSERT INTO Student VALUES(:N, :G)")
SQLite.bind!(insert_stmt, Dict(:N=>"George", :G=>"4"))
SQLite.execute(insert_stmt)
# This fails, with error: SQLiteException("values should be provided for all query placeholders")
insert_stmt = SQLite.Stmt(db, "INSERT INTO Student VALUES(?1, ?2)")
SQLite.bind!(insert_stmt, ["George", "4"])
SQLite.execute(insert_stmt)
# This fails, with error: SQLiteException("values should be provided for all query placeholders")
insert_stmt = SQLite.Stmt(db, "INSERT INTO Student VALUES(':N', ':G')")
SQLite.bind!(insert_stmt, Dict(:N=>"George", :G=>"4"))
SQLite.execute(insert_stmt)
# This doesn't bind, it inserts ':N' and ':G'
What's the right syntax? Thanks!

You could try:
stmt = SQLite.Stmt(db, "INSERT INTO Student VALUES(?, ?)")
DBInterface.execute(stmt, ["Jack",2])
Let's check if this worked:
julia> DBInterface.execute(db, "SELECT * FROM Student") |> DataFrame
2×2 DataFrame
Row │ Name Grade
│ String Int64
─────┼───────────────
1 │ Harry 1
2 │ Jack 2

(Similar to Przemyslaw Szufel's answer, just with named parameters like in the question.)
The documentation for DBInterface.execute (which the one for SQLite.execute recommends to be used) says:
DBInterface.execute(db::SQLite.DB, sql::String,
[params]) DBInterface.execute(stmt::SQLite.Stmt, [params])
Bind any positional (params as Vector or Tuple) or named (params as
NamedTuple or Dict) parameters to an SQL statement, given by db and
sql or as an already prepared statement stmt, execute the query and
return an iterator of result rows.
So you can pass on your Dict to execute directly as:
julia> insert_stmt = SQLite.Stmt(db, "INSERT INTO Student VALUES(:N, :G)")
julia> DBInterface.execute(insert_stmt, Dict(:N => "Palani", :G => 3))
SQLite.Query(SQLite.Stmt(SQLite.DB("mydb.sqlite"), 1), Base.RefValue{Int32}(101), Symbol[], Type[], Dict{Symbol, Int64}(), Base.RefValue{Int64}(0))

Related

Snowflake, Recursive CTE , Getting error String 'AAAA_50>BBBB_47>CCCC_92' is too long and would be truncated in 'CONCAT'

I am creating a recursive CTE in snowflake for getting complete path an getting following error:
String 'AAAA_50>BBBB_47>CCCC_92' is too long and would be truncated in 'CONCAT'
My script is as follows: (it works fine for 2 levels, starts failing for 3rd level)
with recursive plant
(child_col,parent_col,val )
as
(
select child_col, '' parent_col , trim(child_col) from My_view
where condition1 = 'AAA'
union all
select A.child_col,A.parent_col,
concat(trim(A.child_col),'>')||trim(val)
from My_view A
JOIN plant as B ON trim(B.child_col) = trim(A.parent_col)
)
select distinct * from plant
Most likely the child_col data type is defined as VARCHAR (N), this type is being passed on. Because CONCAT Returns:
The data type of the returned value is the same as the data type of
the input value(s).
Try to explicitly cast a type to a string like this cast(trim(child_col) as string):
Full code:
with recursive plant (child_col,parent_col,val )
as (
select child_col, '' parent_col , cast(trim(child_col) as string)
from My_view
where condition1 = 'AAA'
union all
select A.child_col, A.parent_col, concat(trim(A.child_col),'>')||trim(val)
from My_view A
join plant as B ON trim(B.child_col) = trim(A.parent_col)
)
select distinct * from plant
Remember that recursion in Snowflake is limited to 100 loops by default.
If you want to increase them, you need to contact support.
Reference: CONCAT Troubleshooting a Recursive CTE

Error: Query schema does not match table schema. QuerySchema=('long')

I am running a .set-or-append command to ingest from 1 table into another. I know that the query from the source table is fine and the target table, if it exists, should have the same query but if not the command should just create it. Initially I was not having a problem with this. But a few of my .set-or-append queries have been getting this error:
Invalid query for distributed set / append / replace operation. Error: Query schema does not match table schema. QuerySchema=('long'), TableSchema=('datetime,string,string,string,string,dynamic,dynamic,dynamic'). Query:...'
I know for a fact that the schemas match. I ran the same command again and again and on about the 3rd try the call succeeded. Which makes 0 sense to me. So what is this above error and why did the same command work after failing with no change to the query whatsoever?
The query/command I am running is essentially the following:
.set-or-append async TargetTable <|
SourceTable
| where __id in ("...", "....", ........) // is aproximately 250 distinct ids in "in" operator
It appears that the query you're using is extending data with extra column:
extend hashBucket = hash(row_number(), ...) | where hashBucket == ... - and thus you're getting schema mismatch.
Perhaps, your intention was to filter based on the hashBucket, and in this case you can just use filtering without extension:
where hash(row_number(), ...) == ...
Yoni could you explain what you mean by bag_unpack giving an inconsistent mismatch? i aligned my bag_unpack ... project-reorder to match the target table i'm unpacking to, but it just changes around a few variables types in the error message:
Query schema does not match table schema.
QuerySchema=(
'datetime,long,datetime,string,string,datetime,string,
long,real,string,bool,guid,guid,string,real'),
TableSchema=(
'datetime,long,datetime,string,string,datetime,string,
long,real,guid,guid,string,bool,string,real')
really confused what the table schema and query schema even are at this point.
For reference my query is like this:
.set-or-append async apiV2FormationSearchTransform <|
//set notruncation;
apiV2FormationSearchLatest
| where hash(toguid(fullRecord["id"]), 1) == 0
| project fullRecord
| evaluate bag_unpack(fullRecord)
| extend dateCatalogued = todatetime(column_ifexists("dateCatalogued", ""))
, simpleId = tolong(column_ifexists("simpleId", ""))
, dateLastModified = todatetime(column_ifexists("dateLastModified", ""))
, reportedFormationName = tostring(column_ifexists("reportedFormationName", ""))
, comments = tostring(column_ifexists("comments", ""))
, dateCreated = todatetime(column_ifexists("dateCreated", ""))
, formationName = tostring(column_ifexists("formationName", ""))
, internalId = tolong(column_ifexists("internalId", ""))
, topDepth = toreal(column_ifexists("topDepth", ""))
, wellId = column_ifexists("wellId", toguid(""))
, id = column_ifexists("id", toguid(""))
, methodObtained = tostring(column_ifexists("methodObtained", ""))
, isTarget = tobool(column_ifexists("isTarget", ""))
, completionId = tostring(column_ifexists("completionId", ""))
, baseDepth = toreal(column_ifexists("baseDepth", ""))
| project-reorder dateCatalogued
, simpleId
, dateLastModified
, reportedFormationName
, comments
, dateCreated
, formationName
, internalId
, topDepth
, wellId
, id
, methodObtained
, isTarget
, completionId
, baseDepth
and this is the getschema output of my target table:
dateCatalogued 0 System.DateTime datetime
simpleId 1 System.Int64 long
dateLastModified 2 System.DateTime datetime
reportedFormationName 3 System.String string
comments 4 System.String string
dateCreated 5 System.DateTime datetime
formationName 6 System.String string
internalId 7 System.Int64 long
topDepth 8 System.Double real
wellId 9 System.Guid guid
id 10 System.Guid guid
methodObtained 11 System.String string
isTarget 12 System.SByte bool
completionId 13 System.String string
baseDepth 14 System.Double real
I had a similar problem. In my case, it was that there was there in the DB another table with the same name (although in a different folder. I was using the with (folder = 'foo/bar') <| option ). I have changed the table name and the error disappeared.

Parameters and NULL

I'm having trouble passing NULL as an INSERT parameter query using RPostgres and RPostgreSQL:
In PostgreSQL:
create table foo (ival int, tval text, bval bytea);
In R:
This works:
res <- dbSendQuery(con, "INSERT INTO foo VALUES($1, $2, $3)",
params=list(ival=1,
tval= 'not quite null',
bval=charToRaw('asdf')
)
)
But this throws an error:
res <- dbSendQuery(con, "INSERT INTO foo VALUES($1, $2, $3)",
params=list(ival=NULL,
tval= 'not quite null',
bval=charToRaw('asdf')
)
)
Using RPostgres, the error message is:
Error: expecting a string
Under RPostgreSQL, the error is:
Error in postgresqlExecStatement(conn, statement, ...) :
RS-DBI driver: (could not Retrieve the result : ERROR: invalid input
syntax for integer: "NULL"
)
Substituting NA would be fine with me, but it isn't a work-around - a literal 'NA' gets written to the database.
Using e.g. integer(0) gives the same "expecting a string" message.
You can use NULLIF directly in your insert statement:
res <- dbSendQuery(con, "INSERT INTO foo VALUES(NULLIF($1, 'NULL')::integer, $2, $3)",
params=list(ival=NULL,
tval= 'not quite null',
bval=charToRaw('asdf')
)
)
works with NA as well.
One option here to workaround the problem of not knowing how to articulate a NULL value in R which the PostgresSQL pacakge will be able to successfully translate is to simply not specify the column whose value you want to be NULL in the database.
So in your example you could use this:
res <- dbSendQuery(con, "INSERT INTO foo (col2, col3) VALUES($1, $2)",
params=list(tval = 'not quite null',
bval = charToRaw('asdf')
)
)
when you want col1 to have a NULL value. This of course assumes that col1 in your table is nullable, which may not be the case.
Thanks all for the help. Tim's answer is a good one, and I used it to catch the integer values. I went a different route for the rest of it, writing a function in PostgreSQL to handle most of this. It looks roughly like:
CREATE OR REPLACE FUNCTION add_stuff(ii integer, tt text, bb bytea)
RETURNS integer
AS
$$
DECLARE
bb_comp bytea;
rows integer;
BEGIN
bb_comp = convert_to('NA', 'UTF8'); -- my database is in UTF8.
-- front-end catches ii is NA; RPostgres blows up
-- trying to convert 'NA' to integer.
tt = nullif(tt, 'NA');
bb = nullif(bb, bb_comp);
INSERT INTO foo VALUES (ii, tt, bb);
GET DIAGNOSTICS rows = ROW_COUNT;
RETURN rows;
END;
$$
LANGUAGE plpgsql VOLATILE;
Now to have a look at the RPostgres source and see if there's an easy-enough way to make it handle NULL / NA a bit more easily. Hoping that it's missing because nobody thought of it, not because it's super-tricky. :)
This will give the "wrong" answer if someone is trying to put literally 'NA' into the database and mean something other than NULL / NA (e.g. NA = "North America"); given our use case, that seems very unlikely. We'll see in six months time.

refer to stored proc output param in macro?

Is there a way to refer to an output parameter of a stored procedure in a macro?
My stored procedure is:
CREATE PROCEDURE db.ssis_load_nextID
(IN tbl VARCHAR(30), OUT nextID SMALLINT )
BEGIN
DECLARE maxID SMALLINT;
SELECT MAX(loadID) INTO maxID
FROM db.SSIS_Load
WHERE TABLENAME = tbl
GROUP BY TABLENAME;
IF maxID IS NULL THEN
SET nextID = 1;
ELSE
SET nextID = maxID + 1;
END if;
END;
I want to refer to this result in a macro like:
CREATE MACRO db.tbSTG_m AS (
INSERT INTO db.tbProd (ID1, ID2, f1, f2, ..., fn, loadID)
SELECT ID1, ID2, f1, f2,..., fn,
CALL db.ssis_Load_nextID('tbProd',nextID)
FROM db.tbstg
; );
because running CALL db.ssis_Load_nextID('tbProd',nextID) returns the result I want in the first (only) row of the first (only) column.
I tried storing the result in a variable in the macro, but apparently, that's unsupported.
Also, I'd like to start with an empty SSIS_load table, so it creates the first row when the first table is loaded, instead of pre-populating the load table before the automated load process starts.
All help appreciated,
-Beth
fyi, We got it to work by removing the 'group by tablename' clause and embedding the sp in the macro:
CREATE MACRO db.tbSTG_m AS (
INSERT INTO db.tbProd
SELECT ID1, ID2, f1, f2, ..., fn (
SELECT ZEROIFNULL(MAX(loadID))+1
FROM db.ssis_load
WHERE TABLENAME = 'tbStg') mx
FROM db.tbSTG;
);
You can't use a stored proc for that (you would have to use a UDF not a procedure)
however you can do it in your macro
syntax may not be 100% correct.. working from memory but should get you close
I am assuming tbl is a parameter passed in correct?
basically you join to the id table and use that in your insert...
then you update the id table with the maximum freshly inserted ids
CREATE MACRO db.tbSTG_m AS (
INSERT INTO db.tbProd (ID1, ID2, f1, f2, ..., fn, loadID)
SELECT ID1, ID2, f1, f2,..., fn, MAXloadID + SUM(1) OVER(ROWS UNBOUNDED PRECEDING)
FROM db.tbstg
cross join (SELECT MAX(loadID) as MAXloadID
FROM db.SSIS_Load
WHERE TABLENAME = tbl
GROUP BY TABLENAME) as IDGEN
;
update db.SSIS_Load from (select MAX(loadID) as MAXloadID from tbl) as upid
set loadID = upid.MAXloadID
where db.SSIS_Load.TABLENAME = tbl
);

SQLITE Select results into a string

I am looking for a method to return the results of an SQLite query as a single string for use in an internal trigger.
Something like Python's 'somestring'.join() method.
Table: foo
id | name
1 | "foo"
2 | "bar"
3 | "bro"
Then a select statement:
MAGIC_STRING_CONCAT_FUNCTION(SELECT id FROM foo,",");
To return
"1,2,3"
You're looking for the group_concat function:
group_concat((SELECT id FROM foo), ",");
The following is the description of the function group_concat from the documentation:
group_concat(X) group_concat(X,Y)
The group_concat() function returns a string which is the
concatenation of all non-NULL values of X. If parameter Y is present
then it is used as the separator between instances of X. A comma (",")
is used as the separator if Y is omitted. The order of the
concatenated elements is arbitrary.

Resources