I have a sql like this, executing it first time is ok and the second time will be fail. Because the file exist.
How to use write_nos overwrite parquet file ?
SELECT ObjectSize, RecordCount
FROM WRITE_NOS_FM (
ON ( SELECT * FROM RiverFlowPerm_csv )
USING
LOCATION('/s3/bigdaxxxxxx.s3.amazonaws.com/RiverFlowPerm/')
STOREDAS('PARQUET')
COMPRESSION('GZIP')
) AS d ;
Related
I have a sql command below， I don't understand what the flowdata_manifest.json for.
SELECT NodeId, AmpId, Sequence, ObjectName, ObjectSize, RecordCount
FROM WRITE_NOS_FM (
ON ( SELECT * FROM RiverFlowPerm )
PARTITION BY SiteNo ORDER BY SiteNo
USING
LOCATION('YOUR-STORAGE-ACCOUNT/RiverFlowPerm_PartitionBy/')
STOREDAS('PARQUET')
NAMING('DISCRETE')
MANIFESTFILE('YOUR-STORAGE-ACCOUNT/RiverFlowPerm_PartitionBy/flowdata_manifest.json')
INCLUDE_ORDERING('TRUE')
MAXOBJECTSIZE('4MB')
COMPRESSION('GZIP')
) AS d
ORDER BY AmpId;
When using a stored procedure to dynamically generate a table based on configuration and return a result set (SELECT) with the records in that table, the second call to that procedure to generate a different table structure returns no records and it reports a missing column from a previous temporary table of the same name.
I tried this with MariaDB 10.3 and 10.1.21 and received the same result. I have minimized my code here to the minimum to demonstrate the error after trying several variations of single and multiple sub-procedures.
I also tried using some transaction control with COMMITS after executing the process, before trying to start the process with a different parameter, but got the same results.
DROP PROCEDURE IF EXISTS CreateATable;
DELIMITER $$
CREATE PROCEDURE CreateATable( _TableType tinyint )
BEGIN
DROP TEMPORARY TABLE IF EXISTS aTable;
IF _TableType = 1 THEN
SET #SQL_Statement :=
CONCAT(
'CREATE TEMPORARY TABLE aTable (',
'the_id bigint, ',
'the_column varchar(100) ',
') engine=INNODB',
';');
ELSE
SET #SQL_Statement :=
CONCAT(
'CREATE TEMPORARY TABLE aTable (',
'the_id bigint, ',
'the_other_column varchar(100) ',
') engine=INNODB',
';');
END IF;
PREPARE stmtCreateTable FROM #SQL_Statement;
EXECUTE stmtCreateTable;
DEALLOCATE PREPARE stmtCreateTable;
SET #SQL_Statement := NULL;
END$$
DELIMITER ;
DROP PROCEDURE IF EXISTS GetATable;
DELIMITER $$
CREATE PROCEDURE GetATable()
BEGIN
CALL CreateATable( 1 );
SELECT * FROM aTable;
CALL CreateATable( 2 );
SELECT * FROM aTable;
END$$
DELIMITER ;
DROP PROCEDURE IF EXISTS GetATable2;
DELIMITER $$
CREATE PROCEDURE GetATable2(_TableType tinyint)
BEGIN
CALL CreateATable( _TableType );
SELECT * FROM aTable;
END$$
DELIMITER ;
/*
Test execution script starts here
*/
-- Just CALL Create for one and Select
CALL CreateATable( 1 );
DESCRIBE aTable;
SELECT * FROM aTable;
CALL CreateATable( 2 );
DESCRIBE aTable;
SELECT * FROM aTable;
-- -> no errors
-- now CALL procedure to Create and Select from two different temp tables
CALL GetATable();
-- -> no errors
-- now CALL procedure to CREATE AND SELECT from ONE temp table definition using a parameter to select
CALL GetATable2(1);
CALL GetATable2(2);
-- Error Code: 1054. Unknown column 'mySchema.aTable.the_column' in 'field list'
I would expect that I can pass a parameter to a stored procedure to generate a temporary table, and return the records of that temporary table. Even if I call that same procedure multiple times with different parameters on the same session.
The actual results are that when the stored procedure is called to generate the temporary table with a different table structure, it returns this error complaining about the column missing from the temporary table created in the previous invocation of that same stored procedure.
Error Code: 1054. Unknown column 'mySchema.aTable.the_column' in 'field list'
The only way I have found to prevent this error is
a. ending the jdbc connection and ending the server session
b. recompiling one of the stored procedures in the call stack
Recompiling is not viable. And ending the session seems unreasonable.
This seems like a defect. But would be interested to find if there is some way to get this to work.
This seems like a bug and you can report it directly to the MariaDB team at MariaDB bugs database.
A temporary solution is to use a prepared statement in the stored procedure GetATable2 (my test on MariaDB 10.3.16 to use EXECUTE IMMEDIATE):
...
CREATE PROCEDURE `GetATable2`(`_TableType` TINYINT)
BEGIN
CALL CreateATable(`_TableType`);
-- SELECT * FROM `aTable`;
EXECUTE IMMEDIATE 'SELECT * FROM `aTable`';
END$$
...
See dbfiddle.
I need to import data from an Access app into VFP tables. I have found solutions on the net for reading VFP data into Access but not the opposite. The connection string suggested by MS* seems to be missing something because I keep getting a dialog box asking me for the type of the source data
"Microsoft.ACE.OLEDB.12.0;Data Source=MyAccessApp.accdb;Persist Security Info=False;"
I would be grateful for any suggestion and/or explanation.
/bernard
if you are writing the program within VFP, then you can do a connect from VFP to the Access Database without the use of a "DSN", but requires a full connection string setting. Once that is done, you can query the data down into a local VFP cursor, then do what you want once it is in VFP... copy to a VFP table, query subsets of data, add records to another VFP table by whatever criteria you need to process.
nAccessHandle = sqlstringconnect( "DRIVER=Microsoft Access Driver (*.mdb, *.accdb); DBQ=C:\YourFullPath\YourAccessDatabase.accdb;" )
if nAccessHandle < 1
messagebox( "Invalid connection to access database" )
return
endif
*/ Valid handle, now, query down the data to a local VFP cursor
nSQLAns = SQLExec( nAccessHandle, "select * from YourAccessTable", "C_CursorInVFP" )
if nSQLAns < 1
messagebox( "Unable to get any data..." )
sqldisconnect( nAccessHandle )
return
endif
*/ Done with connection
sqldisconnect( nAccessHandle )
select C_CursorInVFP
copy to C:\SomeOtherPath\NowItsAVFPTable
*/ Or, query from it within VFP or loop from it...
select C_CursorInVFP
scan
*/ Look for records in VFP to do/ignore as needed...
select * from SomeVFPTable ;
where SomeKey = C_CursorInVFP.KeyFromAccessTable;
into cursor C_WasItFound readwrite
if reccount( "C_WasItFound" ) > 0
*/ Do what if it WAS found
else
*/ Do what if it WAS NOT found
endif
endscan
I am new to lua programming, I don't know about how to create database, In that create tables and retrieving it.
I want to create a database and tables only for first time. I need to insert the values into table and I want to retrieve the list from table.
I don't know how to start because I am beginner for lua programing. Can you guys please help me or suggest any tutorial to go through.
Please help me
Thank you,
Madan Mohan
I have followed http://wiki.garrysmod.com/?title=LUA:SQLite_Tutorial for sql lite & Lua tutorial and it was good. Before start, you need to install or copy lua dlls/modules into system path so that lua db module should be able to locate the database.
LuaSQL is good choice for client side library..
http://www.capricorn76.com/c76_scriptengine_docsdk/luasql/index.html
You mention Corona in your tags, although you didn't mention that in your question. That makes a difference, since Corona has built-in support for SQLite but not any other kind of database.
There's documentation on their website about how to use the database functionality: http://developer.anscamobile.com/content/data-storage
Note that their examples all use [[ ]] syntax for the database commands, but I find it looks better and is easier to understand to use " "
This is an old question but I will explain the way I do it:
Download Sqlite Browser (https://github.com/sqlitebrowser/sqlitebrowser/releases)
Create an empty database
Put it in the your project folder
In your code copy it to the system.DocumentsDirectory:
local function copyDatabase()
util.copyFile( "model.db", system.ResourceDirectory, db_name, system.DocumentsDirectory, false )
end
util.copyFile:
function M.copyFile( srcName, srcPath, dstName, dstPath, overwrite )
local results = false
local srcPath = M.doesFileExist( srcName, srcPath )
if ( srcPath == false ) then
return nil -- nil = source file not found
end
--check to see if destination file already exists
if not ( overwrite ) then
if ( M.doesFileExist( dstName, dstPath ) ) then
return 1 -- 1 = file already exists (dont overwrite)
end
end
--copy the source file to the destination file
local rfilePath = system.pathForFile( srcName, srcPath )
local wfilePath = system.pathForFile( dstName, dstPath )
local rfh = io.open( rfilePath, "rb" )
local wfh = io.open( wfilePath, "wb" )
if not ( wfh ) then
print( "writeFileName open error!" )
return false
else
--read the file from 'system.ResourceDirectory' and write to the destination directory
local data = rfh:read( "*a" )
if not ( data ) then
print( "read error!" )
return false
else
if not ( wfh:write( data ) ) then
print( "write error!" )
return false
end
end
end
results = 2 -- 2 = file copied successfully!
--clean up file handles
rfh:close()
wfh:close()
return results
end
To open the database:
local function openDatabase()
local path = system.pathForFile( db_name, system.DocumentsDirectory )
db = sqlite3.open( path )
end
To create some tables
local function createTables()
-- I dont create a Primary Key manually as Sqlite creates ROWID for us
db:exec( "CREATE TABLE IF NOT EXISTS categories (category TEXT, description TEXT, icon_location TEXT )" )
end
You can easily update and insert records using db:exec (db being your local variable to the database). db:exec returns a 0 if the query was completed without any errors.
How to handle a missing feature of SQLite: disable triggers?
I don't have it stored the name of triggers for a specific table.
For example how can I drop all triggers?
What would you do?
So here it is 2015 and there still is no 'disable triggers' in SQLite. For a mobile Application this can be problematic--especially if it's a corporate App requiring offline functionality and local data.
An initial data load can be slowed to crawl by trigger execution even when you don't wrap each insert in an individual transaction.
I solved this issue using SQLite SQL fairly simply. I have a settings table that doesn't participate in the init load. It holds 'list' of key/value pairs. I have one key called 'fireTrigger' with a bit value of 0 or 1. Every trigger I have has an expression that selects value and if it equals 1 it fires the trigger, otherwise it doesn't.
This expression is in addition to any expressions evaluated on the data relating to the trigger. e.g.:
AND 1 = (SELECT val FROM MTSSettings WHERE key = 'fireTrigger')
In simple clean effect this allows me to disable/enable the trigger with a simple UPDATE to the settings table
I wrote a very simple extension function to set a boolean value to true or false.
And a function to retrieve this value (GetAllTriggersOn()).
With this function I can define all my triggers like:
CREATE TRIGGER tr_table1_update AFTER UPDATE ON TABLE1 WHEN GetAllTriggersOn()
BEGIN
-- ...
END
SQLite stores schema (meta) information in the built-in sqlite_master table.
To get a list of available triggers use the below query:
SELECT name FROM sqlite_master
WHERE type = 'trigger' -- AND tbl_name = 'a_table_name'
Set a flag in your database and use it in the triggers WHEN condition.
Say you want to create a trigger on the "clients" table after an insert. You have created a table "trigger_settings" with a TINYINT "triggers_on" field - this is your flag. Then you can set the field to 0 if you want to turn off the filters and to 1 when you want to turn them back on.
Then you create your filter with a WHEN condition that checks the "triggers_on" field.
For example:
CREATE TRIGGER IF NOT EXISTS log_client_data_after_insert
AFTER INSERT
ON [clients]
WHEN (SELECT triggers_on FROM trigger_settings)=1
BEGIN
your_statement
END;
Maybe you can make a stored procedures for droping and creating them. Is that good for you ?
Expanding on Nick Dandoulakis's answer, you could drop all relevant triggers and then reinstate them before the transaction's completion:
BEGIN;
SELECT name, sql FROM sqlite_master WHERE type = 'trigger' AND tbl_name = 'mytable';
-- store all results
-- for each name: DROP TRIGGER $name;
-- do normal work
-- for each sql: execute the SQL verbatim
COMMIT;
Expanding other answers this is how i'm doing it. Take into account that this is disabling all triggers for all tables in the database except some of then used by spatialite
SQLITE_FILE=/tmp/my.sqlite
# Define output sql files as variables
CREATE_TRIGGER_SQL=/tmp/create_triggers.sql
DROP_TRIGGER_SQL=/tmp/drop_triggers.sql
## Dump CREATE TRIGGER statements to a file ##
# To wrap statements in a transaction
echo -e "BEGIN;\n\n" > "${CREATE_TRIGGER_SQL}"
# `SELECT sql` does not output semicolons, so we must concatenate them
sqlite3 -bail "${SQLITE_FILE}" "SELECT sql || ';' FROM sqlite_master WHERE type = 'trigger' AND (name NOT LIKE 'gid_%' AND name NOT LIKE 'ggi_%' AND name NOT LIKE 'ggu_%' AND name NOT LIKE 'gii_%' AND name NOT LIKE 'giu_%' AND name NOT LIKE 'vwgcau_%' AND name NOT LIKE 'vtgcau_%' AND name NOT LIKE 'gcau_%' AND name NOT LIKE 'geometry_columns_%' AND name NOT LIKE 'gcfi_%' AND name NOT LIKE 'gctm_%' AND name NOT LIKE 'vtgcfi_%' AND name NOT LIKE 'vwgcfi_%' AND name NOT LIKE 'vtgcs_%' AND name NOT LIKE 'vwgc_%' AND name NOT LIKE 'vtgc_%' AND name NOT LIKE 'gcs_%');" >> "${CREATE_TRIGGER_SQL}"
echo -e "\n\nCOMMIT;" >> "${CREATE_TRIGGER_SQL}"
## Dump DROP TRIGGER statements to a file ##
echo -e "BEGIN;\n\n" > "${DROP_TRIGGER_SQL}"
sqlite3 -bail "${SQLITE_FILE}" "SELECT 'DROP TRIGGER ' || name || ';' FROM sqlite_master WHERE type = 'trigger' AND (name NOT LIKE 'gid_%' AND name NOT LIKE 'ggi_%' AND name NOT LIKE 'ggu_%' AND name NOT LIKE 'gii_%' AND name NOT LIKE 'giu_%' AND name NOT LIKE 'vwgcau_%' AND name NOT LIKE 'vtgcau_%' AND name NOT LIKE 'gcau_%' AND name NOT LIKE 'geometry_columns_%' AND name NOT LIKE 'gcfi_%' AND name NOT LIKE 'gctm_%' AND name NOT LIKE 'vtgcfi_%' AND name NOT LIKE 'vwgcfi_%' AND name NOT LIKE 'vtgcs_%' AND name NOT LIKE 'vwgc_%' AND name NOT LIKE 'vtgc_%' AND name NOT LIKE 'gcs_%');" >> "${DROP_TRIGGER_SQL}"
echo -e "\n\nCOMMIT;" >> "${DROP_TRIGGER_SQL}"
## Execute like ##
sqlite3 -bail /"${SQLITE_FILE}" < "${DROP_TRIGGER_SQL}"
# do things
sqlite3 -bail /"${SQLITE_FILE}" < "${CREATE_TRIGGER_SQL}"