Is there a way to trigger a stored procedure in redshift from airflow?
Best scenario would be to not use a python operator, but I haven't found a reshift operator in airflow version < 2.
In Airflow <2.1 there is no RedShiftSqlOperator because RedShift is compatible with PostgreSQL so you can just use PostgresOperator:
from airflow.providers.postgres.operators.postgres import PostgresOperator
PostgresOperator(
sql='SELECT * FROM my_table',
postgres_conn_id='redshift_default',
task_id='task',
)
That said, in more recent versions (Airflow >= 2.1) since some users were struggling with this question as well (See GitHub issue) Airflow added RedshiftSQLOperator which is available in Amazon provider version 2.4.0 :
pip install apache-airflow-providers-amazon>=2.4.0
Then you can use the operator as:
from airflow.providers.amazon.aws.operators.redshift import RedshiftSQLOperator
setup__task_create_table = RedshiftSQLOperator(
task_id='task',
redshift_conn_id="redshift_default"
sql="SELECT * FROM my_table",
)
Is there an easy way to enable the FTS5 extension for SQLite3 installed with Brew? Some older posts say there should be an install option --with-fts5, however:
$ brew reinstall sqlite3 --with-fts5
...
Error: invalid option: --with-fts5
The fts3_tokenizer is not enabled. I assume there must be an easy way to install/enabled the extension with Brew without compiling from source outside of Brew.
$ sqlite3
SQLite version 3.35.5 2021-04-19 18:32:05
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .dbconfig
defensive off
dqs_ddl on
dqs_dml on
enable_fkey off
enable_qpsg off
enable_trigger on
enable_view on
fts3_tokenizer off
legacy_alter_table off
legacy_file_format off
load_extension on
no_ckpt_on_close off
reset_database off
trigger_eqp off
trusted_schema on
writable_schema off
$ brew info sqlite3
sqlite: stable 3.35.5 (bottled) [keg-only]
Command-line interface for SQLite
https://sqlite.org/
/usr/local/Cellar/sqlite/3.35.5 (11 files, 4.2MB)
Built from source on 2021-05-18 at 08:54:33
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/sqlite.rb
...
Try to use CFLAGS environment:
CFLAGS="-DSQLITE_ENABLE_FTS5" brew reinstall sqlite
Edited:
Rebuild to enable fts5 is unnecessary. The sqlite 3.35.5 package is already fts5 module enabled.
$ brew fetch sqlite
...
$ tar xzf ~/Library/Caches/Homebrew/downloads/61d40ad2021e894bcf4c7475eea2dbbfee14c4426b1bbb1816c4055ad1c70b50--sqlite--3.35.5.catalina.bottle.tar.gz -O sqlite/3.35.5/lib/libsqlite3.0.dylib \
| strings - | grep '^fts5: 20\|trigram'
trigram
fts5: 2021-04-19 18:32:05 1b256d97b553a9611efca188a3d995a2fff712759044ba480f9a0c9e98fae886
I wrote you on the trac mailing list too but will post here too.
If you check https://sqlite.org/fts5.html there seems to be a option --enable-fts5 but it also seems disabled by the fault. That said they point to a " amalgamation" (there is a link to it in the page) where you can use this option if you compile the " amalgamation".
Markus
FTS5 is enabled. I was mistaken about the source of the problem I'm experiencing.
>>> import sqlite3
>>> import pprint
>>> db = sqlite3.connect(':memory:')
>>> cursor = db.execute('PRAGMA COMPILE_OPTIONS')
>>> pprint.pprint(cursor.fetchall())
[(u'BUG_COMPATIBLE_20160819',),
(u'COMPILER=clang-12.0.5',),
(u'DEFAULT_CACHE_SIZE=2000',),
(u'DEFAULT_CKPTFULLFSYNC',),
(u'DEFAULT_JOURNAL_SIZE_LIMIT=32768',),
(u'DEFAULT_PAGE_SIZE=4096',),
(u'DEFAULT_SYNCHRONOUS=2',),
(u'DEFAULT_WAL_SYNCHRONOUS=1',),
(u'ENABLE_API_ARMOR',),
(u'ENABLE_COLUMN_METADATA',),
(u'ENABLE_DBSTAT_VTAB',),
(u'ENABLE_FTS3',),
(u'ENABLE_FTS3_PARENTHESIS',),
(u'ENABLE_FTS3_TOKENIZER',),
(u'ENABLE_FTS4',),
(u'ENABLE_FTS5',),
(u'ENABLE_JSON1',),
(u'ENABLE_LOCKING_STYLE=1',),
(u'ENABLE_PREUPDATE_HOOK',),
(u'ENABLE_RTREE',),
(u'ENABLE_SESSION',),
(u'ENABLE_SNAPSHOT',),
(u'ENABLE_SQLLOG',),
(u'ENABLE_STMT_SCANSTATUS',),
(u'ENABLE_UNKNOWN_SQL_FUNCTION',),
(u'ENABLE_UPDATE_DELETE_LIMIT',),
(u'HAS_CODEC_RESTRICTED',),
(u'HAVE_ISNAN',),
(u'MAX_LENGTH=2147483645',),
(u'MAX_MMAP_SIZE=1073741824',),
(u'MAX_VARIABLE_NUMBER=500000',),
(u'OMIT_AUTORESET',),
(u'OMIT_LOAD_EXTENSION',),
(u'STMTJRNL_SPILL=131072',),
(u'THREADSAFE=2',),
(u'USE_URI',)]
I want to connect to MongoDB and query from MongoDB collection.
I have installed Mongodb by installing following libraries:
pip install pymongo
pip install robotframework-MongoDBLibrary
It installed properly.
after that i wrote the following statements in RIDE to query from mongodb
Connect to MongoDB dbHost=${host} dbPort=${port}
I just ran this statement, test script is pass.
Then to query, i have added one more statement as below:
${fields} = Retrieve Mongodb Records With Desired Fields ${MongoDBName} ${MongoDBCollection} {} profileDetails.customerCategory.masterCode return__id=False
after execution, i got the following error:
OperationFailure: database error: not authorized for query on clmpreprod.Profile
Normally, in java to connect to Mongodb we will follow the below steps
MongoClient mongoClient = new MongoClient(Arrays.asList(
new ServerAddress(MONGO_DBURL, 27017),
new ServerAddress(MONGO_DBURL, 27018),
new ServerAddress(MONGO_DBURL, 27019)));
DB database = mongoClient.getDB(MONGO_DBNAME);
boolean auth = database.authenticate(MONGO_USERNAME,MONGO_PASSWORD).toCharArray());
DBCollection collection = getCollection(MONGO_CUSTOMER_COLLECTION, database);
List<DBObject> obj = collection.find(queryDBParams, returnDBParams).sort(sortDBParams).limit(1).toArray();
Can anyone help me what keyword or what series of steps need to follow to use in robotframework for database authentication and then query
Thanks
Sarada
I found the answer in MongoDBLibrary documentation, here the link RobotFramework-MongoDBLibrary
Syntax is:
Connect To MondoDB | mongodb://admin:admin#192.20.33.226 | 27017 | 10 | None | <type 'dict'> | False |
Thanks
Sarada
i using start terminal
-macbook:sqlTest user1$ sqlite3 sqlTest.sqlite
SQLite version 3.7.13 2012-07-17 17:46:21
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> ATTACH DATABASE 'encrypted.sqlite' AS encrypted KEY 'testkey';
sqlite> SELECT sqlcipher_export('encrypted');
Error: no such function: sqlcipher_export
sqlite>
what makes no such function: sqlcipher_export?
As answered on the mailing list:
The first step is to build the sqlcipher command line tool, as described here:
http://sqlcipher.net/introduction/
Once you have done this, you should run the command like this:
$ ./sqlcipher sqlTest.sqlite
or
$ /full/path/to/sqlcipher/sqlcipher sqlTest.sqlite
On unix systems, if you don't provide an explicit path for a command, the system will look for the program in $PATH. On OSX, the system ships with a sqlite3 command, so you've probably been using that instead of the version compiled with SQLCipher. Please let us know if that resolves the problem. Thanks!
I'm trying to use System.Data.SQLite to connect to a sqlite 2 file. I use SQLiteConnectionStringBuilder class to build the connection string but I couldn't set the version to 2 (exception). In the System.Data.SQLite document, it says the version is always 3. So how can I connect to sqlite 2? Should I just use a string to build the connection string instead of using SQLiteConnectionStringBuilder class?
Thanks.
SQLite 3 and SQLite 2 are not compatible.
You should convert the database into the SQLite 3 format.
This can be done with the sqlite2 and sqlite3 command-line tools:
sqlite2 olddb .dump | sqlite3 newdb