I have two sqlalchemy scripts, one that creates a database and a few tables and another that selects data from them.
create_database.py
from sqlalchemy import create_engine, Table, Column, Integer, String, MetaData, ForeignKey, select
engine = create_engine('sqlite:///test.db', echo=True)
metadata = MetaData()
addresses = Table ('addresses', metadata,
Column('id', Integer, primary_key=True),
Column('user_id', None, ForeignKey('users.id')),
Column('email_addresses', String, nullable=False)
)
users = Table ('users', metadata,
Column('id', Integer, primary_key=True),
Column('name', String),
Column('fullname', String),
)
metadata.create_all(engine)
select.py
from sqlalchemy import create_engine, select
engine = create_engine('sqlite:///test.db', echo=True)
conn = engine.connect()
s = select([users])
result = conn.execute(s)
I am able to run the create_database.py script but when I run the select.py script I get the following error
$ python select.py
Traceback (most recent call last):
File "select.py", line 5, in <module>
s = select([users])
I am able to run the select statement from within the create_database.py by appending the following to create_database.py
conn = engine.connect()
s = select([users])
result = conn.execute(s)
How can I run the select statements from a separate script than create_database.py
The script select.py does not see users and addresses defined in create_database.py. Import them in select.py before using them.
In select.py:
from create_database import users, addresses
## Do something with users and addresses
Related
I don't know how to open a db.sqlite3 file in reader-friendly way.
I hope the data in it would be shown in tables
Upload your file here and get the tabulated result:
http://inloop.github.io/sqlite-viewer/
OR run a Python script like below
def create_connection(db_file):
""" create a database connection to the SQLite database specified by the db_file :param db_file: database file :return: Connection object or None """ conn = None
try:
conn = sqlite3.connect(db_file) except Error as e:
print(e) return conn
def select_all_tasks(conn):
""" Query all rows in the tasks table :param conn: the Connection object :return: """
cur = conn.cursor()
cur.execute("SELECT * FROM tasks") rows = cur.fetchall() for row in rows: print(row)
I understand that sqlite doesn't support RETURNING at least that is what sqlAlchemy is telling me:
sqlalchemy.exc.CompileError: RETURNING is not supported by this dialect's statement compiler.
I get this error when using sqlAlchemy's Core library. Here is a code example:
from sqlalchemy.engine.url import URL
from sqlalchemy import create_engine, MetaData
from sqlalchemy import Table, Column, Integer, String
engine = create_engine('sqlite:///:memory:', echo=False)
# create table
meta = MetaData(engine)
table = Table('userinfo', meta,
Column('id', Integer, primary_key=True),
Column('first_name', String),
Column('age', Integer),
)
meta.create_all()
# generate rows
data = [{'first_name': f'Name {i}', 'age': 18+i} for i in range(10)]
# this seems to work on PostgreSQL only
stmt = table.insert().values(data).returning(table.c.id)
for rowid in engine.execute(stmt).fetchall():
print(rowid['id'])
Now, when I use similar code with sqlAlchemy's ORM library the IDs are returned. Here is the source code:
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String
from sqlalchemy import ForeignKey
from sqlalchemy.orm import sessionmaker, scoped_session
from sqlalchemy.orm import relationship
Base = declarative_base()
class UserInfo(Base):
__tablename__ = "userinfo"
id = Column(Integer, primary_key=True)
first_name = Column(String)
age = Column(Integer)
engine = create_engine('sqlite:///:memory:', echo=False)
Base.metadata.create_all(engine)
session = scoped_session(sessionmaker(bind=engine))
data = [dict(first_name=f'Name {i}', age=18+1) for i in range(10)]
session.bulk_insert_mappings(UserInfo, data, return_defaults=True)
session.commit()
print([s['id'] for s in data])
How come that this is working while the Core one is not? When I look at the generated sql I don't see.
After some digging I found this link
In this document the use of bulk_insert_mappings is just Batched INSERT statements via the ORM "bulk", using dictionaries.. When setting return_defaults=True I assume sqlalchemy is repeatedly calling upon last row id. Hence the IDs are available.
Ok I am new to sqlite and python in general so please be nice =)
I have a simple dictionary -
time = data[0]['timestamp']
price = data[0]['price']
myprice = {'Date':time,'price':price}
myprice looks like this (time is a timestamp) -
{'Date': 1553549093, 'price': 1.7686}
I now want to add the data to sqlite3 database...so I created this -
#Create database if not exist...
db_filename = 'mydb_test.db'
connection = sqlite3.connect(db_filename)
#Get a SQL cursor to be able to execute SQL commands...
cursor = connection.cursor()
#Create table
sql = '''CREATE TABLE IF NOT EXISTS TEST
(PID INTEGER PRIMARY KEY AUTOINCREMENT,
DATE TIMESTAMP,
PRICE FLOAT)'''
#Now lets execute the above SQL
cursor.execute(sql)
#Insert data in sql
sql2 = ("INSERT INTO GBPCAD VALUES (?,?)", [(myprice['Date'],myprice['price'])])
cursor.execute(sql2)
cursor.commit()
connection.close()
But when executing this code I get ValueError: operation parameter must be str
What am I doing wrong?
Pass the arguments of the insert statement in execute():
sql2 = "INSERT INTO GBPCAD (DATE, PRICE) VALUES (?,?)"
cursor.execute(sql2, (myprice['Date'], myprice['price']))
Also include in the insert statement the names of the columns.
I have the following workflow:
Get number from MySqlOperator (dynamic)
Get value stored in Variable (static)
Create a string based on both.
Use the string as sql command for MySqlToGoogleCloudStorageOperator.
Now, it is proven to be difficult.
This is my code:
VALUE_FROM_VARIABLE = Variable.get("my_var")
query = 'SELECT ... FROM orders where orders_id>{0}
and orderid<{1};'.format(VALUE_FROM_MySqlOperator, VALUE_FROM_VARIABLE)
file_name = ...
import_orders_op = MySqlToGoogleCloudStorageOperator(
task_id='import_orders_and_upload_to_storage',
mysql_conn_id='mysql_con',
google_cloud_storage_conn_id='gcp_con',
sql=query,
bucket=GCS_BUCKET_ID,
filename=file_name,
dag=dag)
My problem here is that I can't access the MySqlOperator XCOM which store the number in need for the query.
So I tried to access it in PythonOperator and build the query string as follows:
def func(ds, **kwargs):
ti = kwargs['ti']
VALUE_FROM_MySqlOperator = str(ti.xcom_pull(task_ids='mySQL_task')) # get the XCOM of MySqlOperator
query = 'SELECT ... FROM orders where orders_id>{0}
and orderid<{1};'.format(VALUE_FROM_MySqlOperator, VALUE_FROM_VARIABLE)
return query
py_op = PythonOperator(
task_id='py_op_task',
provide_context=True,
python_callable=func,
xcom_push=True,
dag=dag)
But now I can't pass the new generated query to the MySqlToGoogleCloudStorageOperator because I can't read the XCOM inside this operator.
How can I get out of this ?
SQL operators are intending to execute queries which are not returning any values. You can use such operators(for example) to move data from stage table to production.
In my opinion, try to avoid creating workflows which are using XCOMS.
If you need to query data from database you can use Hooks and Connections
Untested code is below
VALUE_FROM_VARIABLE = Variable.get("my_var")
query_to_retrieve = "SELECT item FROM table"
from airflow.hooks.mysql_hook import MySqlHook
#here we importing hook, using connection and get first row
VALUE_FROM_MySQL = MySqlHook(mysql_conn_id='mysql_default').get_first(query_to_retrieve)[0]
query = 'SELECT ... FROM orders where orders_id>{0}
and orderid<{1};'.format(VALUE_FROM_MySQL, VALUE_FROM_VARIABLE)
The following piece of code creates two databases:
import sqlite3
db = 'brazil'
conn = sqlite3.connect(db+'.db')
c = conn.cursor()
qCreate = """
CREATE TABLE states
(zip_id numeric NOT NULL,
state_name text NOT NULL,
CONSTRAINT pk_brazil
PRIMARY KEY (zip_id) """
c.execute(qCreate)
conn.commit()
conn.close()
db = 'city'
conn = sqlite3.connect(db+'.db')
c = conn.cursor()
qCreate = """CREATE TABLE rio_de_janeiro
(zip_id numeric NOT NULL,
beach_name text NOT NULL,
CONSTRAINT pk_rio
PRIMARY KEY (zip_id)
"""
c.execute(qCreate)
conn.commit()
conn.close()
The following piece of code attaches the database RIO to the database BRAZIL and prints all the databases (Rio and Brazil).
db = 'brazil'
conn = sqlite3.connect(db+'.db')
c = conn.cursor()
qCreate = """ATTACH DATABASE ? AS competition """
c.execute(qCreate, ('rio.db',))
c.execute("PRAGMA database_list")
data = c.fetchall()
print data
conn.commit()
conn.close()
However the following piece of code prints only Brazil database:
db = 'brazil'
conn = sqlite3.connect(db+'.db')
c = conn.cursor()
c.execute("PRAGMA database_list")
data = c.fetchall()
print data
conn.commit()
conn.close()
The attached database is no longer attached.
The sqlite3 documentation hints on these lines:
The ATTACH DATABASE statement adds another database file to the current database connection.
Do I have to attach the database every time?
I planed to use attached databases for schemas, but maybe I should try something else?
I am using python in Pythonista App in iOS
Almost all settings you can change in SQLite apply only to the current connection, i.e., are not saved in the database file.
So you have to re-ATTACH any databases whenever you have re-opened the main database.
Using attached databases makes sense only if you must use multiple database files due to some external constraint. In most cases, you should use only a single database.
SQLite does not have schemas. If you want to emulate them with attached databases, you have to live with the limitations of that approach.