Sqlite3 Do I have to attach a database on every connection? - sqlite

The following piece of code creates two databases:
import sqlite3
db = 'brazil'
conn = sqlite3.connect(db+'.db')
c = conn.cursor()
qCreate = """
CREATE TABLE states
(zip_id numeric NOT NULL,
state_name text NOT NULL,
CONSTRAINT pk_brazil
PRIMARY KEY (zip_id) """
c.execute(qCreate)
conn.commit()
conn.close()
db = 'city'
conn = sqlite3.connect(db+'.db')
c = conn.cursor()
qCreate = """CREATE TABLE rio_de_janeiro
(zip_id numeric NOT NULL,
beach_name text NOT NULL,
CONSTRAINT pk_rio
PRIMARY KEY (zip_id)
"""
c.execute(qCreate)
conn.commit()
conn.close()
The following piece of code attaches the database RIO to the database BRAZIL and prints all the databases (Rio and Brazil).
db = 'brazil'
conn = sqlite3.connect(db+'.db')
c = conn.cursor()
qCreate = """ATTACH DATABASE ? AS competition """
c.execute(qCreate, ('rio.db',))
c.execute("PRAGMA database_list")
data = c.fetchall()
print data
conn.commit()
conn.close()
However the following piece of code prints only Brazil database:
db = 'brazil'
conn = sqlite3.connect(db+'.db')
c = conn.cursor()
c.execute("PRAGMA database_list")
data = c.fetchall()
print data
conn.commit()
conn.close()
The attached database is no longer attached.
The sqlite3 documentation hints on these lines:
The ATTACH DATABASE statement adds another database file to the current database connection.
Do I have to attach the database every time?
I planed to use attached databases for schemas, but maybe I should try something else?
I am using python in Pythonista App in iOS

Almost all settings you can change in SQLite apply only to the current connection, i.e., are not saved in the database file.
So you have to re-ATTACH any databases whenever you have re-opened the main database.
Using attached databases makes sense only if you must use multiple database files due to some external constraint. In most cases, you should use only a single database.
SQLite does not have schemas. If you want to emulate them with attached databases, you have to live with the limitations of that approach.

Related

How to connect to Teradata database using Dask?

The pandas equivalent code for connecting to Teradata, I have used is:
database = config.get('Teradata connection', 'database')
host = config.get('Teradata connection', 'host')
user = config.get('Teradata connection', 'user')
pwd = config.get('Teradata connection', 'pwd')
with teradatasql.connect(host=host, user=user, password=pwd) as connect:
query1 = "SELECT * FROM {}.{}".format(database, tables)
df = pd.read_sql_query(query1, connect)
Now, I need to use the Dask library for loading big data as an alternative to pandas.
Please suggest a method to connect the same with Teradata.
Teradata appears to have a sqlalchemy engine, so you should be able to install that, set your connection string appropriately and use Dask's existing from_sql function.
Alternatively, you could do this by hand: you need to decide on a set of conditions which will partition the data for you, each partition being small enough for your workers to handle. Then you can make a set of partitions and combine into a dataframe as follows
def get_part(condition):
with teradatasql.connect(host=host, user=user, password=pwd) as connect:
query1 = "SELECT * FROM {}.{} WHERE {}".format(database, tables, condition)
return pd.read_sql_query(query1, connect)
parts = [dask.delayed(get_part)(cond) for cond in conditions)
df = dd.from_delayed(parts)
(ideally, you can derive the meta= parameter for from_delayed beforehand, perhaps by getting the first 10 rows of the original query).

Add dictionary or list to Sqlite3 - Gives error operation parameter must be str

Ok I am new to sqlite and python in general so please be nice =)
I have a simple dictionary -
time = data[0]['timestamp']
price = data[0]['price']
myprice = {'Date':time,'price':price}
myprice looks like this (time is a timestamp) -
{'Date': 1553549093, 'price': 1.7686}
I now want to add the data to sqlite3 database...so I created this -
#Create database if not exist...
db_filename = 'mydb_test.db'
connection = sqlite3.connect(db_filename)
#Get a SQL cursor to be able to execute SQL commands...
cursor = connection.cursor()
#Create table
sql = '''CREATE TABLE IF NOT EXISTS TEST
(PID INTEGER PRIMARY KEY AUTOINCREMENT,
DATE TIMESTAMP,
PRICE FLOAT)'''
#Now lets execute the above SQL
cursor.execute(sql)
#Insert data in sql
sql2 = ("INSERT INTO GBPCAD VALUES (?,?)", [(myprice['Date'],myprice['price'])])
cursor.execute(sql2)
cursor.commit()
connection.close()
But when executing this code I get ValueError: operation parameter must be str
What am I doing wrong?
Pass the arguments of the insert statement in execute():
sql2 = "INSERT INTO GBPCAD (DATE, PRICE) VALUES (?,?)"
cursor.execute(sql2, (myprice['Date'], myprice['price']))
Also include in the insert statement the names of the columns.

Update always encrypted column from decrypted column

I would like to encrypt an existing database column with always encrypted. My project is a ASP.NET project using code first and database is SQL Server. The database has already data. I created a migration to achieve my goal.
First I tried to alter the column type, using the following.
ALTER TABLE [dbo].[TestDecrypted] ALTER COLUMN [FloatCol] [float] ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL
I got the following error.
Operand type clash: float is incompatible with float encrypted with (encryption_type = 'RANDOMIZED', encryption_algorithm_name = 'AEAD_AES_256_CBC_HMAC_SHA_256', column_encryption_key_name = 'CEK_Auto1', column_encryption_key_database_name = 'TestEncrypt')
Then I decided to created another column and migrate the data.
ALTER TABLE [dbo].[TestDecrypted] ADD [FloatCol2] [float] ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = [FloatCol]
And I got the same error.
After I looked at this, I noticed that it is possible to insert data like the following
DECLARE #floatCol FLOAT = 1.1
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #floatCol
But if I try to obtain the value from my existing column, it fails.
DECLARE #floatCol FLOAT = (SELECT TOP 1 FloatCol FROM TestDecrypted)
UPDATE [dbo].[TestDecrypted] SET FloatCol2 = #floatCol
The error follows.
Encryption scheme mismatch for columns/variables '#floatCol'. The encryption scheme for the columns/variables is (encryption_type = 'PLAINTEXT') and the expression near line '4' expects it to be (encryption_type = 'RANDOMIZED', encryption_algorithm_name = 'AEAD_AES_256_CBC_HMAC_SHA_256', column_encryption_key_name = 'CEK_Auto1', column_encryption_key_database_name = 'TestEncrypt').
Does anyone knows how can I achieve my goal?
Update 1
#Nikhil-Vithlani-Microsoft did some interesting suggestions.
Always Encrypted Wizard in SSMS - I would like to achieve my goal with code first migrations, so this idea does not fit.
SqlBulkCopy - It does not work inside migrations, because the new column will only exist after all 'Up' method is run. Therefore we cannot insert data into this column in this way inside this method.
Anyway, his suggestions drove me to another attempt: obtain the decrypted values and update the encrypted column with them.
var values = new Dictionary<Guid, double>();
var connectionString = ConfigurationManager.ConnectionStrings["MainDb"].ConnectionString;
using (var sourceConnection = new SqlConnection(connectionString))
{
var myCommand = new SqlCommand("SELECT * FROM dbo.TestDecrypted", sourceConnection);
sourceConnection.Open();
using (var reader = myCommand.ExecuteReader())
{
while (reader.Read())
{
values.Add((Guid)reader["Id"], (double)reader["FloatCol"]);
}
}
}
Sql("ALTER TABLE [dbo].[TestDecrypted] ADD [FloatCol2] [float] ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL");
foreach (var valuePair in values)
{
// The error occurs here
Sql($#"DECLARE #value FLOAT = {valuePair.Value}
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #value WHERE Id = '{valuePair.Key}'");
}
In fact, I did not try to create another column and to migrate the data, as mentioned in an example above. I tried it only on SSMS.
And now I got a different error.
Transaction (Process ID 57) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
I tried to do it without encrypting the new column, and it worked properly.
Any idea why this error occurs?
You will have to do the always encrypted related migration outside of entity framework. This blog should help
https://blogs.msdn.microsoft.com/sqlsecurity/2015/08/27/using-always-encrypted-with-entity-framework-6/
If you want to encrypt an existing column, you can use Always Encrypted Wizard in SSMS, or use this article that explains how to migrate existing data.
Also, please note that doing bulk inserts through a C# (.NET 4.6.1+ client) app is supported.
You can do this in c# using SqlBulkCopy specifically using SqlBulkCopy.WriteToServer(IDataReader) Method.
Create a new table (encryptedTable) with the same schema as that of your plaintext table (unencryptedTable) but with the encryption turned on for the desired columns.
Do select * from unencryptedTable to load the data in a SqlDataReader then use SqlBulkCopy to load it to the encryptedTable using SqlBulkCopy.WriteToServer(IDataReader) Method
For example,
Plaintext Table
CREATE TABLE [dbo].[Patients](
[PatientId] [int] IDENTITY(1,1),
[SSN] [char](11) NOT NULL)
Encrypted Table
CREATE TABLE [dbo].[Patients](
[PatientId] [int] IDENTITY(1,1),
[SSN] [char](11) COLLATE Latin1_General_BIN2
ENCRYPTED WITH (ENCRYPTION_TYPE = DETERMINISTIC,
ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256',
COLUMN_ENCRYPTION_KEY = CEK1) NOT NULL)
As for why your method does not work,
when you use parameterization for always encrypted, the right hand side (RHS) of the declare statement needs to be a literal. Because the driver will identify the literal and encrypt it for you. So, the following will not work, since RHS is a sql expression and cannot be encrypted by the driver
DECLARE #floatCol FLOAT = (SELECT TOP 1 FloatCol FROM TestDecrypted)
UPDATE [dbo].[TestDecrypted] SET FloatCol2 = #floatCol
Update:
The following code will not work because parameterization for Always Encrypted only applies to SSMS
foreach (var valuePair in values)
{
// The error occurs here
Sql($#"DECLARE #value FLOAT = {valuePair.Value}
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #value WHERE Id = '{valuePair.Key}'");
}
However, if you rewrite your code as follows, that should work
foreach (var valuePair in values)
{
SqlCommand cmd = _sqlconn.CreateCommand();
cmd.CommandText = #"UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #FloatVar WHERE Id = '{valuePair.Key}'");";
SqlParameter paramFloat = cmd.CreateParameter();
paramFloat.ParameterName = #"#FloatVar";
paramFloat.DbType = SqlDbType.Float;
paramFloat.Direction = ParameterDirection.Input;
paramFloat.Value = floatValue;
cmd.Parameters.Add(paramFloat);
cmd.ExecuteNonQuery();
}
Hope that helps, if you have additional question, please leave them in the comments.

Pyqt5 Sqlite, interface stop responding when inserting into database

I am using self.button.clicked.connect(self.createdb) connection, where items from listwidget and combox and line are inserted into a an sqlite database.
Here is my code:
def createdb(self):
db = QtSql.QSqlDatabase.addDatabase('QSQLITE')
db.setDatabaseName('testing.db3')
db.open()
query = QtSql.QSqlQuery()
query.exec_("create table sas(test1 varchar(20), test2 varchar(20))")
for i in range(self.listWidget.count()):
query.exec_('insert into sas (test1, test2) values({}","{}")'.format(self.listWidget.item(i).text()+self.comboBox_11.currentText(), self.lineEdit.text()))
QtSql.QSqlDatabase.commit
QtSql.QSqlDatabase.close
self.statusbar.showMessage('Finished')
It takes ages to insert 300 values and it crushes when i want to insert 10k or more values. Any ideas why?

What does "Orient a SQLite database" mean?

In class, we got a code to Orient a SQLite database. I kind of get the code (below), but want to get a conceptual understanding of what "Orient" means.
import sqlite3
conn = sqlite3.connect('example3.db')
c = conn.cursor()
c.execute('SELECT name FROM sqlite_master WHERE type=\'table\'') #list all tables therein
print c.fetchall()
c.execute('SELECT sql FROM sqlite_master WHERE type=\'table\' AND name=\'students\'') #list columns in table 'students'
print c.fetchall()
conn.close()

Resources