I'm using python3.6 and pymysql 0.7.11.
The call to rollback rolls a transaction back, according to the documentation.
A call to commit finalises a transaction.
My question is whether after a call to rollback it is prudent to call commit ... ?
this is not really a python question but about mysql transactions.
you can read about it at http://www.mysqltutorial.org/mysql-transaction.aspx for example.
in short: you need to start a transaction to rollback or commit it.
usually autocommit is enabled if you don't want to use transactions manually .. in that case mysql creates a transaction for every statement.
Related
I've been reading a lot lately and I am now very confused how transactions and locking are working and how are related to each other.
When working with SQLite, imagine the following flow:
begin SQLite transaction
run a select statement on a SQLite connection to return values
if condition is met on the returned values, go to step #4, otherwise go to step #5
do the update
commit SQLite transaction
If two threads run same code, is there a possibility that in one of the threads could get what is called a "dirty read" meaning, between the step #3 and step #4, the other thread could run the update (step #4)?
Yes, it's called isolation level: https://www.sqlite.org/isolation.html
I probably misundertood something about SAVEPOINTS in SQLite.
I am using C API in an application, where we run a bunch of inserts, and in case something fails, we should give up on all the inserts alltogether.
I am aware I could BEGIN TRANSACTION for such a simple scenario, but I fear that the scenario could get a more complicated, and nesting might become a requirement, that's why I went for SAVEPOINTS.
Anyway, here is an extract of SQL statements I run:
SQL> SAVEPOINT SAVEPOINT_20170524_172706;
SQL> INSERT, SELECT STATEMENT (no COMMIT or END TRANSACTION)
SQL> ROLLBACK TO SAVEPOINT SAVEPOINT_20170524_172706;
SQL> RELEASE SAVEPOINT_20170524_172706;
Basically I create a new savepoint based on the timestamp, before I start inserting and selecting data from the database.
Then one operation fails and I need to bail out, so I rollback to the savepoint I just created.
In the end I want to get rid of the savepoint I wont need anymore, since I dont want to clutter the database with useless savepoints, hence I ran RELEASE . In this case I find myself with the database filled with all the data inserted by statements that were supposed to be rolled back.
If I dont execute the RELEASE statement, then the database looks just fine, but I wonder what happens with the abandoned SAVEPOINT which will never be referenced anymore.
Which wrong assumption am I making? What happens to SAVEPOINTS if I dont release them, are they going to be 'dropped' as I close the 'connection' to the DB file?
I had PRAGMA journal_mode = OFF
For a particular requirement, I will have to iterate through a list of 50000 records and insert them into database. The requirement is that if any one of the 50000 records fail, all the other records should be rollback. And hence we did not involve any commit in the processing. But this resulted in the following error:
[2/1/16 14:01:47:939 CST] 000000be SystemOut O ERROR
org.springframework.jdbc.UncategorizedSQLException:
PreparedStatementCallback; uncategorized SQLException for SQL [INSERT
INTO ...) VALUES (...)]; SQL state [null]; error code [0]; Current
thread has not commited in more than [120] seconds and may incur
unwanted blocking locks. Please refactor code to commit more
frequently.
Now, when we implemented batching - We are using PreparedStatement.executeBatch() method to insert data in batches of 100, the above error doesn't arise. The autoCommit is set to true by default for the batching, so a commit happens after every batch execution.
Could anyone suggest how we can handle the rollback mechanism in the above case. If 50th batch execution fails, then we want all the previous 49 batch executions to be reverted. We are using Spring Data/JDBC, Oracle 11g database, WebSphere application server. I have read somewhere that the above 120 seconds timeout for commit can also be configured in the JTA settings of WebSphere. Is it so? Please suggest any alternatives or other possible solutions.
Thank you in advance.
You must set autocommit to false and only commit at the end if all your batches executed successfully.
I am setting up an automated process to sqoop from an oracle table to an hdfs directory with this command:
sqoop-import --connect jdbc:oracle:thin:#redacted.company.com:1234/db --username redacted --password secret123 --num-mappers 1 --table table --target-dir /data/destination/directory/ --as-avrodatafile --compress --compression-codec org.apache.hadoop.io.compress.BZip2Codec
Unfortunately, I'm getting the following error message:
Error:java.io.IOException: SQLException in nextKeyValue
...
Caused by: java.sql.SQLException: ORA-01555: snapshot too old: rollback segment number 336 with name "_SYSSMU336_879580159$" too small
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:447)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:951)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:513)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:208)
The business requirement I am attempting to fulfill is that the entire table is imported into our hdfs. Since we do not own or administrate this database, I lack control over the UNDO tablespace and related parameters. The job is scheduled to run at 1am which is not a peak time, but since automated processes touch it, I cannot coax people to stop using it during the job.
How should I modify my sqoop-import statement to avoid this error?
It is not a Sqoop issue. You would get the same error executing the same statment directly on Oracle. It is an undo tablespace issue. You have to get your query faster or you have to increase the Oracle undo tablespace size.
List of possible fixes:
Schedule your task when there are less database activity (maybe even
ask people to stop working for a while).
Optimize the query that is failing with this error to read less data
and take less time Increase the size of the UNDO tablespace.
Increase the size of the UNDO_RETENTION parameter.
Set the UNDO tablespace in GUARANTEE mode.
If you are exporting a table, consider exporting with the
CONSISTENT=no parameter.
Do not commit inside a cursor loop
Regards
Giova
Usage of --num-mappers=10 (i.e. increased parallelism) was sufficient enough to overcome the problem in this instance without impacting the source too much.
Additionally, adding the --direct parameter will cause Sqoop to use an Oracle specific connector which will speed things up further, and will be added to my solution as soon as I convince the DBA on that database to open up the necessary privileges. Direct also supports the option -Doraoop.import.consistent.read={true|false} which seems to mirror the Oracle export utility's CONSISTENT parameter in function (note, defaults to false), in the sense that the undo tablespace would not be used to try to preserve consistency, eliminating the need to race to import before the Undo tablespace fills up altogether.
I'm Altering multiple sqlite tables with SQL script by calling ExecuteNonQuery. I want to do this operation in transaction and want to roll it back when anything fails.
I looked at BEGIN TRANSACTION and its clear that I have to call ROLLBACK TRANSACTION when anything goes wrong. But I don't know how could TRY...CATCH (Transact-SQL) kind of thing here.
NOTE: Whole of Sql Script file (which contains many other statements apart from these few statements which needs to be fired in one transaction) is read by .ReadToEnd() and then executed in one go as of now. I want to handle this in sql script file itself and don't want to change the code.
Please take a look at SQLite on conflict clause
Start with BEGIN TRANSACTION
You have to add ON CONFLICT ROLLBACK on your actions
And COMMIT TRANSACTION at the end ;-)