I running my first trials on SQLite (with JDBC) as pure memory database. As I need very fast inserts I tried to set the config accordingly. I could find out that the code below fails only at the setting for JournalMode. Please refer to the method shown below. The variables con and isConnected are defined as class vars and not shown here.
Thanks a lot
Rolf
public Boolean connect() {
try {
Class.forName("org.sqlite.JDBC"); // sqlitejdbc_3.7.2
// Set all pragmas
SQLiteConfig config = new SQLiteConfig();
// This statement (JournalMode setting) causes a fail
// Without that statement the connection can be established
// ==> java.sql.BatchUpdateException: batch entry 0: query returns results
config.setJournalMode(JournalMode.MEMORY);
config.setTempStore(TempStore.MEMORY);
config.setSynchronous(SynchronousMode.OFF);
config.enforceForeignKeys(false);
config.enableCountChanges(false);
con = DriverManager.getConnection("jdbc:sqlite::memory:", config.toProperties());
isConnected = true ;
return true ;
}
catch (Exception e) {
LogMessages.instance().print(0, LogMessages.MSG_SEVERITY.ERROR, e.getMessage() + "\n");
e.printStackTrace() ;
return false ;
}
}
I have the same issue, but there is an other way to disable it. After opening the connection you can execute this query :
PRAGMA journal_mode=MEMORY;
Related
I'm working on a basic (non DB) connection pool which allows only 1 connection to be created per project.
The connection pool supports an async-task/threaded environment and therefor I have made use of a semaphore instead of a regular Lock.
I wrote a test, below, which is meant to stress test the connection pool.
The code works but under higher loads, the semaphore throws the following error
I can overcome this error by decreasing the load.
For example, increasing the _waitTimeMs to a higher number (i.e. 50ms or 100ms or 1000ms) or decreasing _numberOfTasks (i.e. to 5 or 3).
I should also mention that sometimes, it manages to run higher load tests without errors.
Is there a mistake or misconception in my code and/or use of semaphores?
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
internal class Program
{
static int _numberOfTasks = 50;
static int _waitTimeMs = 1;
static SemaphoreSlim _dictLock = new SemaphoreSlim(1, 1);
static ConcurrentDictionary<string, bool> _pool = new ConcurrentDictionary<string, bool>();
/// <summary>
/// Only 1 connection allowed per project.
/// We reuse connections if available in pool, otherwise we create 1 new connection.
/// </summary>
static async Task<string> GetConnection(string projId)
{
try
{
// Enter sema lock to prevent more than 1 connection
// from being added for the same project.
if (await _dictLock.WaitAsync(_waitTimeMs))
{
// Try retrieve connection from pool
if (_pool.TryGetValue(projId, out bool value))
{
if (value == false)
return "Exists but not connected yet.";
else
return "Success, exists and connected.";
}
// Else add connection to pool
else
{
_pool.TryAdd(projId, false);
// Simulate delay in establishing new connection
await Task.Delay(2);
_pool.TryUpdate(projId, true, false);
return "Created new connection successfully & added to pool.";
}
}
// Report failure to acquire lock in time.
else
return "Server busy. Please try again later.";
}
catch (Exception ex)
{
return "Error " + ex.Message;
}
finally
{
// Ensure our lock is released.
_dictLock.Release();
}
}
static async Task Main(string[] args)
{
if (true)
{
// Create a collection of the same tasks
List<Task> tasks = new List<Task>();
for (int i = 0; i < _numberOfTasks; i++)
{
// Each task will try to get an existing or create new connection to Project1
var t = new Task(async () => { Console.WriteLine(await GetConnection("Project1")); });
tasks.Add(t);
}
// Execute these tasks in parallel.
Parallel.ForEach<Task>(tasks, (t) => { t.Start(); });
Task.WaitAll(tasks.ToArray());
Console.WriteLine("Done");
Console.Read();
}
}
}
Is there a mistake or misconception in my code and/or use of semaphores?
There's a bug in your code, yes. If the WaitAsync returns false (indicating that the semaphore was not taken), then the semaphore is still released in the finally block.
If you must use a timeout with WaitAsync (which is highly unusual and questionable), then your code should only call Release if the semaphore was actually taken.
I am using sqlite as db in my Micronaut application getting the sqlite busy error in the below code:
#SneakyThrows
#TransactionalAdvice(value = EmpDao.DATASOURCE, propagation = TransactionDefinition.Propagation.REQUIRES_NEW)
public void storeEmp(EmpDto empDto) {
String id = empDto.getId();
try {
if (empDao.existsById(id)) {
log.debug("updating emp for id {}", empDto.getId());
empDao.update(EmpEntity.builder()
.id(id)
.data(getJson(empDto))
.entryCreatedAt(timeService.nowDateTime().toEpochSecond(ZoneOffset.UTC))
.build());
} else {
empDao.save(
EmpEntity.builder()
.id(id)
.data(getJson(empDto))
.entryCreatedAt(timeService.nowDateTime().toEpochSecond(ZoneOffset.UTC))
.build());
}
}catch(Exception e){
log.error("emp db save/update failed for id {} ",id, e);
}
}
#SneakyThrows
#TransactionalAdvice(value = EmpDao.DATASOURCE, propagation = TransactionDefinition.Propagation.REQUIRES_NEW)
public void storeEmployees(List<Emp> empDtos) {
try {
empDao.saveAll(empDto);
} catch (Exception ex) {
log.warn("saveAll failed", ex);
empDtos.forEach(this::storeEmp);
}
}
In stacktrace I can see first saveAll getting failed becuase of Primary keyconstraint Issue, that might be because of duplicate emp ids in the list
SQL error executing INSERT: [SQLITE_CONSTRAINT_PRIMARYKEY] A PRIMARY KEY constraint failed
and After that when It tried to save/update each emp object independently through storeEmp method in forEach, it's failing with the sqlite busy exception.
What I am not sure if saveAll is already failed, how there can be multiple connection to sqlite. Can anyone suggest what's wrong with the above code.
Thanks
When I execute the following code
public async Task<ObservableCollection<CommentModel>> GetTypeWiseComment(int refId, int commentType)
{
try
{
var conn = _dbOperations.GetSyncConnection(DbConnectionType.UserDbConnetion);
var sqlCommand = new SQLiteCommand(conn)
{
CommandText = "bit complex sqlite query"
};
List<CommentModel> commentList = null;
Task commentListTask =
Task.Factory.StartNew(() => commentList = sqlCommand.ExecuteQuery<CommentModel>().ToList());
await commentListTask;
var commentsList = new ObservableCollection<CommentModel>(commentList);
return commentsList;
}
catch (Exception)
{
throw;
}
finally
{
GC.Collect();
}
}
Sometimes I get the following exception
Message: database is locked
InnerException: N/A
StackTrace: at SQLite.SQLite3.Prepare2(IntPtr db, String query)
at SQLite.SQLiteCommand.Prepare()
at SQLite.SQLiteCommand.<ExecuteDeferredQuery>d__12<com.IronOne.BoardPACWinAppBO.Meeting.MeetingModel>.MoveNext()
at System.Collections.Generic.List<System.Diagnostics.Tracing.FieldMetadata>..ctor(Collections.Generic.IEnumerable<System.Diagnostics.Tracing.FieldMetadata> collection)
at BoardPACWinApp!<BaseAddress>+0xaa36ca
at com.IronOne.BoardPACWinAppDAO.Comments.CommentsDAO.<>c__DisplayClass4_0.<GetCommentTypeWiseComment>b__0()
at SharedLibrary!<BaseAddress>+0x38ec7b
at SharedLibrary!<BaseAddress>+0x4978cc
Can anyone point out what's wrong with my code?
There is another sync process going on the background and sometimes it has a bulk of records which may take more than 10 seconds to execute. If this above code happens to execute at the same time as the sync writes to the DB, it might block the reads, right?
If so how do I read from SQLite while another process writes to the DB?
Thank you.
as #Mark Benningfield mentioned enabling WAL mode almost solved my problem. However, there was another issue that creates a lot of SQLite connections on my app so I solved that by creating a Singleton module which handles database connections.
Please comment and ask if you require more information if you encounter a similar issue. Thanks.
I am currently making use of csharp-sqlite. In the SQLite database I have some uniqueness and foreign key constraints. But when I write a "duplicate" to a table an exception is not raised and the only way I found to check for an exception is to use something like the following:
var error = sqliteCommand.GetLastError();
I was hoping to be able to do the following:
var sqlstring = "some insert sql";
var command = SqliteCommand(sqlstring);
command.Parameters.Add(... some parameter...);
using (var sqlConnection = new SqliteConnection(connectionString))
{
try
{
command.ExecuteNonQuery();
}
catch (Exception ex)
{
do something with the exception
}
finally
{
sqlConnection.Close();
}
}
The "catch" block is never hit, even though the
var error = sqliteCommand.GetLastError();
does return with a description of the error that I was expecting. Does anyone have an idea how to solve this, or what standard practice is using the csharp-sqlite library?
im using the asp.net Enterprise Library to insert and update data to an sql database.
c# code:
Database db = DatabaseFactory.CreateDatabase();
IDbConnection connection = db.GetConnection();
connection.Open();
IDbTransaction transaction = connection.BeginTransaction();
DBCommandWrapper command;
try
{
//insert part
command = db.GetStoredProcCommandWrapper("stored_procedure1");
command.CommandTimeout = 900;
command.AddInParameter("#parameter1", DbType.Int32, 3);
db.ExecuteNonQuery(command, transaction);
//update part
command = db.GetStoredProcCommandWrapper("stored_procedure2");
command.CommandTimeout = 900;
command.AddInParameter("#param1", DbType.Int32, 5);
db.ExecuteNonQuery(command, transaction);
transaction.Commit();
}
catch (SqlException e)
{
transaction.Rollback();
throw (new ApplicationException("sql error", e));
}
catch (Exception e)
{
transaction.Rollback();
throw (new ApplicationException("error", e));
}
finally
{
connection.Close();
}
the code is on a method.
The method is executed many times and sometimes im getting error:
"Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached."
My questions are: Is it ok to have the insert and update part together as it is above?
Having the insert and update part together is causing this error?
Thanks.
First you do not roll back your transaction on error.
Secondly in order to guarantee that your finally block is called you need to add the catch block.
Try {
...
}
Catch {
}
finally {
close connection
}
Read the second paragraph of the link located below
http://msdn.microsoft.com/en-us/library/zwc8s4fz(v=vs.100).aspx