This is what my code currently looks like:
import java.sql.*
import java.sql.SQLException
class SqliteDB {
val conn = DriverManager.getConnection("jdbc:sqlite:cs2820-database.db")
fun createUser123(userID: String, password: String, adminStatus: String) {
val statement = conn.prepareStatement("INSERT INTO Users(id,pass,admin) VALUES(?,?,?)")
statement.setString(1, userID)
statement.setString(2,password)
statement.setString(3,adminStatus)
println("123")
statement.executeUpdate()
conn.commit()
println("User Created")
}
// create a user
fun createUser(userID: String, password: String, adminStatus: String) {
println("inside createUser")
val sql = "INSERT INTO Users(id,pass,admin) VALUES(?,?,?)"
Class.forName("org.sqlite.JDBC")
try {
conn.use { conn ->
conn.prepareStatement(sql).use { pstmt ->
pstmt.setString(1, userID)
pstmt.setString(2, password)
pstmt.setString(3, adminStatus)
pstmt.executeUpdate()
//conn.commit()
pstmt.close()
}
}
} catch (e: SQLException) {
println(e.message)
}
}
fun main(args: Array<String>) {
val db = SqliteDB()
db.createUser("Jim", "password", "false")
}
I have tested two different createUser methods and most everything I have tried will return the error [SQLITE_BUSY] The database file is locked (database is locked). I have several other methods within the same SqliteDB class that query data ("SELECT") that all work correctly, but every time I try to perform any kind of update of information I am given the same error. I am at a loss for what to do at this point having searched many different forums and posts about syntax and such.
The full stacktrace is as follows:
Exception in thread "main" org.sqlite.SQLiteException: [SQLITE_BUSY] The database file is locked (database is locked)
at org.sqlite.core.DB.newSQLException(DB.java:909)
at org.sqlite.core.DB.newSQLException(DB.java:921)
at org.sqlite.core.DB.execute(DB.java:822)
at org.sqlite.core.DB.executeUpdate(DB.java:863)
at org.sqlite.jdbc3.JDBC3PreparedStatement.executeUpdate(JDBC3PreparedStatement.java:99)
at SqliteDB.createUser(SqliteDB.kt:50)
at SqliteDBKt.main(SqliteDB.kt:122)
I don't believe the issue is with an open connection as the error seems to occur as soon as I try to execute the update to the User table. The insert seems to hit some sort of loop of some kind at the execution stage.
EDIT: Something else I noticed, is that when attempting to create a new user with a primary key (userID) that already exists, I am given a uniqueness error, suggesting the update is going thru and realizing the userID is already in the table; however, there is still the issue with the INSERT creating a new row in the table. I'm just not sure how to go about debugging that specific issue.
So this has been quite an issue for me for the past week or so and it looks like I have been able to figure out the issue.
I am aware that you have to make sure to close any connections to the database before opening a new one to avoid locks and any other similar issues. What I did not realize is that I have been using a program in IntelliJ called "DB Browser". This plugin creates a UI that much more easily allows you to access and change any aspect of the database you want without using actual SQL commands. What I didn't realize is this plugin takes up the only writeable connection the SQLite and JDBC allow to the database.
So, after deleting the connection to the database through the DB Browser plugin, all of my functions are working properly.
Related
Token for email verification is created with User registration and needs to be deleted from database within 24 hours with crons job help. In a delete function using query builder, token gets deleted only if date value is manually provided in form of string: {delDate: "2021-02-08T17:59:48.485Z" }. Here, all tokens with date before or equal 2021-02-08 get deleted, and is working fine. But thats a static input, manually put in hard code!
Since this must be a dynamic input, I set up variable 'delTime',which stores current date minus 24 hrs in it, but it seems .where condition will not take a variable as value, and will not delete, as: {delDate: deltime}. In fact, 'delDate' consoles exactly the info I need, but it will only work in form of string.
There is a ton of content online teaching how to delete stuff with a static value in typeorm querybuilder, but so hard to find with dynamic values....
How else can I make this work in a dynamic way ?
async delete(req: Request, res: Response){
try {
const tokenRepository = getRepository(Token);
var delTime = new Date();
delTime.setDate( delTime.getDate() - 1 );
console.log(delTime) //consoles 24 hors ago
await tokenRepository
.createQueryBuilder()
.delete()
.from(Token)
.where("tokenDate <= :deleteTime", { deleteTime: delTime})//value dynamically stored in variable does not delete
//.where("tokenDate <= :deleteTime", { deleteTime: "2021-02-08T18:01:10.489Z"})//static hard code value deletes
//.where("tokenDate <= :delTime", { delTime})//Variable put this way will not work either...
.execute();
} catch (error) {
res.status(404).send("Tokens not found");
return;
}
res.status(200).send('Tokens deleted successfuly');
}
Your parameterized query looks correct.
Switch on TypeOrm Query Logging to see the generated SQL, maybe you will be able to see what's going wrong. Paste the generated SQL into the SQL query console to debug.
Alternatively you can write your delete query without parameters and let Sqlite calculate current date -1 day:
.where("tokenDate <= DateTime('Now', '-1 Day'"); // for Sqlite
Note 1: Sqlite DateTime('Now') uses UTC time, so your tokenDate should use UTC also.
Note 2: This syntax is specific to Sqlite, you can do the same in other databases but the syntax is different, e.g. Microsoft SQL Server:
.where("tokenDate <= DATEADD(day, -1, SYSUTCDATETIME()); // for MS SQL Server
I'm now implement a program to migrate large amount of data to ADX base on Ingest from Storage feature of ADX and I'm need to check that status of each ingestion request each time the request finish but I'm facing an issue
Base on MS document in here
If I set the persistDetails = true for example with the command below it must save the ingestion status but currently this setting seem not work (with or without it)
.ingest async into table MigrateTable
(
h'correct blob url link'
)
with (
jsonMappingReference = 'table_mapping',
format = 'json',
persistDetails = true
)
Above command will return an OperationId and when I using it to check export status when the ingest task finish I always get this error message :
Error An admin command cannot be executed due to an invalid state: State='Operation 'DataIngestPull' does not persist its operation results' clientRequestId: KustoWebV2;
Can someone clarify for me what is the root cause relate to this? With me it seem like a bug relate to ADX
Ingesting data directly against the Data Engine, by running .ingest commands, is usually not recommended, compared to using Queued Ingestion (motivation included in the link). Using Kusto's ingestion client library allows you to track the ingestion status.
Some tools/services already do that for you, and you can consider using them directly. e.g. LightIngest, Azure Data Factory
If you don't follow option 1, you can still look for the state/status of your command using the operation ID you get when using the async keyword, by using .show operations
You can also use the client request ID to filter the result set of .show commands to view the state/status of your command.
If you're interested in looking specifically at failures, .show ingestion failures is also available for you.
The persistDetails option you specified in your .ingest command actually has no effect - as mentioned in the docs:
Not all control commands persist their results, and those that do usually do so by default on asynchronous executions only (using the async keyword). Please search the documentation for the specific command and check if it does (see, for example data export).
============ Update sample code follow suggestion from Yoni ========
Turn out, other member in my team mess up with access right with adx, after fixing it everything work fine
I just have one concern relate to PartiallySucceeded that need clarify from #yoni or someone have better knowledge relate to that
try
{
var ingestProps = new KustoQueuedIngestionProperties(model.DatabaseName, model.IngestTableName)
{
ReportLevel = IngestionReportLevel.FailuresAndSuccesses,
ReportMethod = IngestionReportMethod.Table,
FlushImmediately = true,
JSONMappingReference = model.IngestMappingName,
AdditionalProperties = new Dictionary<string, string>
{
{"jsonMappingReference",$"{model.IngestMappingName}" },
{ "format","json"}
}
};
var sourceId = Guid.NewGuid();
var clientResult = await IngestClient.IngestFromStorageAsync(model.FileBlobUrl, ingestProps, new StorageSourceOptions
{
DeleteSourceOnSuccess = true,
SourceId = sourceId
});
var ingestionStatus = clientResult.GetIngestionStatusBySourceId(sourceId);
while (ingestionStatus.Status == Status.Pending)
{
await Task.Delay(WaitingInterval);
ingestionStatus = clientResult.GetIngestionStatusBySourceId(sourceId);
}
if (ingestionStatus.Status == Status.Succeeded)
{
return true;
}
LogUtils.TraceError(_logger, $"Error when ingest blob file events, error: {ingestionStatus.ErrorCode.FastGetDescription()}");
return false;
}
catch (Exception e)
{
return false;
}
I wanted to know how I can handle and succesfully catch my Database Exception. I had a primary key ID and another key called name in my Sqlite database. Im using the flutter plugin sqflite. I wanted to handle the case when there is already a entry in my database with the given ID i want to display ID already exists, for that I want to know how I can handle the exceptions.
I tried to surround the code in a try catch block but Im not able to make FLutter realize about the DatabaseException class, it gives me error saying "DatabaseException" isn't a type and can't be used in on catch clause. I want to add a new Subject into my database after a button is pressed and if user enters the same ID value as already existing one I want to handle the error.
onSubmitted: (test) async {
Subject newSubject = Subject(name: name, id: test);
try {
await DBProvider.db.newSubject(newSubject);
} on DatabaseException {}
},
You should import sqflite in the dart file where you're using the exception, add
this:
import 'package:sqflite/sqflite.dart';
I'm using Azure Mobile App with Xamarin.Forms to create an offline capable mobile app.
My solution is based on https://adrianhall.github.io/develop-mobile-apps-with-csharp-and-azure/chapter3/client/
Here is the code that I use for offline sync :
public class AzureDataSource
{
private async Task InitializeAsync()
{
// Short circuit - local database is already initialized
if (client.SyncContext.IsInitialized)
{
return;
}
// Define the database schema
store.DefineTable<ArrayElement>();
store.DefineTable<InputAnswer>();
//Same thing with 16 others table
...
// Actually create the store and update the schema
await client.SyncContext.InitializeAsync(store, new MobileServiceSyncHandler());
}
public async Task SyncOfflineCacheAsync()
{
await InitializeAsync();
//Check if authenticated
if (client.CurrentUser != null)
{
// Push the Operations Queue to the mobile backend
await client.SyncContext.PushAsync();
// Pull each sync table
var arrayTable = await GetTableAsync<ArrayElement>();
await arrayTable.PullAsync();
var inputAnswerInstanceTable = await GetTableAsync<InputAnswer>();
await inputAnswerInstanceTable.PullAsync();
//Same thing with 16 others table
...
}
}
public async Task<IGenericTable<T>> GetTableAsync<T>() where T : TableData
{
await InitializeAsync();
return new AzureCloudTable<T>(client);
}
}
public class AzureCloudTable<T>
{
public AzureCloudTable(MobileServiceClient client)
{
this.client = client;
this.table = client.GetSyncTable<T>();
}
public async Task PullAsync()
{
//Query name used for incremental pull
string queryName = $"incsync_{typeof(T).Name}";
await table.PullAsync(queryName, table.CreateQuery());
}
}
The problem is that the syncing takes a lot of time even when there isn't anything to pull (8-9 seconds on Android devices and more than 25 seconds to pull the whole database).
I looked at Fiddler to find how much time takes the Mobile Apps BackEnd to respond and it is about 50 milliseconds per request so the problem doesn't seem to come from here.
Does anyone have the same trouble ? Is there something that I'm doing wrong or tips to improve my sync performance ?
Our particular issue was linked to our database migration. Every row in the database had the same updatedAt value. We ran an SQL script to modify these so that they were all unique.
This fix was actually for some other issue we had, where not all rows were being returned for some unknown reason, but we also saw a substantial speed improvement.
Also, another weird fix that improved loading times was the following.
After we had pulled all of the data the first time (which, understandably takes some time) - we did an UpdateAsync() on one of the rows that were returned, and we did not push it afterwards.
We've come to understand that the way offline sync works, is that it will pull anything that has a date newer than the most recent updated at. There was a small speed improvement associated with this.
Finally, the last thing we did to improve speed was to not fetch the data again, if it already had cached a copy in the view. This may not work for your use case though.
public List<Foo> fooList = new List<Foo>
public void DisplayAllFoo()
{
if(fooList.Count == 0)
fooList = await SyncClass.GetAllFoo();
foreach(var foo in fooList)
{
Console.WriteLine(foo.bar);
}
}
Edit 20th March 2019:
With these improvements in place, we are still seeing very slow sync operations, used in the same way as mentioned in the OP, also including the improvements listed in my answer here.
I encourage all to share their solutions or ideas on how this speed can be improved.
One of the reasons for the slow Pull() is when more than (10) rows get the same UpdatedAt value. This happens when you update the rows at once, for example running an SQL command.
One way to overcome this is to modify the default trigger on the tables. To ensure every row gets a unique UpdateAt, we did something like this:
ALTER TRIGGER [dbo].[TR_dbo_Items_InsertUpdateDelete] ON [dbo].[TableName]
AFTER INSERT, UPDATE, DELETE
AS
BEGIN
DECLARE #InsertedAndDeleted TABLE
(
Id NVARCHAR(128)
);
DECLARE #Count INT,
#Id NVARCHAR(128);
INSERT INTO #InsertedAndDeleted
SELECT Id
FROM inserted;
INSERT INTO #InsertedAndDeleted
SELECT Id
FROM deleted
WHERE Id NOT IN
(
SELECT Id
FROM #InsertedAndDeleted
);
--select * from #InsertedAndDeleted;
SELECT #Count = Count(*)
FROM #InsertedAndDeleted;
-- ************************ UpdatedAt ************************
-- while loop
WHILE #Count > 0
BEGIN
-- selecting
SELECT TOP (1) #Id = Id
FROM #InsertedAndDeleted;
-- updating
UPDATE [dbo].[TableName]
SET UpdatedAt = Convert(DATETIMEOFFSET, DateAdd(MILLISECOND, #Count, SysUtcDateTime()))
WHERE Id = #Id;
-- deleting
DELETE FROM #InsertedAndDeleted
WHERE id = #Id;
-- counter
SET #Count = #Count - 1;
END;
END;
I have this function in my application. If the insert of Phrase fails then can someone tell me if the Audit entry still gets added? If that's the case then is there a way that I can package these into a single transaction that could be rolled back.
Also if it fails can I catch this and then still have the procedure exit with an exception?
[Route("Post")]
[ValidateModel]
public async Task<IHttpActionResult> Post([FromBody]Phrase phrase)
{
phrase.StatusId = (int)EStatus.Saved;
UpdateHepburn(phrase);
db.Phrases.Add(phrase);
var audit = new Audit()
{
Entity = (int)EEntity.Phrase,
Action = (int)EAudit.Insert,
Note = phrase.English,
UserId = userId,
Date = DateTime.UtcNow,
Id = phrase.PhraseId
};
db.Audits.Add(audit);
await db.SaveChangesAsync();
return Ok(phrase);
}
I have this function in my application. If the insert of Phrase fails
then can someone tell me if the Audit entry still gets added?
You have written your code in a correct way by calling await db.SaveChangesAsync(); only one time after doing all your modifications on the DbContext.
The answer to your question is: No, the Audit will not be added if Phrase fails.
Because you are calling await db.SaveChangesAsync(); after doing all your things with your entities, Entity Framework wil generate all the required SQL Queries and put them in a single SQL transaction which makes the whole queries as an atomic operation to your database. If one of the generated query e.g. Auditgenerated query failed then the transaction will be rolled back. So every modification that have been done to your database will be removed and so Entity Framework will let your database in a coherent state.