I am trying to copy all data from sqlite table to another one (reason being changing primary key and recommended way is to create new table, copy data, rename, etc).
Something along:
insert into new_table select * from old_table;
drop old_table;
alter table new_table rename to old_table;
Now problem is, that sometimes when I create new table, it's order of columns is not the same and data is completely messed up. Not sure why, probably cause some people have brand new table and some have old table which lived trough lot of upgrade scripts. Anyway my point is, I need to make sure every column is inserted into correct column. I can for sure just name all the columns manually, but I'd like this to work even if I add some columns in future and also for other tables. I can probably somehow query name of columns and then construct query in language that I am using above (kotlin), but I'd like to do this on db level if possible.
It's on android, but not very relevant to the question :)
Thanks!
Sadly I had add 1 extra query to get columns.
Other queries had all columns specifically mentioned instead of using star to ensure correct order. My kotlin way:
fun columns(db: SQLiteDatabase, tableName: String): List<String> {
val sql = "pragma table_info($tableName)"
return db.rawQuery(sql, arrayOf()).use { cursor ->
(1..cursor.count).map {
cursor.moveToNext()
cursor.getString(cursor.getColumnIndexOrThrow("name"))
}
}
}
fun copyFromTableToTable(db: SQLiteDatabase, originalTableName: String, backupTableName: String) {
val columns = columns(db, backupTableName).joinToString(separator = ",")
db.execSQL("""INSERT INTO $backupTableName
($columns)
SELECT $columns FROM $originalTableName
""".trimMargin())
//db.execSQL("DROP TABLE $originalTableName;")
//db.execSQL("ALTER TABLE $backupTableName RENAME TO $originalTableName;")
}
Related
Currently, I am trying to create some software on Progress OpenEdge that sorts by customer's names or account codes.
So essentially, a box will open up when the program runs, the user will select "Name" from the drop-down list, and the program will display all the names in the database from alphabetical order.
Or, they will pick "Account" from the drop-down list, and it will display all the account codes in numeric order. I have attached a picture of the program here:
And this is currently the code I am using to print the results:
However, I'm not sure what I need to add for the others. Would I need IF statements, such as:
OR IF [drop down list] = "Account" THEN or something like that?
Any help would be appreciated.
While you can perform a static order by with a convulted set of if statements, it is a lot cleaner with a dynamic query.
To expand on Tom and Stefan's answers, you can use a query. The ABL lets you create a lot of things with static constructs and use them as dynamic. I think that in this case, you want to so something like the below.
Note that you can do build the query string either using OPEN QUERY qry FOR EACH ... or QUERY qry:QUERY-PREPARE('FOR EACH ... ') ; both will work equally well.
What I think you'd want is
(a) having a static definition of the query (ie DEFINE QUERY) since the cost of adding the buffer(s) to the query is done at compile time, not run time, and
(b) accessing the buffer fields statically (ie slmast.name rather than b::name )
define query qry for slmast.
define variable wc as character no-undo.
if condition eq true then
wc = "WHERE kco = s-kco AND warecode = lv-warecode AND pcode = fi-pcode AND name = fi-name BY name".
else
wc = "WHERE TRUE".
/* alternate
if condition then
open query qry for each slmast no-lock WHERE kco = s-kco AND warecode = lv-warecode AND pcode = fi-pcode AND name = fi-name BY name.
else
open query qry for each slmast no-lock.
*/
query qry:query-prepare(wc).
open query qry.
query qry:get-first().
do while available slmast:
/* do stuff with the buffer */
{&OUT} slmast.name.
query qry:get-next().
end.
query qry:query-close().
Using static constructs as far as possible means that you have less cleanup code to write and the code becomes more readable (IMO).
There are multiple ways to loop through the query results: using DO WHILE NOT QUERY qry:QUERY-OFF-END works as well as AVAILABLE slmast or b:AVAILABLE (if using a purely dynamic query).
As Stefan says, a dynamic query is what you want. This might help get you started:
define variable wc as character no-undo.
define variable q as handle no-undo.
define variable b as handle no-undo.
/* run your UI to get selection criteria amd then
* create a WHERE clause as appropriate
*/
wc = "WHERE kco = s-kco AND warecode = lv-warecode AND pcode = fi-pcode AND name = fi-name BY name".
create buffer b for table "slmast".
create query q.
q:set-buffers( b ).
q:query-prepare( substitute( "FOR EACH slmast NO-LOCK &1", wc )).
q:query-open().
do while q:get-next():
display
b:buffer-field( "name" ):buffer-value
b:buffer-field( "acode" ):buffer-value
b:buffer-field( "pcode" ):buffer-value
b:buffer-field( "trunmtd" ):buffer-value
b:buffer-field( "turnytd" ):buffer-value
.
end.
I need to merge some randomly uppercased data that has been collected in an SQLite table key_val, such that key is always lowercase and no vals are lost. There is a unique compound index on key,val.
The initial data looks like this:
key|val
abc|1
abc|5
aBc|1
aBc|5
aBc|3
aBc|2
AbC|1
abC|3
The result after the merge would be
key|val
abc|1
abc|2
abc|3
abc|5
In my programmer brain, I would
for each `key` with upper case letters;
if a lower cased `key` is found with the same value
then delete `key`
else update `key` to lower case
Re implementing the loop has a sub query for each row found with upper case letters, to check if the val already exists as a lower case key
If it does, I can delete the cased key.
From there I can UPDATE key = lower(key) as the "duplicates" have been removed.
The first cut of the programming method of finding the dupes is:
SELECT * FROM key_val as parent
WHERE parent.key != lower(parent.key)
AND 0 < (
SELECT count(s.val) FROM key_val as s
WHERE s.key = lower(parent.key) AND s.val = parent.val
)
ORDER BY parent.key DESC;
I'm assuming there's a better way to do this in SQLite? The ON CONFLICT functionality seems to me like it should be able to handle the dupe deletion on UPDATE but I'm not seeing it.
First delete all the duplicates:
DELETE FROM key_val AS k1
WHERE EXISTS (
SELECT 1
FROM key_val AS k2
WHERE LOWER(k2.key) = LOWER(k1.key) AND k2.val = k1.val AND k2.rowid < k1.rowid
);
by keeping only 1 combination of key and val with the min rowid.
It is not important if you kept the key with all lower chars or not, because the 2nd step is to update the table:
UPDATE key_val
SET key = LOWER(key);
See the demo.
Honestly it might just be easier to create a new table and then insert into it. As it seems you really just want a distinct select here, use:
INSERT INTO kev_val_new ("key", val)
SELECT DISTINCT LOWER("key"), val
FROM key_val;
Once you have populated the new table, you may drop the old one, and then rename the new one to the previous name:
DROP TABLE key_val;
ALTER TABLE key_val_new RENAME TO key_val;
I agree with #Tim that it would be easire to re-create table using simple select distict lower().. statement, but that's not always easy if table has dependant objects (indexes, triggers, views). In this case this can be done as sequence of two steps:
insert lowered keys which are not still there:
insert into t
select distinct lower(tr.key) as key, tr.val
from t as tr
left join t as ts on ts.key = lower(tr.key) and ts.val = tr.val
where ts.key is null;
now when we have all lowered keys - remove other keys:
delete from t where key <> lower(key);
See fiddle: http://sqlfiddle.com/#!5/84db50/11
However this method assumes that key is always populated (otherwise it would be a strange key)
If vals can be null then "ts.val = tr.val" should be replaced with more complex stuff like ifnull(ts.val, -1) = ifnull(tr.val, -1) where -1 is some unused value (can be different). If we can't assume any unused value like -1 then it should be more complex check for null / not null cases.
I have to import a CSV file into a SQL Server table.
The CSV has 8 columns and 2 of them are date and time values.
In my SQL table I have a column which store the difference between 2 the columns of CSV file.
I am unable to check if the cell is empty or not.
My code follows:
protected void btnImport_Click(object sender, EventArgs e)
{
string csvPath = Server.MapPath("~/Files/") + Path.GetFileName(FileUpload1.PostedFile.FileName);
FileUpload1.SaveAs(csvPath);
DataTable dt = new DataTable();
dt.Columns.AddRange(new DataColumn[8] { new DataColumn("basid", typeof(string)),
new DataColumn("EmployeeName", typeof(string)),
new DataColumn("Designation",typeof(string)),
new DataColumn("OfficeLocation",typeof(string)),
new DataColumn("Division",typeof(string)),
new DataColumn("Intime",typeof(DateTime)),
new DataColumn("Outtime",typeof(DateTime)),
new DataColumn("timedifference",typeof(DateTime))
});
string csvData = File.ReadAllText(csvPath);
foreach (string row in csvData.Split('\n'))
{
if (!string.IsNullOrEmpty(row))
{
dt.Rows.Add();
int i = 0;
foreach (string cell in row.Split(','))
{
if (i == 7)
{
dt.Rows[dt.Rows.Count - 1][i] = ((DateTime)(dt.Rows[dt.Rows.Count - 1][i - 1]) - (DateTime)(dt.Rows[dt.Rows.Count - 1][i - 2]));
}
else {
dt.Rows[dt.Rows.Count - 1][i] = cell;
}
i++;
}
}
}
string consString = ConfigurationManager.ConnectionStrings["constr"].ConnectionString;
using (SqlConnection con = new SqlConnection(consString))
{
using (SqlBulkCopy sqlBulkCopy = new SqlBulkCopy(con))
{
//Set the database table name.
sqlBulkCopy.DestinationTableName = "dbo.basids";
con.Open();
sqlBulkCopy.WriteToServer(dt);
con.Close();
}
}
Understanding
Importing from a CSV file (or any other file that does not conform to the Relational Model), is not a problem. You need to understand that the file is not, and never will be, a Relational table. At best it is a file with good field types, at worst one with hopeless field types.
You may create a raw table in the database, for convenience, that is, in order to use SQL on it
At best, the fields in the file have reliable datatypes, thus the raw table may have SQL Datatypes.
At worst, the Datatypes in the raw table are all CHAR(x).
In any case, the columns in the raw table must match the import file exactly.
Usually, such raw tables do not have an Index, they are just Heaps.
Your app code will not query the raw tables, they are manipulated on by the Import code, only.
DataTable dt is such a raw table.
The Relational table (or tables) that are in the Relational database are quite a different animal. They are properly designed, Normalised, and compliant with the Relational Model. Relational tables have Primary Keys that are "made up from the data", and may have additional Unique Keys. The datatypes are "tight", and disallow illegal values.
Your app code will query the Relational tables, only.
Hence it is quite normal, that one CSV file (or import file or raw table) translates to more than one Relational table.
Therefore Import is always a two-step process:
First, just import the CSV file into the raw table, which has been suitably set up for that act.
You have not asked the question, but if I were loading this CSV file into a raw table, I wouldn't bother with all that middle-tier code, I would use the MS SQL bcp utility.
Second, INSERT-SELECT from the raw table into the Relational table(s).
In this step, the SELECT will transform the raw fields into proper Datatypes, and perform any checking that is required.
In order to ensure that the FK Constraints do not fail, the Reference tables must be populated before the data tables. For populating the Reference tables (not the raw file), eg. Operator, the code may be:
INSERT Operator
SELECT ...
FROM RawTable.Operator
WHERE RawTable.Operator NOT IN (
SELECT Operator
FROM Operator
)
ETL stands for Extract; Transform; Load. It is never Load only.
Problem
You have a raw table, only. And you think it is a Relational table. It is not.
You have no Relational tables.
Third, ...
I am unable to check if the cell is empty or not.
Yes, you can. After importing the CSV file into a raw table (possibly DataTable dt, or BAS_Raw below), which is step [1], in step [2], use ordinary SQL checks and transformations.
SELECT
...
[Hours] = CASE OutTime
WHEN NULL THEN CONVERT( TIME, "00:00" )
ELSE CONVERT( TIME, OutTime - InTime )
END
...
FROM BAS_Raw
Relational Database
The solution is, of course, to:
implement a Relational database, that has both:
Relational tables for all app queries, and
the raw table for the importation only
understand the two-step process that is required for importation of CSV files, including any checking and data transformation, as required.
Note • Content
I have no idea what basid is, therefore I have not modelled it. If you declare what it is, I will model it.
the time difference or hours worked does not need to be stored, because it is easily derived (OutTime minus InTime).
Note • Notation
All my data models are rendered in IDEF1X, the Standard for modelling Relational databases since 1993
My IDEF1X Introduction is essential reading for beginners or novices, wrt a Relational database.
I've got a function that I'd like use to delete a row in my database. This is the only way I've used the DELETE statement to remove a row before but I want the 1 to be replaced by a variable called recID so that the value of recID is the row ID number which is deleted. So if recID = 6, I want the function to delete the row with ID = 6. I hope that makes sense.
'DELETE FROM MyRecords WHERE ID=1';
The notation I've been using is the following, if it helps or makes any difference.
db.transaction(function(transaction) {
transaction.executeSql( //DELETE STATEMENT HERE );
});
executeSql supports arguments (check definition).
Use it like:
db.transaction(function(transaction) {
transaction.executeSql("DELETE FROM MyRecords WHERE ID=?", [recId]);
});
If you're certain that your variable, recID, will only ever contain numbers, you can just use:
transaction.executeSql("DELETE FROM MyRecords WHERE ID=" + recID);
If recID comes from outside your application (user input) however, it either needs to be sanitized, or use a prepared statement and use the database API to set the parameter after the statement has been prepared. Otherwise you open yourself up to SQL injection attacks.
I don't know the details of your SQLite wrapper, or what version of SQLite it wraps, but creating a prepared statement using the SQLite3 C API would go something like this:
// sqlite3* db = ...
sqlite3_stmt* stmt;
sqlite3_prepare_v2(db, "DELETE FROM MyRecords WHERE ID=?", -1, &stmt, 0);
sqlite3_bind_int(stmt, 1, recID);
sqlite3_step();
// ...
sqlite3_finalize(stmt);
This simple example excludes all the error checking you'd want to do in a real application, but since you're using a wrapper that has different syntax anyway, you'd have to figure out how it wraps these functions anyway.
I need to get column names and their tables in a SQLite database. What I need is a resultset with 2 columns: table_name | column_name.
In MySQL, I'm able to get this information with a SQL query on database INFORMATION_SCHEMA. However the SQLite offers table sqlite_master:
sqlite> create table students (id INTEGER, name TEXT);
sqlite> select * from sqlite_master;
table|students|students|2|CREATE TABLE students (id INTEGER, name TEXT)
which results a DDL construction query (CREATE TABLE) which is not helpful for me and I need to parse this to get relevant information.
I need to get list of tables and join them with columns or just get columns along with table name column. So PRAGMA table_info(TABLENAME) is not working for me since I don't have table name. I want to get all column metadata in the database.
Is there a better way to get that information as a result set by querying database?
You've basically named the solution in your question.
To get a list of tables (and views), query sqlite_master as in
SELECT name, sql FROM sqlite_master
WHERE type='table'
ORDER BY name;
(see the SQLite FAQ)
To get information about the columns in a specific table, use PRAGMA table_info(table-name); as explained in the SQLite PRAGMA documentation.
I don't know of any way to get tablename|columnname returned as the result of a single query. I don't believe SQLite supports this. Your best bet is probably to use the two methods together to return the information you're looking for - first get the list of tables using sqlite_master, then loop through them to get their columns using PRAGMA table_info().
Recent versions of SQLite allow you to select against PRAGMA results now, which makes this easy:
SELECT
m.name as table_name,
p.name as column_name
FROM
sqlite_master AS m
JOIN
pragma_table_info(m.name) AS p
ORDER BY
m.name,
p.cid
where p.cid holds the column order of the CREATE TABLE statement, zero-indexed.
David Garoutte answered this here, but this SQL should execute faster, and columns are ordered by the schema, not alphabetically.
Note that table_info also contains
type (the datatype, like integer or text),
notnull (1 if the column has a NOT NULL constraint)
dflt_value (NULL if no default value)
pk (1 if the column is the table's primary key, else 0)
RTFM: https://www.sqlite.org/pragma.html#pragma_table_info
There are ".tables" and ".schema [table_name]" commands which give kind of a separated version to the result you get from "select * from sqlite_master;"
There is also "pragma table_info([table_name]);" command to get a better result for parsing instead of a construction query:
sqlite> .tables
students
sqlite> .schema students
create table students(id INTEGER, name TEXT);
sqlite> pragma table_info(students);
0|id|INTEGER|0||0
1|name|TEXT|0||0
Hope, it helps to some extent...
Another useful trick is to first get all the table names from sqlite_master.
Then for each one, fire off a query "select * from t where 1 = 0". If you analyze the structure of the resulting query - depends on what language/api you're calling it from - you get a rich structure describing the columns.
In python
c = ...db.cursor()
c.execute("select * from t where 1=0");
c.fetchall();
print c.description;
Juraj
PS. I'm in the habit of using 'where 1=0' because the record limiting syntax seems to vary from db to db. Furthermore, a good database will optimize out this always-false clause.
The same effect, in SQLite, is achieved with 'limit 0'.
FYI, if you're using .Net you can use the DbConnection.GetSchema method to retrieve information that usually is in INFORMATION_SCHEMA. If you have an abstraction layer you can have the same code for all types of databases (NOTE that MySQL seems to swich the 1st 2 arguments of the restrictions array).
Try this sqlite table schema parser, I implemented the sqlite table parser for parsing the table definitions in PHP.
It returns the full definitions (unique, primary key, type, precision, not null, references, table constraints... etc)
https://github.com/maghead/sqlite-parser
The syntax follows sqlite create table statement syntax: http://www.sqlite.org/lang_createtable.html
This is an old question but because of the number of times it has been viewed we are adding to the question for the simple reason most of the answers tell you how to find the TABLE names in the SQLite Database
WHAT DO YOU DO WHEN THE TABLE NAME IS NOT IN THE DATABASE ?
This is happening to our app because we are creating TABLES programmatically
So the code below will deal with the issue when the TABLE is NOT in or created by the Database Enjoy
public void toPageTwo(View view){
if(etQuizTable.getText().toString().equals("")){
Toast.makeText(getApplicationContext(), "Enter Table Name\n\n"
+" OR"+"\n\nMake Table First", Toast.LENGTH_LONG
).show();
etQuizTable.requestFocus();
return;
}
NEW_TABLE = etQuizTable.getText().toString().trim();
db = dbHelper.getWritableDatabase();
ArrayList<String> arrTblNames = new ArrayList<>();
Cursor c = db.rawQuery("SELECT name FROM sqlite_master WHERE
type='table'", null);
if (c.moveToFirst()) {
while ( !c.isAfterLast() ) {
arrTblNames.add( c.getString( c.getColumnIndex("name")) );
c.moveToNext();
}
}
c.close();
db.close();
boolean matchFound = false;
for(int i=0;i<arrTblNames.size();i++) {
if(arrTblNames.get(i).equals(NEW_TABLE)) {
Intent intent = new Intent(ManageTables.this, TableCreate.class
);
startActivity( intent );
matchFound = true;
}
}
if (!matchFound) {
Toast.makeText(getApplicationContext(), "No Such Table\n\n"
+" OR"+"\n\nMake Table First", Toast.LENGTH_LONG
).show();
etQuizTable.requestFocus();
}
}