I will try to describe my problem of choosing good technology.
I have many machines which stores data locally in database. And there is one client machine with its own database. What I need is to pull data from all machines and put in client's database.
For now I have started implementing some RPC, but I don't know if its good idea. Because I need to manually take care of each table. Database is SQLite.
What is better. Making some RPC calls or find some light database with replication? Maybe NoSQL db like MonoDB?
I have a similar setup where I have a couple of servers that collect various statistics and store in a sqlite3 database. Combining them is really easy. I have a python script that connect to each server, downloads each database file into a temporary folder. I then open the first one, and use ATTACH for each file, and then insert * for each table to merge in all the other databases into a combined database:
conn = connect('/tmp/database1.sl3');
curs = conn.cursor();
mergeDatabases(curs, 8);
def mergeDatabases(curs, j):
for i in range(2, j):
print "merge in database%d" %i
print "ATTACH '/tmp/database%d.sl3' AS foo%d;" %(i,i)
curs.execute("ATTACH '/tmp/database%d.sl3' AS foo%d;" %(i,i))
curs.execute("insert into db select * from foo%d.db;" %i)
curs.execute("insert into vars select * from foo%d.vars;" %i)
curs.execute("detach foo%d;" %i)
Related
I'm working on a tool that allows Python developers to write pythonic code to interact with a sqlite3 database, similar to sqlalchemy but without the "translation" phase. If I can generate a sqlite3 prepared statement, how can I directly pass it to the evaluation system?
As a rough example, here's how I roughly view a user being able to interact with my tool:
myTable = Table("field1", "field2", "field3")
mytable.insert("foo", "bar", "baz")
select = mytable.select("field1")
---------------
print(select)
>>> ["foo"]
There is no (public) API in SQLite3 that allows you to execute pre-built SQLite bytecode. The bytecode for an SQL statement can be viewed with the EXPLAIN SQL command, but this is meant for debugging and learning purposes, not for what you're trying to do.
And for most purposes, you shouldn't need this. If you feel that the time spent compiling a prepared statement will be a burden, sqlite3_stmt objects can be stored for the lifetime of the sqlite3 database connection it was created with. Prepared statements that have been executed can be reset, allowing them to be executed again. So as long as the database connection exists, you can compile the statement once and use it as many times as you need to.
But that's about it. There is no mechanism to persist a prepared statement beyond the lifespan of the sqlite3 connection. You can't extract the bytecode by any public API, and you can't use some bytecode you've obtained to reconstitute a prepared statement.
If you want persistence beyond the connection, then you need to store the SQL statement text in whatever place you want to be persistent, and then simply recompile the prepared statement when you reconnect to the database. That one recompilation (or many depending on how many you store) shouldn't be a particular burden, depending on the life span of your application.
I have a specific architecture to set up with postgresql.
I have a system that is based on two databases N and N+1.
The database N is available for clients in read only mode, and the database N+1 is available for modification for clients.
The client can also send two commands to the system:
An "apply" command: all the modifications made on the N+1 db are kept and the new state of the system is a readonly db with N+1 data and a N+2 db with same data available for writes.
A "reset" command: the N+1 db is dropped and a new copy of the N database is made for writes access for the users.
My first idea was to keep two databases in an instance of postgresql and perform pg_dump and pg_restore command on apply or reset command, and rename the database for the apply(N+1 -> N). The db can possibly reach the size of 8 Go, so I am currently performing test of such dump&restore on a Centos 6.7 vm.
Then I looked to the pg_basebackup command, to set up a hot standby database that will be the readonly one. The problem is that such an architecture is based on the idea of data replication from a master to a slave, and that is something I don't want since the client can ask a reset command that will drop the N+1 db.
The thing is I don't know if a system based on daily dump/restore is viable or not, or if there is with postgresql a simple way to handle two databases with the same schema and "detect and apply" the differences between the two: with that ability, I will be able , on apply command, to copy from N+1 to N only the difference, and the contrary with a reset command.
Any idea ?
I want to transfer tables data from SQL server to Informix and vice versa.
The transferring should be run scheduled and sometimes when the user make a specific action.
I do this operation through delete and insert transactions and it takes along long time through the web between 15 minute to 30 minute.
How to do this operation in easy way taking the performance in consideration?
Say I have
Vacation table in SQL Server and want to transfer all the updated data to the Vacation table in Informix.
and
Permission table in Informix and want to transfer all the updated data to the Permission table in SQL Server.
DISCLAIMER: I am not an SQL Server DBA. However, I have been an Informix DBA for over ten years and can make some recommendations as to its performance.
Disclaimer aside, it sounds like you already have a functional application, but the performance is a show-stopper and that is where you are mainly looking for advice.
There are some technical pieces of information that would be helpful to know, but in their absence, I'm going to make the following assumptions about your environment and application. Please comment or edit your question if I am wrong on any of these.
Database server versions. From the tags, it appears you are using SQL server 2012. However, I cannot determine the Informix server and version. I will assume you are running at least IDS 11.50 or greater.
How the data is being exchanged currently. Are you connecting directly from your .NET application to Informix? I would assume that is the case with SQL Server and will make the same assumption for your Informix connection as well.
Table structures. I assume you have proper indexing on the tables. On the Informix side, dbschema -d *dbname* -t *tablename* will give the basic schema.
If you haven't tried exporting data to CSV and as long as you don't have any compliance concerns doing this, I would suggest loading the data from a comma-delimited file. (Informix normally deals with pipe-delimited files, so you'll either need to adjust the delimiter on the SQL Server side to a pipe | or on the Informix import side). On the Informix end, this would be a
LOAD FROM 'source_file_from_sql_server' DELIMITER '|' INSERT INTO vacation (field1, field2, ..)
For reusability, I would recommend putting this in a stored procedure. Just wrap that load statement inside a BEGIN WORK; and COMMIT WORK; to keep your transactional integrity. Michał Niklas suggested some ways to track changes. If there is any correlation between the transfer of data to the vacation table in Informix and the permission table back in SQL Server, I would propose another option, which is adding a trigger to the vacation table so that you write all new values to a staging table.
With the import logic in a stored procedure, you can fire the import on demand:
EXECUTE PROCEDURE vacation_import();
You also mentioned the need to schedule the import, which can be accomplished with Informix's "dbcron". Using this feature, you'll create a scheduled task that executes vacation_import() periodically as well. If you haven't used this feature before, using OAT will be helpful. You will also want to do some housekeeping with the CSV files. This can be addressed with the system() call, which you can make from stored procedures in Informix.
Some ideas:
Add was_transferred column to source tables setting its default value to 0 (you can use 0/1 instead of false/true).
From source table select data with was_transferred=0.
After transferring data update selected source row, set its was_transferred to 1.
Make table syncro_info with fields like date_start and date_stop. If you discover that there is record with date_stop IS NULL it will mean that you are tranferring data. This will protect you against synchronizing data twice.
According to one of my posts (below) it seems that there is no such thing as a database in Oracle. What we call database in MySQL and MS-SQL is called schema in Oracle.
If that is the case, then why do the oracle docs mention the create database statement ?
For the record, I am using Oracle 11g and oracle SQL Developer GUI tool.
Post-
How to create a small and simple database using Oracle 11 g and SQL Developer?
The create database statement from oracle docs is given below. If there is no database concept, then how did this command come into the picture ?
CREATE DATABASE
CREATE DATABASE [ database ]
{ USER SYS IDENTIFIED BY password
| USER SYSTEM IDENTIFIED BY password
| CONTROLFILE REUSE
| MAXDATAFILES integer
| MAXINSTANCES integer
| CHARACTER SET charset
| NATIONAL CHARACTER SET charset
| SET DEFAULT
{ BIGFILE | SMALLFILE } TABLESPACE
| database_logging_clauses
| tablespace_clauses
| set_time_zone_clause
}... ;
There is concept of a "database" in Oracle. What the term "database" means in Oracle terms is different than what the term means in MySQL or SQL Server.
Since you are using the express edition, Oracle automatically runs the CREATE DATABASE statement as part of the installation process. You can only have 1 express edition database on a single machine. If you are installing a different edition, you can choose whether to have the installer create a database as part of the installation process or whether to do that manually via the CREATE DATABASE statement later. If you are just learning Oracle, you're much better off letting Oracle create the database for you at installation time-- you can only create the database via command-line tools (not SQL Developer) and it is rare that someone just starting out would need to tweak the database settings in a way that the installer didn't prmopt you for.
In Oracle, a "database" is a set of data files that includes the data files for the SYS and SYSTEM schemas which contain all the Oracle data dictionary tables, the data files for the TEMP tablespace where sorts and other temporary operations occur, and the data files for whatever schemas you want to create. In SQL Server and other RDBMSs, these would be separate "databases". In SQL Server, you have a master database, a tempdb database, additional database for different products (i.e. msdb for the SQL Server Agent), and then additional user-defined databases. In Oracle, these would all be separate schemas in a larger container that Oracle refers to as a "database".
Occasionally, a DBA will want to run multiple Oracle databases on the same server-- most commonly when there are different packaged applications that have different requirements about database versions or parameters. If you want to run application A that requires an 11.2 database and application B that doesn't support 11.2 yet, you would need to have two different databases on the server. The DBA could create a separate database and a separate instance but that doubles the memory requirements, doubles the number of background processes required to run the database, and generally makes things less scalable. It's necessary if you really want to run different versions of the database simultaneously but it's not ideal.
The person who answered your original question is correct. The DDL (Data Definition Language) above prepares a space for schemas, which is analogous to MySQL's 'database'. The above statement defines characteristics of the schemas, such as timezone, MBs of space for tables, encoding characterset, root account, etc. You would then issue DDL statements such as those in your other post to create schemas, which define what each user can see.
I have a project that requires us to maintain several MySQL databases on multiple computers. They will have identical schemas.
Periodically, each of those databases must send their contents to a master server, which will aggregate all of the incoming data. The contents should be dumped to a file that can be carried via flash drive to an internet-enabled computer to send.
Keys will be namespace'd, so there shouldn't be any conflict there, but I'm not totally sure of an elegant way to design this. I'm thinking of timestamping every row and running the query "SELECT * FROM [table] WHERE timestamp > last_backup_time" on each table, then dumping this to a file and bulk-loading it at the master server.
The distributed computers will NOT have internet access. We're in a very rural part of a 3rd-world country.
Any suggestions?
Your
SELECT * FROM [table] WHERE timestamp > last_backup_time
will miss DELETEed rows.
What you probably want to do is use MySQL replication via USB stick. That is, enable the binlog on your source servers and make sure the binlog is not thrown away automatically. Copy the binlog files to USB stick, then PURGE MASTER LOGS TO ... to erase them on the source server.
On the aggregation server, turn the binlog into an executeable script using the mysqlbinlog command, then import that data as an SQL script.
The aggregation server must have a copy of each source servers database, but can have that under a different schema name as long as your SQL all does use unqualified table names (does never use schema.table syntax to refer to a table). The import of the mysqlbinlog generated script (with a proper USE command prefixed) will then mirror the source servers changes on the aggregation server.
Aggregation across all databases can then be done using fully qualified table names (i.e. using schema.table syntax in JOINs or INSERT ... SELECT statements).