I've got a (2) node galera setup and I'm having this weird issue where creating databases/tables works and replicates across the nodes, mean that if you create a database or a table under a database on one node 1, it will appear on node 2. But when data is inserted into database.table on node 1, it does not appear on node 2.
What could be the issue?
I figured out the problem. I was trying to load MYISAM dumps without converting them to INNODB.
Problem Solved and here is the quick conversion for anybody else impacted:
sed -i.bak 's#MyISAM#innodb#g' MIGRATION_DATA.sql
Related
Recently I made a mistake, that, all the tables in my oracle database were dropped (running drop command from SQL Developer tool) in a sequence of all transactions unbeknownst to me that I was dropping tables at the wrong database. Each time I dropped a set of tables, I committed the operations. By the time, I realized that I have dropped the tables at the wrong DB, I was too late to rollback, since it rolled back the last operation only. I searched online as well, and found SavePoints to be the resolution, but I had no savepoints configured. I had the backup of all the tables taken 2 days before, so, I ran scripts from there and made lost changes for the past 2 days.
Is there any other way that I take to get my DB state back. I have made it a practice to make savepoints while performing such operations though. But, I am still confused. One who doesn't know about savepoints will lose all the data.
DDL statements like drop do an implicit commit so you cannot rollback to undrop a table. Nor would a savepoint help you since that only lets you go back to an earlier point in the current transaction.
If you have merely dropped a table, assuming you haven't disabled the recycle bin, you can simply do a flashback drop
flashback table <<table name>> to before drop
If you have a more complicated script that is doing more than just dropping tables, you may want to set a restore point prior to running the script. You could then flashback the entire database to that restore point if something goes wrong.
create restore point <<your restore point name>>
<<run your script>>
flashback database to restore point <<your restore point name>>
I've a clustered partitioned table exported from GA 360. Attached is the image. I would like to create exact replica of the same. Using Web UI it's not possible. I created backup table using bq command line tool, still no luck.
Also, whenever we check preview it has a day filter. It looks like this:
Whenever data is appended to the backup table, I don't find this filter there even though this option is set to true while creating a table.
If you can give more context about handling this kind of table it would be beneficial.
Those are indeed sharded tables. As explained by #N. L they follow a time-based naming approach: [PREFIX]_YYYYMMDD. They then get grouped together. The explained procedure to backup them seems correct. Anyhow, I would recommend to use partitioned tables as it will be easier to backup them and they perform better in general.
This is not a cluster / partitioned table. This one is a sharded non-partitioned table having one common prefix. Once you start creating multiple tables with same prefix we can see them under the same prefix.
Ex:
ga_session_20190101
ga_session_20190102
both these tables will be grouped together.
To take backup of these tables you need to create a script to copy source to destination table with same name and execute that script using bq command line tool under the same project.
We have a huge table in Teradata holding the transnational data sizing approximately 13 TB.
As a typical old Data Warehouse enhancement case, we have now found that the PI selection made in the past for this table is not the best. And due to this, we are facing lots of performance issues with the SQL's pulling data from this table.
So the very first idea that we wanted to implement is to load the data in temp table, alter the PI of the existing table or create a new table with new PI and load the data from temp table to new \ altered table.
But the challenge here is the table is live and this solution will not be the best due the size of the data movement. Also, other way we thought about delta load (to new table) - delta delete (from main table). But this also can not be the best workaround as the table is live and it will involve too much of efforts in data movement as well as in matching the row counts in source and target table.
I need your ideas on this scenario, how can I change the PI of this table by making small efforts without any system downtime.
I am new to datastage, I created a simple job to get data from .ds file and load it in teradata using teradata connector, in the properties of teradata connector I set the
access_method=Bulk, max_session=2, min_session=1,load_type=load,
max_buffer_size=10000 and max_partition_sessions=1
but the job is continously in running state without displaying amount of rows transfered. Whereas when I choose the access_method=immediate then it starts to proceed, can any one suggest me the right way to do load in parallel.
Are you aware of the nodes you are using? Add APT_CONFIG_FLE environment variable and try to see from director how many nodes are used. N number of nodes means 2n number of processes. This might help you.
If this doesn't help try looking into the source database connector stage to see if increasing values for any options helps.
I just started to use KNIME and it suppose managed a huge mount of data, but isn't, it's slow and often not response. I'll manage more data than that I'm using now, What am I doing wrong?.
I set in my configuration file "knime.ini":
-XX:MaxPermSize=1024m
-Xmx2048m
I also read data from a database node (millions of rows) but I can't limit it by SQL (I don't really mind, I need this data).
SELECT * FROM foo LIMIT 1000
error:
WARN Database Reader com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'LIMIT 0' at line 1
I had the same issue... and was able to solve it really simply, KNIME has a KNIME.ini file, this one is like the paramethers KNIME uses to execute...
The real issue is that JBDC driver is set for 10 Fetch Size. By default, when Oracle JDBC runs a query, it retrieves a result set of 10 rows at a time from the database cursor. This is the default Oracle row fetch size value... so whenever you are reading database you will have a big pain waiting to retrieve all the lines.
The fix is simply, go to the folder where KNIME is installed, look for the file KNIME.ini, open it and then add the following sentences to the bottom, it will override the defauld JBDC fetching, and then you will get the data in literally seconds.
-Dknime.database.fetchsize=50000
-Dknime.url.timeout=9000
Hope this helps :slight_smile:
see http://tech.knime.org/forum/knime-users/knime-performance-reading-from-a-database for the rest of this discussion and solutions...
I'm not sure if your question is about the performance problem or the SQL problem.
For the former, I had the same issue and only found a solution when I started searching for Eclipse performance fixes rather than KNIME performance fixes. It's true that increasing the Java heap size is a good thing to do, but my performance problem (and perhaps yours) was caused by something bad going on in the saved workspace metadata. Solution: Delete contents of the knime/workspace/.metadata directory.
As for the latter, not sure why you're getting that error; maybe try adding a semicolon at the end of the SQL statement.