Database : Maria DB 10.6.5
Issue: Data transfer from partitioned table to Non-Partitioned table
Table A - partitioned on Date (range partitioned)
Table B - Not partitioned.
Both tables are exactly same without partition.
Case 1:
Transfer data from table A to B
It is successful.
Case 2:
Now again transfer data from Table A into table B(having case 1 data).
Data transfer successful but , table B existing data gets transferred back to table A.
How to prevent that? Table B existing data should not move back to table A
Any help will be of great value.
regards
Anand
Related
I have an excel spreadsheet with multiple entries that I want to insert into an SQLite DB from UIPath. How do I do this?
You could do it one of two ways. For both methods, you will need to use the Excel Read Range to put the excel into a table.
Scenario 1: You could read the table in a for each loop, line by line, converting each row to SQL and use a Execute non-query activity. This is too long, and if you like O notation, this is an O(n) solution.
Scenario 2: You could upload the entire table (as long as its compatible with the DB Table) to the database.
you will need Database > Insert activity
You will need to provide the DB Connection (which I answer in another post how to create)
Then enter the sqlite database table you want to insert into in Quotes
And then enter the table name that you have created or pulled from another resource in the last field
Output will be an integer (Affected Records)
For O Notation, this is an O(1) solution. At least from our coding perspective
We are dealing with log data of machines which are in presto . A view is created in Teradata Query Grid which would query the presto and have the result set in databricks. For some other internal use case we are trying to create a table in Teradata but facing difficulty in doing so.
When we try to create a table in Teradata for a particular date which has 2610117,459,037,913088 records only 14K odd records get inserted to the target table. Below is the query for the same. xyz.view is the view created in TD query grid which eventually fetches the data from presto.
CREATE TABLE abc.test_table AS
( SELECT * FROM xyz.view WHERE event_date = '2020-01-29' )
WITH DATA PRIMARY INDEX (MATERIAL_ID, SERIAL_ID);
But when we create the table with sample data (say sample 10000000), exact number of records we get in the table created like below:
CREATE TABLE abc.test_table AS
( SELECT * FROM xyz.view WHERE event_date = '2020-01-29' sample 10000000)
WITH DATA PRIMARY INDEX (MATERIAL_ID, SERIAL_ID);
But again creating with a sample of 1 billion records gets us only 208 million odd records in our target table.
Can anyone please help in here as to why is this happening and if it is possible to create the table with 2610117,459,037,913088 records.
We are using TD 16 .
I would like to cluster raw table with raw data of events from Firebase in BQ, but without reprocessing/creating another tables (keeping costs at minimum).
The main idea is to find a way to cluster tables when they create from intraday table.
I tried to create empty tables with pre-defined schema (same as previous events tables), but partitioned by _partition_time column (NULL partition) and clustered by event_name column.
After Firebase inserts all the data from intraday table, the column event_name stays in details tab of table as cluster field, but no reducing costs happens after querying.
What could be another solution or way how to make it working ?
Thanks in advance.
/edit:
Our table has detail tab as:
detail tab of table
After running this query:
SELECT * FROM 'ooooooo.ooooooo_ooooo.events_20181222'
WHERE event_name = 'screen_view'
the result is:
how query processed whole table
So no cost reducing.
But if I try to create the same table clustered by event_name manually with:
Create TABLE 'aaaa.aaaa.events_20181222'
partition by DATE(event_timestamp)
cluster by event_name
AS
Select * from ooooooo.ooooooo_ooooo.events_20181222
Then the same query from first IMG applied to created table processes only 5mb - so clustering really works.
I have a live table in dynamo with about 28 million records in it.
The table has a number of GSI that I'd like to change to be LSIs however LSIs can only be created when the table is created.
I need to create a new table and migrate the data with minimum downtime. I was thinking I'd do the following:
Create the new table with the correct indexes.
Update the code to write records to the old and new table. When this starts, take a note of the timestamp for the first record.
Write a simple process to sync existing data for anything with a create date prior to my first date.
I'd have to add a lock field to the new table to prevent race conditions when an existing record is updated.
When it's all synced we'd swap to using the new table.
I think that will work, but it's fairly complicated and feels prone to error. Has anyone found a better way to do this?
Here is an approach:
(Let's refer to the table with GSIs as oldTable and the new table with LSIs as newTable).
Create newTable with the required LSIs.
Create a DynamoDB tirgger for the oldTable such that for every new record coming to the oldTable insert the same record to the newTable. (This logic needs to be in the AWS Lambda).
Make your application point to the newTable.
Migrate all the records from oldTable to newTable.
I have 2 table Table A, Table B. Both the tables are of size 500GB, Some of the columns of tables are as below.
Table A
ID
Type
DateModified
Added a new column to Table as CID, which is available in Table B.
Table B
ID
CID
DateGenerated
Table A is partitioned on dateModified, Table B is not partitioned, My task is to get the CID from Table B and update it in Table A. Both the tables are having billions of records.
I have tried Merge/SQL but its too slow, which cannot be completed in 2 days.
Adding a new column to an existing table causing row fragmentation. Updating the new column to some value will probably cause massive row chaining, partitioned or not. And yes, that is slow, even when there are sufficient indexes etc.
Recommended approach:
You are on Enterprise Edition since you have partitioning, so you might be able to solve this using the schema versions functionality.
But if this is a one time action and you do not know how to use it well, I would use a "create table ... as" approach. Building the table from scratch and then switching it when ready. Take care to not miss any trickle loaded transactions. With partitioning it will be fast (writing 500 GB at say 50 MB/sec on a strong server is not unrealistic, taking 3 hours).