I have a DynamoDB table in my microservice, which I choose to add an LSI to the table after few days of creating it.
Once I redeploy the microservice, all the records in the table got wiped off because of the addition of the LSI into the table.
Is there any smart way to keep the records backed up post the re-deployment? AWS datalake can be used to export and import data to and from s3. But this does not make sure the tables have the data in there during the time of deployment.
Related
I have a lot of documents in offline books. It's in table format and I don't want to manually input these tables into a dynamoDB table. Can I use AWS rekognition to help me here OR I should look at some other service ?
Rekognition won't generally do a good job at this. AWS has a new OCR service called Textract. It's able to figure out tables and forms (content and structure).
You can join the preview at https://aws.amazon.com/textract/
Our application back-end is made of Symfony 2.5.x.
We are using MySQL 5.x and Elastic search 1.7.x for storing our data.
In our application, we store data of different companies in their individual database i.e. multi-tenant approach. Similarly we store companies data in their respective aliases in elastic search.
We had to enhance default fos:elastica:populate command with our custom command so that it checks every company database and sync it with corresponding elastic search alias data.
All this is working fine. No problem so far. Till now we have almost 2000K records in elastic search.
Now, due to one of the requirement, we need to update mapping and store another property in elastic search i.e. store value of another column in elastic search.
Last time we add a new column for all existing aliases and perform bulk updating, 340K records were update on an average in per hour. So this time it might take 6 hours to complete the process.
Is there any way to speed up the process to say within 2 hours.
Our MySQL database is in AWS RDS, application code in Heroku and elastic search is in Found Elastic search provider.
Any help/idea will be much appreciated.
Thanks in advance !
I'm fetching myriad records from a web service and am want to store them in local DB. perhaps there maybe some duplicate records so I'm checking record by record to find duplicate ones and not store them.
Is there any fast solution such that I give the bulk data to DBMS and it stores those records which are not exists in DB?
It's worth noting that I'm using repository pattern in my ASP.NET web application.
You can use the MERGE statement.
In Oracle EBS, when we are doing data conversions and interfaces, loading data into Oracle from another system, how do we make sure the record is being locked? How do we make sure no other person is updating our records?
Oracle EBS seeded API's will take care about locking. We don't insert data into EBS base tables directly,
validate the data and insert into interface tables later we run oracle standard programs to import interface tables data into base tables.
These oracle standard programs use oracle seeded API to insert data into multiple base tables.
How do we make sure no other person is updating our records?
Developers use their own custom staging tables to import data into EBS.
when data upload staging tables to interface table maintain each interface data source is different, usually other persons don't update other interfaces records, We can't track if anyone updated from database backend tools like sql developer or TOAD. we can track transaction from last updated by column if they update from applications
If you have any specific issue related locking let us know
I have 2 questions.
I am developing a ASP.NET web application that uses the standard ASP.NET membership. We intend to have the membership tables in 1 database. We have 2 other databases that stores data for 2 different applications.
Shared - Membership info
DB1 - Application1
DB2 - Application2
Both applications uses the membership info in the "Shared" database.
The Shared database has a table called userdetals that will store additional users' info such as name, phone and job title for example.
However, DB1 also has a table called employees that store the same fields as name, phone and job title. Each employee may be an user.
Also for each table in DB1 and DB2, we keep audit trial, i.e. which user updated the tables in the database. Hence, we need to store UserID in the tables of DB1 and DB2.
We thought of having a Users table added in DB1 and DB2. So everytime a new user is created in Shared, the same user will be created in Users table in DB1 and DB2.
Our questions are:
What is the best way to maintain database integrity given the above setup? E.g. Each employee is assigned as an user. If any fields in DB1 such as username, name and phone is updated, then the same fields in Shared DB should be updated and vice versa.
Is it advisable to have membership database in a different database in our case? What is the best solution since almost all the tables in DB1 and DB2 references userID in the Shared database.
1.
The technology you are looking for is Merge Replication (http://bit.ly/KUtkPl). Essentially, you would create a common Users table on both databases, create a Merge Replication publisher on one application database, and then create a Merge Replication subscriber on the other application database. You could also set this up to synchronize the schema as well (which also means you only need to create the table once on the publishing database: it will push the table, schema with data, to the subscriber).
But if you are looking for more of a manual approach, I would not denormalize the user data to the employee(s) table, instead create a supplemental table and a view on each Application server. Kind of like inheritance in OOP: Any common data between the Employee table and Users table, leave on the shared user table. Any unique columns for the Employee, add to the supplemental table only and store on each database. The view would merge both the supplemental table and shared table. (http://bit.ly/9KPxt0)
Even if you do use Replication Services, I would still use this view design with the synchronized table.
You COULD update through the view, but I would not recommend that. It has been done before successfully in production, but there are too many constraints that could blow up (http://bit.ly/LJCJev). Instead update the table directly that holds the data.
Absolutely avoid "triggers that synchronize". Too risky (can cause an infant loop on your SQL server) and too much maintenance overhead.
2.
I would do the Merge Replication, it is just less for you to worry about and maintain after it is configured correctly. But your approach is OK if want something more manual or if you are not familiar with Replication services in SQL... just use the view noted above and you'll be set.
Easy way:
You can create link server to these databases.
And then create synonym to easy access to tables of each database.
Create trigger to update data when any data was updated on each table.