I have some data contained in a CSV file, I need to efficiently access that information and want to importing it into my existing database.
I am wondering if I can make a pre-loaded database with the tables I need and then build the rest of the database on top of it (or make a second separate connection), or load the database from the CSV files on first startup.
What would be the preferred method and either way how would can I achieve it efficiently?
p.s 2 files are about 1000 lines long and 2 columns wide which seems to me to be considered fairly small... and the other ones really shouldn't be more then 10 lines long and 6-7 columns wide
Edit: realised I have a bunch of tables that need to be updated yearly, so any form that risks the user input data is unacceptable so using the existing DB is a not an option...
Related
I have created a MS Access database that is written using three (3) databases. One is a front end for the users (several users). The other two (2) are data storage tables only. The two data storage tables are identical (databases named the same, tables named the same...). One is stored locally on each users laptop and one is on the network (shared). The purpose of this arrangement is to allow the user a read only version of the data when off line traveling by using their local tables. They can make changes to the data when on line - these changes will be made to the on-line tables. I've written databases in MS Access for about 10 years, but I build mostly in queries. I'm not strong in VBA. Ideally I could get the VBA code to link to a button that changes the link from one data storage database to the other. I will then use this code to make an 'on-line' button and an 'off-line' so I can toggle back and forth. Thanks so much for your time and knowledge. I do appreciate it.
I have another case where I don't know how to find a solution with BizTalk.
I have this two flat files (in real there are 9 files to combine) and the output must be like shown in the picture:
How can I combine files which ID repeat several times in the main file.
In the below picture, the main file is "People". Is there way to do this without writing any code in BizTalk, or must I store this data in SQL DB after that i join them with a stored procedure?
Can you help me lay-out the steps I need to take, because I know how to combine files together but that is without the repeated ID's.
I have got a pretty fat settings table in SQL Server 2012, now with over 100 columns. As the name suggests, this table keeps track of all kinds of setting values within our website. It used to be having less than 50 columns but now its size is doubled.
The reason why I store setting values into database is because users will need to have ability to change these settings via UI.
Should I really be worried about this table getting bigger and bigger over time? Or I will have to find some other ways to store settings data, e.g save into files, perhaps?
First, you don't need to store settings in a database in order to update them at runtime by users. You can simply store them in a settings file that gets updated whenever the user makes changes. This is an xml config file and works well.
If, however, the application is network based, and you want the settings to follow the user from machine to machine, it makes more sense to put it in a database.
Second, yes... 100 columns is huge. Instead of storing each setting in a separate column, you might consider storing each setting in a separate row, and then have a common row format which is ID, SettingName, SettingValue, (maybe) DefaultValue. Then your table can grow as large as you like.
We are using JSON to store user settings. The table obtains only two columns - the user Id and the setting string. This string is quite long, but it doesn't matter. You can also use XML to store this data.
This is worse solution to modify data by finger, but faster to get from your DB and process by the client or by the ASP.NET server.
I am imagining that you are concerned about performance on huge tables?
One question is how many rows in this table? 100 columns with 10000 rows is not real problem. 100 columns over 10million rows is a slightly different ballgame. Not worse of better, just different.
The same considerations apply for small and large tables:
1. Are you indexing properly
2. Is your IO fine
3. Is your space fine
4. Are you querying efficiently
There is no right answer for this, it would depend of why you have big column counts and whether it's hitting your overall performance.
We run 1000s of tables with > 150 columns and no problems, even with millions of rows between them and I can't complain about performance.
And this is relatively de-normalized data, so lots of text.
I have a database that is used to store transactional records, these records are created and another process picks them up and then removes them. Occasionally this process breaks down and the number of records builds up. I want to setup a (semi) automated way to monitor things, and as my tool set is limited and I have an R shaped hammer, this looks like an R shaped nail problem.
My plan is to write a short R script that will query the database via ODBC, and then write a single record with the datetime, the number of records in the query, and the datetime of the oldest record. I'll then have a separate script that will process the data file and produce some reports.
What's the best way to create my datafile, At the moment my options are
Load a dataframe, add the record and then resave it
Append a row to a text file (i.e. a csv file)
Any alternatives, or a recommendation?
I would be tempted by the second option because from a semantic point of view you don't need the old entries for writing the new ones, so there is no reason to reload all the data each time. It would be more time and resources consuming to do that.
I want to store data which is dynamic in nature. The user can create forms for capturing data. The data can be stored in various formats as per the configuration. The major ones are a RDBMS and a XML file. XML file format is pretty easy to store dynamic data and load it back.
I am not able to devise a data structure for a RDBMS. I currently store data in a key-value format and do a PIVOT for fetching it. For fields which have multiple values I store them as CSV in the value column.
Is there a better way for storing such dynamic data which helps in performance and extensibility?
Without knowing more about your application it is hard to say.
You could save the data as XML in a BLOB in the database. That would mean all your data was (sorta) handled the same way (as XML).
The other approach would be to change your database structure to hold nested data (which appears to be your problem). So instead of a straight key-value table you might hace a table structure that could reference itself (e.g. parent - key - value) and have a header table to hold the top level keys.
The real question though is why you want to use a database to hold the data. It seems the real problem is trying to fit a round peg into a square hole (or vice versa).