I'm reading on hash table and data structure, and one question come to mind. Where is hash table implemented? Is it on server code or database?
The resource I've read seems to implement them on the server code, but isnt storing data the job of database? PS: I've havent get to a point of knowing non-sql database yet, maybe that's where my knowledge lack.
Many applications need to store some data internally, even if they're also using or updating data in a database at times. Often they'll even retrieve related data from a remote (across the network) database and have it available in RAM on the local machine for the application to access quickly.
Other times, an application may use a data structure such as a hash table to support some application behaviours that are not part of the business data model, and therefore don't belong in the database. For example, a GUI application might keep help strings to display when the mouse hovers over a widget/button/whatever - they might be stored in a hash table keyed on some GUI object identifier, screen region or whatever the GUI library finds useful to help it display the tooltips at the right time. Another application might keep a table of usernames and activity statistics that it generated by scraping some website - it might display them to the user on demand, or aggregate them or something, without ever saving them down to a database (historic data may be of no value, and it can scrape the website again).
In summary - non-trivial programs tend to use hash tables to provide quick access to the data they consult or manipulate, whether the programs are themselves databases, applications that do also use databases, or applications that run without any database support.
Related
I have a Xamarin.Forms app that uses a local SqLite database as its source for data. The data is proprietary, so I want to protect it so that if someone gets access to the database file, they would have to decrypt it to access the data.
I also want to limit the number of queries users can make against the database so that at a certain point they have to purchase the ability to use more of the data (in-app purchase).
I want to avoid making network calls as much as possible to minimize impact to the user's data plan and allow the app to work well in conditions where there is poor or no connectivity. So, I want the data stored in a local database (perhaps in SqLite).
I'm curious how different people would approach this problem to protect the data and at the same time minimize network usage.
Here is kind of what I was thinking (if it's possible):
1) Let the user download/install the app.
2) On first load, the app will upload a key based on the device id and the user's current purchase information. Then it will download a SqLite database file that has been encrypted using the uploaded key.
3) When the user reaches their limit of queries, the database file is deleted. If they purchase more data, then a new key is uploaded and a new encrypted database is downloaded to be used.
Thoughts? Is there a better way?
I would suggest SQLCipher! It is a Component within Xamarin (http://components.xamarin.com/view/sqlcipher-for-xamarin-ios) but can also be built from source as it is Open Source (https://www.zetetic.net/sqlcipher/open-source/)
That will totally secure your database :)
UPDATE 8/2/2018 - SQL Cipher is now free and easy to implement thanks to the greatness of Frank Krueger. sqlite-net (https://github.com/praeclarum/sqlite-net) is the defacto sqlite library for Xamarin now (if you're still using the Sqlite.Net fork I recommend going back to sqlite-net as soon as possible as Sqlite.Net has been abandoned) and it now includes SQL Cipher support completely free of charge.
As clb mentioned, SQLCipher is open source. So if you don't want to pay for the component you can download and build the source yourself, then wrap it for use in Xamarin. This is, admittedly, a technically challenging task.
If that's not an option, I would recommend two other options:
Reevaluate your need to store data locally. It's extremely unlikely that you need to transfer enough data to even cause a blip on a user's data plan. And between cellular and wifi, it's not that common anymore for users to be without a connection. It certainly does happen, and there are certain apps where this is very important, but you may have to make concessions if the data is that sensitive.
If you absolutely have to store the data locally, and you can't use SQLCipher, your last real option is to use a cryptography library and encrypt the data itself, rather than the database file. This is less than ideal, typically, for a variety of reasons, but it may be your last resort. PCL Crypt is a PCL capable crypto library that you can look into.
https://github.com/aarnott/pclcrypto
I'm currently developing an app where the users are first asked to create an account trough a website (ASP.NET) to use the app. For a special reason I need to automatically generate a database for each customer creating an account, on the hosted SQL Server. The databases for all the customers are the same.
I was thinking about doing like that: as I have the script for creating the database, I was thinking to insert it in stored procedure or a trigger that will be launched as soon as the user has fully created his account.
I don't really see other solutions, maybe somebody could give me some guidelines? Thanks in advance.
I think such a design has been shown to not scale. I'd recommend redesigning the schema to allow multiple customers in a single database.
Amazon does not such thing. Neither should you.
I agree duffymo on you would have scalability issues.
However there are situations where in you might prefer separate database as your multi-tenant data approach.
In my last project I had to adopt separate DB approach as business wanted complete isolation for each customer. It was a school administrative system and number of customer was not expected to grow in more than three digits in 5-10 years time.
So the solution I designed was, I used Entity Framework code first approach. Every school will have a unique school identifier which will be used to name the database uniquely for each school. The connection string was generated at runtime obviously. A connection factory was used to create the appropriate DataContext based on passed school identifier. The database is created on first usage if not exist. At the same time SQL script was executed to create db users during db creation if not exist.
If this approach sounds appealing I can share code snippet if that helps.
I'm new to ASP.Net, MVC and the Entity Framework.
I'd like to understand the best practice for small databases. For example, say at Contoso University we know there are only going to be a few hundred or a few thousand students and courses. So all the data would comfortably fit in memory. So then is it better to use an in-memory collection and avoid potentially high-latency database operations?
I am thinking of small-scale production web sites deployed to Windows Azure.
To be more specific, the particular scenario I am thinking of has a few thousand records that are read-only, although users can create their own items too. Think of a collection of movies, albums or song lyrics that has been assembled offline from a list of a few thousand popular titles. The user can browse the collection (read-only), and most of the time they find what they are looking for there. However the user can also add their own records.
Since the popular titles fit in memory, and these are read-only, is it maybe better not to use a database for the popular titles? How would you organize data and code for this scenario?
Thanks for any thoughts and pointers.
I think a database is good place to store your information.
However, you are concerned about database latency.
You can mitigate that with caching - the data is stored in memory.
In short, it isn't an either or scenario...
You should definitely store your data in some persistent storage medium (SQL, Azure Tables, XML file, etc). The issues with storing items in memory are:
You have to find a way to store them once for the application and not once per user. Else, you will have potentially several copies of a 2-5 MB dataset floating around your memory space.
Users can add records, are these for everyone to see or just them. How would you handle user specific data.
If your app pool recycles, server gets moved by the Azure engineers, etc, you have to repopulate that data.
As said above, caching can really help to alleviate any SQL Azure latency (which btw, is not that high, we use SQL Azure and web roles and have not had any issues).
Complex queries. Sure, you can use LINQ to process in memory lists, but SQL is literally built to perform relational queries in a fast, efficient, data-safe manner.
Thread safe operations on an in-memory collection could be troublesome.
Edit/Addendum
The key, we have found, to working with SQL Azure is to not issue tons of little tiny queries, but rather, get the data you need in as few queries as possible. This is something all web applications should do, but it becomes much more apparent when using SQL Azure rather than a locally hosted database. Lastly, as far as performance/caching/etc, don't prematurely optimize! Get your application working, then identify bottlenecks. More often than not, it will be a code solution to fix the bottleneck and not necessarily a hardware/infrastructure issue.
I am trying to create an app that receives an Sqlite database from a server for offline use but cloud synchronization. The server has a postgres database with information from many clients.
1) Is it better to delete the sql database and create a new one from a query, or try to synchronize and update the existing separate sqlite files (or another better solution). The refreshes will be a few times a day per client.
2) if it is the latter, could you give me any leads to resources on how I could do this?
I am pretty new to database applications so please excuse my ignorance and let me know if there is any way I could clarify.
There is no one size fits all approach here. You need to carefully consider exactly what needs to be done, what you are replicating, how much data is involved, and what your write models are, all before you build a solution. Along the way you have to decide how to handle write conflicts and more.
In general the one thing I would say is that such synchronization works best with append-only write models (i.e. inserts, no deletes, no updates), and one way to do it is to log changes that need to be made and replicate those changes.
However, master-master replication is difficult on the best of days and with the best of tools available. Jumping between databases with very different capabilities will introduce a number of additional problems. You are in for a big job.
Here's an open source product that claims to solve this for many database types including Postgres. I have no affiliation or commercial interest in this company.
https://github.com/sqlite-sync/SQLite-sync.com
http://sqlite-sync.com/
If you're able and willing to step outside relational databases to use an object store you might want to have a look at CouchDb and perhaps PouchDb that use a MVCC based replication protocol designed to support multi-master replication including conflict resolution. Under the covers, PouchDb uses adaptors for Sqlite, IndexDb, Local storage or a remote CouchBb instance to persist client side data. It auto selects the best client side storage option for the given desktop or mobile browser. The Sqlite engine can be either WebSQL or a Cordova Sqlite plugin.
http://couchdb.apache.org/
https://pouchdb.com/
We are going to be selling a service that will be hosted by us, and each client we host will have their own database, but there will be one centralized website. I currently have a blank database with the few things that a new client will need. What is the best way to copy this database so I can setup another client? I want to be able to do this from an .aspx page. Thanks in advance!
Update:
By .aspx page, I just meant that I need to be able to kick off the process from an .aspx page.
Update2:
We're running SQL Server 2008.
Update 3: Referencing Cade Roux's answer... Thanks for a great answer, but...
What is the reason for merging all of the databases into one, and then distinguishing clients based on an identifier in each table? Wouldn't this greatly complicate the architecture of the entire product? I would need to add these Client ID columns to practically every table, and the DAL would need to know which client data its looking for. With the current setup I have, I just switch out the connection string in the DAL, depending on which user is accessing the site. That way, after the connection string is set, I never need to worry about finding client specific data! How do these approaches compare (and should I add this as a separate question?
You have a few different options:
You can detach your empty database, then when a user signs up, copy that database and mount it under a unique name for them and map it to their account in your master database, say.
You can create a database from scratch using scripts and populate any base data either from an online template database or scripting the base data and map it to their account in your master database.
You should seriously consider going to a multi-tenant architecture where all users are in the same database (with most tables having CustomerID columns to segregate the data) if you are going to have more than a few dozen customers.
Regarding your notes about option 3 - it depends on your application. Multi-tenant can be difficult to retrofit. On the other hand, managing and upgrading hundreds of individual customer databases can be difficult in the long haul.
There are previous Stack Overflow questions regarding this:
What are the advantages of using a single database for EACH client?
One database or many?
I think I'll see about re-tagging them with multi-tenant-db or something. Anyhow, I think that this comes up as a consideration secondary to your answer about a particular tactic does show the importance of including details about your overall goals in strategy in every question on StackOverflow.
Depending on what database you're using, there are several approaches. The simplest is to ask your database software to generate SQL code for creating the database and include that with your software. Another would be to just script out in C#/VB the steps needed to recreate your empty database.
Why the need for .aspx page?
You don't say what db version you're using but in SQL2005-2008, you have the ability to "script database as" and then "create to" and have it port the sql to a query window. You could then work with that to create a stored procedure that can be called from your .aspx page.
SQL Server has a system database called 'model'. Any database objects (tables, views, stored procedures) that exist in the model are added to any new database created.
You could create your 'client database' schema as model, and any new database would have all the same tables...
But, if you need to change your database schema later, your best option is to write change scripts which are part of your code-behind file. Since changes to the 'model' database are not propagated to existing databases, the application needs to detect and upgrade the database schema as necessary.
Disadvantage to this approach: If you want a database which isn't a 'client database' then you would need to create the database, and then delete the 'client database' tables.