I am developing a Spring Based Web Application which will handle large volume of requests per minute and this web app needs to respond very quickly.
For this purpose, We decided to implement a flat-file based queuing mechanism, which would just write the requests (set of database columns values) to flat files and another process would pick this data from flat files periodically and write it to the database. I pick up only those files that am done writing with.
As am using a flat file, For each request I receive, I need to open and close the flat file inside my controller method.
My Question is : Is there a better way to implement this solution ? JMS is out of scope as we don't have the infrastructure right now.
If this file based approach seems good, then is there a better way to reduce the file I/O ? With the current design, I open/write/close the flat file for each web request received, which I know is bad. :(
Env : SpringSource ToolSuite, Apache/Tomcat with back-end as Oracle.
File access has to be synchronized, otherwise you'll corrupt it. Synchronized access clashes with the large volume of requests you plan.
Take a look at things like Kestrel or just go with a database like SQLite (at least you can delegate the synchronization burden)
Related
I am planning to create sqlite table on my android app. The data comes from the the server via webservice.
I would like to know what is the best way to do this.
Should I transfer the data from the webservice in a sqlite db file and merge it or should i get all the data as a soap request and parse it in to table or should I use rest call.
The general size of the data is 2MB with 100 columns.
Please advise the best case where I can quickly get this data, with less load on the device.
My Workflow is:
Download a set of 20000 Addresses and save them to device sqlite database. This operation is only once, when you run the app for the first time or when you want to refresh the whole app data.
Update this record when ever there is a change in the server.
Now I can get this data either in JSON, XML or as pure SqLite File from the server . I want to know what is the fastest way to store this data in to Android Database.
I tried all the above methods and I found getting the database file from server and copying that data to the database is faster than getting the data in XML or JSON and parsing it. Please advise if I am right or wrong.
If you are planning to use sync adapters then you will need to implement a content provider (or atleast a stub) and an authenticator. Here is a good example that you can follow.
Also, you have not explained more about what is the use-case of such a web-service to decide what web-service architecture to suggest. But REST is a good style to write your services and using JSON over XML is advisable due to data format efficiency (or better yet give protocol-buffer a shot)
And yes, sync adapters are better to use as they already provide a great set of features that you will want to implement otherwise when written as a background service (e.g., periodic sync, auto sync, exponential backoff etc.)
To have less load on the device you can implement a sync-adapter backed by a content provider. You serialize/deserialize data when you upload/download data from server. When you need to persist data from the server you can use the bulkInsert() method in content-provider and persist all your data in a transaction
I have a couple of ActionMethods that returns content from the database that is not changing very often (eg.: a polygon list of available ZIP-Areas, returned as json; changes twice per year).
I know, there is the [OutputCache(...)] Attribute, but this has some disadvantages (a long time client-side caching is not good; if the server/iis/process gets restartet the server-side cache also stopps)
What i want is, that MVC stores the result in the file system, calculates the hash, and if the hash hasn't changed - it returns a HTTP Status Code 304 --> like it is done with images by default.
Does anybody know a solution for that?
I think it's a bad idea to try to cache data on the file system because:
It is not going to be much faster to read your data from file system than getting it from database, even if you have it already in the json format.
You are going to add a lot of logic to calculate and compare the hash. Also to read data from a file. It means new bugs, more complexity.
If I were you I would keep it as simple as possible. Store you data in the Application container. Yes, you will have to reload it every time the application starts but it should not be a problem at all as application is not supposed to be restarted often. Also consider using some distributed cache like App Fabric if you have a web farm in order not to come up with different data in the Application containers on different servers.
And one more important note. Caching means really fast access and you can't achieve it with file system or database storage this is a memory storage you should consider.
I am facing a situation where I am stuck in a very heavy traffic load and keeping the performance high at the same time. Here is my scenario, please read it and advise me with your valuable opinion.
I am going to have a three way communication between my server, client and visitor. When visitor visits my client's website, he will be detected and sent to a intermediate Rule Engine to perform some tasks and output a filtered list of different visitors on my server. On the other side, I have a client who will access those lists. Now what my initial idea was to have a Web Service at my server who will act as a Rule Engine and output resultant lists on an ASPX page. But this seems to be inefficient because there will be huge traffic coming in and the clients will continuously requesting data from those lists so it will be a performance overhead. Kindly suggest me what approach should I do to achieve this scenario so that no deadlock will happen and things work smoothly. I also considered the option for writing and fetching from XML file but its also not very good approach in my case.
NOTE: Please remember that no DB will involve initially, all work will remain outside DB.
Wow, storing data efficiently without a database will be tricky. What you can possibly consider is the following:
Store the visitor data in an object list of some sort and keep it in the application cache on the server.
Periodically flush this list (say after 100 items in the list) to a file on the server - possibly storing it in XML for ease of access (you can associate a schema with it as well to make sure you always get the same structure you need). You can perform this file-writing asynchronously as to avoid keeping the thread locked while writing the file.
The Web Service sounds like a good idea - make it feed off the XML file. Possibly consider breaking the XML file up into several files as well. You can even cache the contents of this file separately so the service feeds of the cached data for added performance benefits...
On websites that allow you to upload images how are the thumbs generated?
I know imagemagick is typically used but the question is how does it work on the db end? I suspect it has a queue in a db and it can process N images at a time.
I'm talking about asp.net based websites for the moment
How does that queue work? One thought is put code in Application_Start to launch (a) thread(s) as a deamon which reads in the db for images that need to be process and sleep once no work is required.
Is this a good way or 'proper'? which way is 'good practice'
I would not start any external threads like that within an asp.net process due to recycling.
In many cases, you can probably do it in an async page right when uploaded. If not, then perhaps a separate app (service, perhaps) which handles the queue of thumbnails needing to be generated.
On our asp.net site we have used standard .Net implementation of image re-sizing and perform it on the user post of picture. Idea behind that time of post much more longer than processing of image, so place to DB 2 images (thumb and origin) is much more faster than traverse it over HTTP.
If you really need more time processing images and video you might want to consider writing a windows service to do the processing.
You store the raw data in a temp folder and add an entry to some table in your DB which is constantly read by your service so it knows when to process data, it then reports back to your DB so you webapp knows when the data is processed. In a simpler way you could have a service monitor a folder and process whatever you put in there, moving the processed files to a 'processed files' folder.
We have an existing, proprietary data processing application that runs on one of our servers and we wish to expose it as a web service for our clients to submit jobs remotely. In essence, the system takes a set of configuration parameters and one or more data files (the number of files depends on the particular configuration template, but the normal config is 2 files). The application then takes the input files, processes them, and outputs a single result data file (all files are delimited text / CSV or tab).
We want to now expose this process as a service. Based on our current setup and existing platforms, we are fairly confident that we want to go with WCF 4.0 as the framework and likely REST for the service format, though a SOAP implementation may be required at some point.
Although I am doing a lot of reading on SOA, WCF and REST, I am interested in other thoughts on how to model this service. In particular, the one-to-many relationship of job to required files for input. It seems pretty trivial to model a "job" in REST with the standard CRUD commands. However, the predefined "job type" parameter defines the number of files that must be included. A job type of "A" might call for two input files, while "B" requires 3 before the job can run.
Given that, what is best way to model the job? Do I include the multiple files in the initial creation of the job? Do I create a job and then have an "addFile" method where by I can then upload the necessary number of files?
The jobs will then have to run asynchronously because they can take time. Once complete, is it best to then just have a status field in the job object and require the client to regularly query the system for job status, or perhaps have the client provide a URL to "ping" when the job is complete?
We are only in the planning stages for the service, so any insights would be appreciated.
To model it for REST, think of resources. Are the files part of the job resource or are they seperate resources.
If they are seperate resources then I would have a method to upload them seperately. How they link is up to you - so you could have a way to associate a file to a job when you upload the file or do you have a way to create links (now treating links as individual resources too) between existing files and jobs.
If you files are not seen as seperate resources then I would have them inline with job, as a single create.