I need to generate thousands of personalized letters from more than 800 workstations in diverse locations.
While we had only a dozen of customers per location, we used Crystal Reports over the web and its PDF/Word export function to produce multi-page document for print. With thousand of customers this just doen't work for the following reasons:
Browser unable to reliably download huge report
Server resourses insufficient to handle simultanious requests from several workstations
Even if Server and Client are Ok, you cannot start printing until whole reort is downloaded (i.e. for hours)
Workstations only have browsers and local high perfromance printers at the moment.
What is the solution for this problem? How should I do this?
In the past, we had a similar challenge. To solve it, we created a C# application which was installed at each location. It would communicate with the server via web services to get the data it needed to print.
Then we used the native printing functionality built into .NET for the actual printing. The downloads were quick, and it could handle high volume printing.
Since the question was clarified as:
Remote warehouse only has browser and printer. It connects to the server and enters information about the item that just arrived. In responce to that, printing of thousands of shipping labels should start, warehouse cannot wait until labels are printed elswhere and delivered to its location.
...the rational way to do this would be to make all the printers network printers. The workstations could share them using Windows networking, or pick your own favourite print server technology.
The web application needs to be told which printer needs the labels. It then prints to whichever print server is appropriate.
The alternative - download the document and print locally - is too user-driven to be appropriate in my opinion.
Related
We have a system running Windows Server 2008R2 x64 and SQL Server 2008R2 x64 with SSRS installed/configured. This is a shared reporting server used by a large number of people, with some fairly large inefficient databases (400-500gb of data ish), and these users use the system to generate ad-hoc reports based of a reporting model that sits on top of the aforementioned databases. Note that the users are using NTLM to logon and identify for running reports.
Most reports are quick, but if you are running a report for 1 or 2 years worth of data, they can take a while to return (5minutes ish). This is fine for most users, however some of the users are stuck behind a proxy, which has a connection timeout set at 2minutes. As SSRS 2008R2 does not seem to send back a "keep-alive" signal (confirmed via wireshark), when running one of these long reports the proxy server thinks the connection has died, and as such it just gives up and kills the connection. This gives the user a 401 or 503 error and obviously cancels the report (the incorrect error is a known bug in SSRS which Microsoft refuse to fix).
We're getting a lot of flak from the user's about this, even though it's not really our issue..so I am looking for a creative solution.
So far I have come up with:
1) Discovering some as yet unknown setting for SSRS that can make it keep the connection alive.
2) installing our own proxy in between the users and our reports server, which WILL send a keep-alive back (not sure this will work and it's a bit hacky, just thinking creatively!)
3) re-writing our reports databases to be more efficient (yes this is the best solution, but also incredibly expensive)
4) ask the experts :) :)
We have a call booked in with Microsoft Support to see if they can help - but can any experts on Stack help out? I appreciate that this may be a better question for server fault (and I may post it there) but it's a development question too really :)
Thanks!
A few things:
A. For SSRS overall on it's service:
I personally use a keep alive service as I believe the default recycle is 12 hours for SSRS server. I use a tool someone turned me onto called 'VisualCron' that can do many task processes automatically. You can also just make a call in a WCF service or similar to. Basically I know the first report from a user for the day is generally slow. Usually you need to hit http:// (servername)/ReportServer to keep it alive.
B. For cachine report level items:
If this does not help I would suggest possibly caching DataSets when possible. Some people have data that is up to the moment but for a lot of people that is not the case. You may create a shared dataset in SSRS and then cache that on a schedule. So if you have domain like tables that only need to be updated once in a blue moon put them there. Same with data that is nightly or in batches. If you are transactional based shop that is up to the moment this may not help but for batch based businesses this can help tremendously.
You can also cache the reports for their data as a continuation of this. Under 'Manage' drop down for a report when in the /Reports landing page you can set the data to run under a specific schedule. You can also set a snapshot which is an extension of this as it executes with some default parameters set on a schedule and is a copy of the report when it was ran.
You are mentioning ASP.NET so I am not certain how much some of this will work if you are doing this all through a site you are setting up internally as a pass through. But you could email or save files on a schedule as well through SSRS's subscription service.
C. Change how you store your data for reporting.
You can create a Report Warehouse of select item level values of queries. Create a small database that is just a few recent years of data and only certain fields and certain tables. Then index it to death and report off of that. In my experience this method will fly in terms of performance but it does take the extra overhead of setting it up. Generally most companies will whine about this but it often takes a single day to set up and then you create one SSMS job that does it all nightly or an SSIS package then you don't worry about it. I like this method as I know my data is not being reported off of production and is isolated personally.
I have a asp.net 4.0 web application that uses linqtosql to insert/update/delete data from a sql server 2005 database.
The application is elastic and therefore fits on any device... phone, ipad or desktop.
On phones and ipads users will normally have a data plan.
What I want to know is when a user is connected to my application, but not inserting/updating/deleting data, is it affecting data usage?
Also is there data usage if a user is just looking at pages or does data usage only occur when data is being inserted/updated/deleted?
Thanks
I can't tell if you're trolling or not...
But if you aren't, then you don't understand how web applications work. If it appears in the browser window then it means the user agent (the browser in the device) downloaded it from the web server, that download eats into the device's data allowance. The fact that data was possibly moved around on the server side is irrelevant.
When a web user agent (i.e. a browser) submit data back to the server (usually via a POST request) then that too will eat into the data allowance, but HTML forms typically consume less than a kilobyte of data when being sent from the client (unless you're uploading binary files or large textareas). Data allowances are usually bi-directional, so you might have 1GB to use for both upstream and downstream - the few kilobytes you use uploading a form's data are massively outweighed by the fact that most webpages thesedays weigh a few hundred kilobytes, if not more.
Of course, the right answer in this case is "profile!" - run your application yourself and see how much data you go through when performing common tasks, record your findings, and act accordingly.
At the company I work for, we are building a data warehousing application which will be a web based front end for a lot of queries that we run. [Using ASP.net]
Some of these queries will bring in over a million records (and by year end maybe around 2 million records) -
Most will be thousands, etc.
What is a good way to architect the application in such a way that you can browse to the query that you want, export it and have a CSV file generated of the data requested -
I was thinking a web based interface that calls BCP to generate a file, and shows you when the report has been created so that it can be downloaded, and expires within 24 hours of being created -
Any Ideas?
Sas
I architected something like this in a former life. Essentially a C# command-line app that ran queries against our reporting server.
Some key points:
It didn't matter how long they took because it was in the background - when a file was ready, it would show up in the UI and the user could download it. They weren't waiting in real time.
It didn't matter how inefficient the query was - both because of the point above, and because the reports were geared to the previous day. We ran them off of a reporting copy of production, not against production, which was kept on a delay via log shipping.
We didn't set any expiration on the files because the user could request a report on a Friday and not expect to look at it until Monday. We'd rather have a file sitting around on the disk than run the report again (file server disk space is relatively cheap). We let them delete reports once they were done with them, and they would do so on their own to prevent clutter in the UI.
We used C# and a DataReader rather than bulk export methods because of various requirements for the data. We needed to provide the flexibility to include column headers or not, to quote data or not, to deal with thousand separators and decimal points supporting various cultures, to apply different line endings (\r\n, \n, \r), different extensions, Unicode / non-Unicode file formats, and different delimiters (comma, tab, pipe, etc). There were also different requirements for different customers and even for different reports - some wanted an e-mail notification as soon as a report was finished, others wanted a copy of the report sent to some FTP server, etc.
There are probably a lot of requirements you haven't thought of yet but I hope that gives you a start.
What are some good automated tools for load testing (stress testing) web applications, that do not use record and replay of HTTP network packets?
I am aware that there are numerous load testing tools on the market that record and replay HTTP network packets. But these are unsuitable for my purpose, because of this:
The HTTP packet format changes very often in our application (e.g. when
we optimize an AJAX call). We do not want to adapt all test scripts just because
there is a slight change in HTTP packet format.
Our test team shall not need to know any internals about our application
to write their test scripts. A tool that replays HTTP packets, however, requires
the team to know the format of HTTP requests and responses, such that they
can adapt details of the replayed HTTP packets (e.g. user name).
The automated load testing tool I am looking for should be able to let the test team write "black box" test scripts such as:
Invoke web page at URL http://... .
First, enter XXX into text field XXX.
Then, press button XXX.
Wait until response has been received from web server.
Verify that text field XXX now contains the text XXX.
The tool should be able to simulate up to several 1000 users, and it should be compatible with web applications using ASP.NET and AJAX.
JMeter I've found to be pretty helpful, it also has a recording functionality to record use cases so you don't have to specify each GET/POST manually but rather "click" the use case once and then let JMeter repeat it.
http://jmeter.apache.org/
A license can be expensive for it (if you dont have MSDN), but Visual Studio 2010 Ultimate edition has a great set of load and stress testing tools that do what you describe. You can try it out for free for 90 days here.
TestMaker by PushToTest.com can run recorded scripts such as Selenium as well as many different languages like HTML, Java, Ruby, Groovy, .Net, VB, PHP, etc. It has a common reporting infrastructure and you can create load in your test lab or using cloud testing environments like EC2 for virtual test labs.
They provide free webinars on using open source testing tools on a monthly basis and there is one next Tuesday.
http://www.pushtotest.com
There are a few approaches; I've been in situations, however, where I've had to roll my own load generating utilities.
As far as your test script is concerned it involves:
sending a GET request to http://form entry page (only checking if a 200 response is given)
sending a POST request to http://form submit page with pre-generated key/value pairs for text XXX and performing a regexp check on the response
Unless your web page is complex AJAX there is no need to "simulate a button press" - this is taken care of by the POST request.
Given that your test consists of just a 2-step process there should be several automated load packages that could do this.
I've previously used httperf for load testing a large website: it can simulate a session consisting of several requests and can simulate a large number of users (i.e. sessions) simultaneously. For example, if your website generated a session cookie from the home page you could make that the first request, httperf would then use that cookie for subsequent requests, until it had finished doing the list of requests supplied.
What about http://watin.sourceforge.net/ ?
We had huge performance problem when deploying our ASP.NET app at a customer, which had the DB sitting on a remote location.
We found that it was due to the fact that pages made ridiculous amount of individual SQL queries to the DB. It was never a problem we noticed because usually, the web and DB are on the same local network (low latency). But on this (suddenly) low latency configuration, it was very very slow.
(Notice that each sql request by itself was fast, it is the number and serial nature of the sequence that is the problem).
I asked the engineering team to be able to report and maintain a "wall of shame" (or stats) telling us for each page the number of SQL requests so we can use it as a reference. They claim it is expensive..
anyone can tell me how to be able to maintain or get such report cheaply and easily?
We are using SQL Server 2005
We have a mix of our own DB access layer and subsonic
I know and use the profiler, but that is a bit manual. Asking here if there is a tip on how to automate or maybe I am just crazy?
If you are on SQL Server, read up on Profiler.
http://msdn.microsoft.com/en-us/library/ms187929.aspx
Running profiler from the UI is expensive, but you can run traces without the UI and that will give you what you want.
First, check out SubSonic's BatchQuery functionality--it might help alleviate alot of the stress in the first cut without getting into material modification of your code.
You can schedule trace jobs/dumps from the SQL server's end of things. You can also run perfmon counters to see how many database requests the app is serving.
All that said, I'd try and encourage the customer to move the database (or a mirrored copy of the database) closer to your app. It is probably the cheapest solution in the long term, depending on how thick the app is.
I have had good success using this tool in the past, not sure if the price is right for you but it will uncover any issues you may have:
Spotlight on SQL Server
The MiniProfiler (formerly known as the MVC mini profiler; but it works for all both MVC and Webforms) is a must in such a case IMO. If the code creating the database connections is well architectured it's a piece of cake to get it running for almost any ASP.NET application.
It generates a report on each rendered page with profiling stats, including each SQL query sent to the database for the request. You can see it in action on the Stack Exchange Data Explorer pages (top left corner).