Using Asp.net webforms I want to track down visitor info just like Google Analytics does. Of course, I can use Google Analytic for this purpose but I want to know how can I achieve the same thing with Asp.net 3.5 and SQL Server 2008.
I want to store IP, Country, URL Referrer of the visitor, Resolution on each page request except postback. I am expecting 50k+ visit everyday..
Main concern is I want to do it in a way that it should not block current request.
i.e In general it happens when we save data in to db, current request stops on particular SP calling statment and moves ahead when it finishes executing SP or tsql statement. I want to follow "Insert and Forget" approach. It should insert in background when I pass parameter to particular event or function.
I found below alternatives for this :
1. PageAsynchTask
2. BeginExecuteNonQuery
3. Jquery Post method and Webservice (But I am not confident about this, and wondering how should I go about it)
I hope I've mentioned my problem properly.
Can anybody tell me which one is better approach? Also let me know if you've any other ideas or better approach than the listed one. Your help will be really appreciated.
Problems with any background thread in server side is each and every request is going to occupy two threads. One for serving the ASP.NET request and one for logging the stuff you want to log. So, you end up having scalability issues due to exhaustion of ASP.NET threads. And logging each and every request in database is a big no no.
Best is to just write to log files using some high performance logging library. Logging libraries are highly optimized for multi-threaded logging. They don't produce I/O calls on each and every call. Logs are stored in a memory buffer and flushed periodically. You should use EntLib or Log4net for logging.
You can use an HttpModule that intercepts each and every GET, POST and then inside the HttpModule you can check whether the Request.Url is an aspx or not. Then you can read Request.Headers["__ASYNCPOST"] and see if it's "true", which means it's an UpdatePanel async update. If all these conditions are true, you just log the request into a log file that stores the
You can get the client IP from:
HttpContext.Current.Request.UserHostAddress;
or
HttpContext.Current.Request.ServerVariables["REMOTE_ADDR"];
To get the IP address of the machine and not the proxy use the following code
HttpContext.Current.Request.ServerVariables["HTTP_X_FORWARDED_FOR"];
However you cannot get the country. You will have to log the IP in your log files and then process the log files using some console application or job which will resolve the country of the IP. You need to get some IP->Country database to do the job. I have used http://www.maxmind.com/app/geoip_country before.
For screen size, you will have to rely on some javascript. Use a javascript on each page that finds the size of the screen on the client side and stores in a cookie.
var screenW = 640, screenH = 480;
if (parseInt(navigator.appVersion)>3) {
screenW = screen.width;
screenH = screen.height;
}
else if (navigator.appName == "Netscape"
&& parseInt(navigator.appVersion)==3
&& navigator.javaEnabled()
)
{
var jToolkit = java.awt.Toolkit.getDefaultToolkit();
var jScreenSize = jToolkit.getScreenSize();
screenW = jScreenSize.width;
screenH = jScreenSize.height;
}
Once you store it in a cookie (I haven't shown that code), you can read the screen dimensions from the HttpModule by using Request.Cookies and then log it in the log file.
So, this gives you solution for logging IP, screensize, finding country from IP, and filtering UpdatePanel async postback from logging.
Does this give you a complete solution to the problem?
Talking about server side, if you're running on IIS, and you don't need absolute real time information, I recommend you use IIS logs.
There is nothing faster than this, as it's been optimized for performance since IIS 1.0
You can append your own information in these logs (HttpRequest.AppendToLog), they have a standard format, there is an API if you want to do custom things with it (but you can still use text parser if you prefer), and there are a lots of free tools, for example Microsoft Log Parser which can transfer data in a SQL database (among others).
The first approach is looking good. (And I recommend it.) But it has 2 downsides:
Request will still block until task completes (or aborts on timeout).
You'll have to register your Task on every page.
The second approach looks inconvenient and might lead to errors. (You have to watch for a situation when your page renders faster than your query is processed. I'm not sure what will happen if your query is not complete when your page object is destroyed and GC goes on finalize() spree... but nothing good, I assume. You can avoid it by waiting for IAsyncResult.IsCompleted after render, but that's inconvenient.)
The third method is plain wrong. You should initiate your logging on the server side while processing the request you're going to log. But you still can call a web service from the server side. (Or a win service).
Personally, I'd like to implement logging in BeginRequest to avoid code duplication, but you need IsPostback... Still there might be a workaround.
Hii,
you can fire an asynchronous request and don't wait for response.
here i have some implemented code..
for that you need to create a web service to do your database operation or you can use it for your whole event handling.
from server side you have to call the web service asynchronously like this
Declare a Private Delegate
private delegate void ReEntryDelegate(long CaseID, string MessageText);
Now the method will contain web service calling like this
WebServiceTest.Notification service = new WebServiceTest.Notification();
IAsyncResult handle;
ReEntryDelegate objAscReEntry = new ReEntryDelegate(service.ReEntryNotifications);
handle = objAscReEntry.BeginInvoke(CaseID, MessageText, null, null);
break;
And the variable values will be passed by method here (CaseID,MessageText)
Hope this is clear to you
All the Best
Related
I'm working on an API (Pragmatic Rest API or very similar). I would like to know if it is possible to do an API request that will return a quick response (in JSON) and continue to process heavy code in background.
I suppose this is possible by using queue system but I have no idea where to start with this.
You can have your API delegate long running things to another process.
You mentioned queues, that's one way of doing things, all you need really is an application which can execute whatever long running tasks you have.
Let's imagine a simple system that can do this.
Your API receives a request to do something.
Instead of doing this something, the API writes one record into a database with the details of what needs to be done. Another app watches that table, sees a new record, runs the thing, updates the record with the status / result / whatever it needs.
On any requests from now on, the API can check the record and return whatever is there.
This is the simplest thing I can think of. You can easily do other things as well, talk to a queue system, send it data, let something else execute it.
Looking at your comments, what you are suggesting is not really a good way of building APIs. Why do I say this?
Well, let's say that you receive a request, the API starts a work thread and sends back a 200 to the client. Great the client knows work has started and how does it know when that process had ended and how does it receive whatever data it expects back?
Let's go a bit deeper next.
What happens when 1000 clients call that one endpoint and your API is attempting to start 1000 work threads? You've killed your API, no work gets done and no client gets anything.
This is why I suggest to delegate the work to something else, not the API. Let the API do what it does best, run quick things and return results and delegate other things to something else.
I have an ASP.NET web page that connects to a number of databases and uses a number of files. I am not clear what happens if the end user closes the web page before it was finished loading i.e. does the ASP.NET life cycle end or will the server still try to generate the page and return it to the client? I have reasonable knowledge of the life cycle but I cannot find any documentation on this.
I am trying to locate a potential memory leak. I am trying to establish whether all of the code will run i.e. whether the connection will be disposed etc.
The code would still run. There is a property IsClientConnected on the HttpRequest object that can indicate whether the client is still connected if you are doing operations like streaming output in a loop.
Once the request to the page is generated, it will go through to the unload on the life cycle. It has no idea the client isn't there until it sends the information on the unload.
A unique aspect of this is the Dynamic Compilation portion. You can read up on it here: http://msdn.microsoft.com/en-us/library/ms366723
For more information the the ASP.NET Life Cycle, look here:
http://msdn.microsoft.com/en-us/library/ms178472.aspx#general_page_lifecycle_stages
So basically, a page is requested, ASP.NET uses the Dynamic Compilation to basically create the page, and then it attempts to send the page to the client. All code will be run in that you have specified in the code, no matter if the client is there or not to receive it.
This is a very simplified answer, but that is the basics. Your code is compiled, the request generates the response, then the response is sent. It isn't sent in pieces unless you explicitly tell it to.
Edit: Thanks to Chris Lively for the recommendation on changing the wording.
You mention tracking down a potential memory leak and the word "connection". I'm going to guess you mean a database connection.
You should ALWAYS wrap all of your connections and commands in using clauses. This will guarantee the connection/command is properly disposed of regardless of if an error occurs, client disconnects, etc.
There are plenty of examples here, but it boils down to something like:
using (SqlConnection conn = new SqlConnection(connStr)) {
using (SqlCommand cmd = new SqlCommand(conn)) {
// do something here.
}
}
If, for some reason, your code doesn't allow you to do it this way then I'd suggest the next thing you do is restructure it as you've done it wrong. A common problem is that some people will create a connection object at the top of the page execution then re-use that for the life of the page. This is guaranteed to lead to problems, including but not limited to: errors with the connection pool, loss of memory, random query issues, complete hosing of the app...
Don't worry about performance with establishing (and discarding) connections at the point you need them in code. Windows uses a connection pool that is lightning fast and will maintain connections for as long as needed even if your app signals that it's done.
Also note: you should use this pattern EVERY TIME you are using an un-managed class. Those always implement IDisposable.
I'm using Spring MVC 3 + Tiles for a webapp. I have a slow operation, and I'd like a please wait page.
There are two main approaches to please wait pages, that I know of:
Long-lived requests: render and flush the "please wait" bit of the page, but don't complete the request until the action has finished, at which point you can stream out the rest of the response with some javascript to redirect away or update the page.
Return immediately, and start processing on a background thread. The client polls the server (in javascript, or via page refreshes), and redirects away when the background thread finishes.
(1) is nice as it keeps the action all single-threaded, but doesn't seem possible with Tiles, as each JSP must complete rendering in full before the page is assembled and returned to the client.
So I've started implementing (2). In my implementation, the first request starts the operation on a background thread, using Spring's #Async annotation, which returns a Future<Result>. It then returns a "please wait" page to the user, which refreshes every few seconds.
When the please wait page is refreshed, the controller needs to check on the progress of the background thread. What is the best way of doing this?
If I put the Future object in the Session directly, then the poll request threads can pull it out and check on the thread's progress. However, doesn't this mean my Sessions are not serializable, so my app can't be deployed with more than one web server (without requiring sticky sessions)?
I could put some kind of status flag in the Session, and have the background thread update the Session when it is finished. I'm very concerned that passing an HttpSession object to a non-request thread will result in hard to debug errors. Is this allowed? Can anyone cite any documentation either way? It works fine when the sessions are in-memory, of course, but what if the sessions are stored in a database? What if I have more than one web server?
I could put some kind of status flag in my database, keyed on the session id, or some other aspect of the slow operation. It seems weird to have session data in my domain database, and not in the session, but at least I know the database is thread-safe.
Is there another option I have missed?
The Spring MVC part of your question is rather easy, since the problem has nothing to do with Spring MVC. See a possible solution in this answer: https://stackoverflow.com/a/4427922/734687
As you can see in the code, the author is using a tokenService to store the future. The implementation is not included and here the problems begin, as you are already aware of, when you want failover.
It is not possible to serialize the future and let it jump to a second server instance. The thread is executed within a certain instance and therefore has to stay there. So session storage is no option.
As in the example link you could use a token service. This is normally just a HashMap where you can store your object and access it later again via the token (the String identifier). But again, this works only within the same web application, when the tokenService is a singleton.
The solution is not to save the future, but instead the state of the work (in work, finished, failed with result). Even when the querying session and the executing threads are on different machines, the state should be accessible and serialize able. But how would you do that? This could be implemented by storing it in a database or on the file system (the example above you could check if the zip file is available) or in a key/value store or in a cache or in a common object store (Terracota), ...
In fact, every batch framework (Spring Batch for example) works this way. It stores the current state of the jobs in the database. You are concerned that you mix domain data with operation data. But most applications do. On large applications there is the possibility to use two database instances, operational data and domain data.
So I recommend that you save the state and the result of the work in a database.
Hope that helps.
Say, for example, you are caching data within your ASP.NET web app that isn't often updated. You have another process running outside of the app which ocassionally updates this data, when you do this you would like the cached data to be cleared immediately so that the next request picks up the new data straight away.
The caching service is running in the context of your web app and not externally - what is a good method of calling into the web app to get it to update the cache?
You could of course, just hack a page or web service together called ClearTheCache that does it. This can then be called by your other process. Of course you don't want this process to be externally useable or visible on your web app, so perhaps you could then check that incoming requests to this page are calling localhost, if not throw a 404. Is this acceptable? Could this be spoofed at all (for instance if you used HttpApplication.Request.Url.Host)?
I can think of many different ways to go about this, mainly revolving around creating a page or web service and limiting requests to it somehow, but I'm not sure any are particularly elegant. Neither do I like the idea of the web app routinely polling out to another service to check if it needs to execute something, I'd really like a PUSH solution.
Note: The caching scenario is just an example, I could use out-of-process caching here if needed. The question is really concentrating on invoking code, for any given reason, within a web app externally but in a controlled context.
Don't worry about the limiting to localhost, you may want to push from a different server in future. Instead share a key (asymmetrical or symmetrical doesn't really matter) between the two, have the PUSH service encrypt a block of data (control data for example) and have the receiver decrypt. If the block decrypts correctly and the data is readable you can safely assume that only the service that was supposed to call you has and you can perform the required actions! Not the neatest solution, but allows you to scale beyond a single server.
EDIT
Having said that an asymmetrical key would be better, have the PUSH service hold the private part and the website the public part.
EDIT 2
Have the PUSH service put the date/time it generated the cipher text into the data block, then the client can be sure that a replay attack hasn't taken place by ensuring the date/time is within an acceptable time period (say a minute).
Consider an external caching mechanism like EL's caching block, which would be available to both the web and the service, or a file to cache data to.
HTH.
I am thinking on the following approach but not sure if its the best way out:
step1 (server side): A TaskMangaer class creates a new thread and start a task.
step2 (server side): Store taskManager object reference into the cache for future reference.
step3 (client side): Use periodic Ajax call to check the status of the task.
Basically the intention is to have a framework to run a background task (5mins approx) and provide regular feedback on the web UI for the percentage of task completed.
Is there a neat way around this or any existing asp.net API that will be helpful ?
Edit 1#: I want to run the task in-proc with the app.
Edit 2#: Looks like badge implementation on stack overflow is also using the cache to track background task. https://blog.stackoverflow.com/2008/07/easy-background-tasks-in-aspnet/
I think the problem with storing the result in the cache is that ASP.NET might scavenge that cache entry for other purposes (ie if its short on memory, if its grumpy, etc). Something that is served from the cache should be something you can recreate on demand if its not found in the cache, the ASP.NET runtime is free to dump cache entries whenever it feels like it.
The usage of the cache in the badge discussion seems fundamentally different, in that case the task was shortlived. The cache was just being used as a hacky timer to fire off the task periodically.
Can you confirm this is a task that is going to take 5 minutes, and require its own thread that whole time? This is a performance concern in itself, you will only be able to support a limited number of such requests if each requires its own thread for so long. Only if thats acceptable would I let the task camp a thread for so long.
If its ok for these tasks to camp a thread, then I'd just go ahead and store the result in a dictionary global to the process. The key of the dictionary would correlate to the client request / AJAX callback series. The key should incorporate the user ID as well if security is at all important.
If you need to scale up to many users, then I think you need to break the task down into asynchronous steps, and in that case I'd probably use a DB table to store the results (again keyed per request / user).
Microsoft Message Queuing was built for scenarios like the one you try to solve:
http://www.microsoft.com/windowsserver2003/technologies/msmq/default.mspx
Windows Communicatio Foundation also has message queuing support.
Hope this helps.
Thomas
One approach for doing this is to use application state. When you spawn a worker thread, pass it a request ID that you generate, and return this to the client. The client will then pass that request ID back to the server in its AJAX calls. The server will then fetch the status using the request ID from application state. (The worker thread would be updating the application state based on its status).
I saw an approach to a similar problem somewhere. The solution was something like:
Start the background task on server.Return immediately with a url to the result.
Until the result is posted, this url will return 404.
The client checks periodically for this url.
The client reads the results when
they are finally posted.
The url will be something like http://mysite/myresults/cffc6c30-d1c2-11dd-ad8b-0800200c9a66.
The best document format is probably JSON.
If feedback on progress is important, modify the document to also contain status (inprogress/finish) and progress (42 %).