Scenario:
localhost receives the current HttpRequest with 3 hidden inputs and a posted file. I must then forward this form data to an external image host and get the response.
See the System.Net.WebClient and related classes. You can use them to create a request to the remote server and handle the response. Also get Fiddler to help you replicate what the browser sends.
I hate doing this. It wastes my server's bandwidth and ties up IIS threads as well as using my server's CPU. It sucks and it's worth avoiding at all cost. Many services like, one that comes to mind is fliqz, provide a mechanism such that the files are uploaded directly from the client to their server (bypassing yours) and then they make a request to your server passing it various info on the query string.
Related
I am writing a web application for an academic research group. The researchers need to be able to upload large data sets (100MB - 1GB) in CSV format. I've written the server to process the data as it comes in. This means that if there is an error in the first row of the CSV, we can return an error straight away.
However, when this happens, the browser reports that "The connection was reset" or similar. Clearly, my web server is responding in a way that doesn't make sense.
If I explicitly close the HTTP request stream (this is Kotlin on the JVM by the way) before returning the error to the browser, then the problem goes away. However, it turns out that the close implementation of the request stream first goes and reads the whole stream to its end. So at that point the user still has to wait 30mins+ to find out that there is an error in the first row of their CSV.
Is what I am trying to do possible? Does the HTTP protocol permit a web server, in any circumstances, to begin responding before the full request body has been sent? If not, can you suggest a workaround that would allow me to deliver a user experience where the user doesn't have to wait for the whole file to be uploaded before finding out if there are any problems?
The answer is yes, according to the http spec servers should be able to send responses early and the client should stop sending the request body. Most browsers however, don't implement this correctly.
In theory, your http server needs to return a 4xx error code with a response body, then reset the connection to prevent the upload continuing in the background. See the answers below for a more detailed description of the issue. There are a couple of browser versions that do support this, so if you're doing this in lab conditions where you can control the client being used the links below will help.
https://stackoverflow.com/a/14483857/2274303
https://stackoverflow.com/a/18370751/2274303
[edit]
To answer your question about using a workaround, chunking the uploads using javascript is a good way to mitigate internet connectivity issues, but if you want to parse it in real time it's not as simple as arbitrarily breaking up the file into pieces. You need to make sure you're not splitting the file in the middle of a line, otherwise it will fail even if the data is valid. That brings up the issue of parsing a 1GB file in javascript, which isn't a good idea imo.
If you want to use javascript, continue uploading the entire file at once via an ajax request, so you can get the response outside of the main dom and force a redirect or cancel the upload. Depending on which js libraries you're using there are different ways of doing this.
None of this solves the reverse scenario. What if the file is 95% uploaded before there's an error? The researcher will need to either upload the whole thing again or edit the file to only include the rows from the error going forward. That means your application needs to support partial uploads and know to pick up where it left off. All these things are possible, but you're probably not going to find a simple workaround to get this working well.
Without understanding the dataset and what kind of validation you are doing it's hard to come up with a full solution. If parsing each row doesn't depend on the previous rows being valid, you could always upload the whole file, then display the rows with errors at the end and ask them to upload a second file with just the corrections.
The normal process of a HTTP web server happens like:
Server listens for request
Client creates request
Client sends request to server
Server processes request
Server creates response
Server sends response to client
Client processes response
The client starts the connection for communication and the server is able to respond on that connection, however if you close the connection the server will need to send a response on another connection. The browser may not allow the server to start a new connection that the client didn't request.
You may be able to respond by reading the first line and creating an error quickly, but the client will not read the response until it is done sending the request.
By sending the file in chunks or asynchronously sending lines of the file, you will be able to give feedback more immediately. You will be sending many smaller requests with the ability to respond in between.
The question was about HTTP protocol. I feel like this would be allowed by the protocol if you wrote a custom app and web app, however if you are using browsers then you must use HTTP as the companies have implemented it. In a custom app you could check for interruptions however most browsers will probably fire a full request before listening for a response, which is also a reason AJAX took off 20years ago.
I am writing an application using SignalR which acts as a push service of resources for clients. The service sends notifications to its clients regarding state change of resources at regular intervals. Sometimes, the resource state remains the same. In such cases I need a way to convey the same to clients without sending the same resource again. In a way, I want to implement something like ETags with SignalR. To do that I need to modify SignalR response headers (or I could use query string).
Is there a way to do this?
you are saying to pass the same state to clients (But with NOT MODIFIED kind of header) as well not to pass if it is same.
In such case you should actually not pass any message to the clients if the server hasn't anything new to give to the clients. This is how realtime applications are supposed to work.
Also instead of adding an overhead of extra HEADERS why don't you just send a simple actual signalR ping to the clients saying "Hey, nothing modified. Just chill!"
I tried out different WebDAV clients and all of them behave a little different. In this special case I use WinSCP to connect to a SabreDAV based, self written (in php), WebDAV server which emulates a filesystem.
The problem I have is that WinSCP creates - unlike several other WebDAV clients (eg BitKinex, CarotDAV) - a new php session each time I do a command (refresh, upload file, ...).
So is there a possibility to ensure reusing session (programmatically on server side) or do I need to set a specific option in WinSCP? If the latter one, I could not figure out so far.
WinSCP indeed does not process HTTP cookies.
I've never seen a WebDAV server that would require it. The cookies are definitely not mentioned in WebDAV specifications. WebDAV servers that need to authentication or track its users use HTTP authentication. Not cookies.
Anyway, I've added a request for processing cookies to WinSCP tracker:
https://winscp.net/tracker/1419
Well, I implemented a workaround that allows me to mostly reuse same session every time.
Hopefully WinSCP will process cookies one time though to be sure.
It is quite easy to update the interface by sending jQuery ajax request and updating with new content. But I need something more specific.
I want to send the response to client without their having requested it and update the content when they have found something new on the server. No need to send an ajax request every time. When the server has new data it sends a response to every client.
Is there any way to do this using HTTP or some specific functionality inside the browser?
Websockets, Comet, HTTP long polling.
It has name server push (you can also find it under name Comet technology). Do search using these keywords and you will find bunch examples, tools and so on. No special protocol is required for that.
Aaah! You are trying to break the principles of the web :) You see if the web was pure MVC (model-view-controller) the 'server' could actually send messages to the client(s) and ask them to update. The issue is that the server could be load balanced and the same request could be sent to different servers. Now if you were to send a message back to the client you'll have to know who all are connected to the server. Let's say the site is quite popular and you have about 100,000 people connecting to it every day. You'll actually have to store the IPs of each of them to know where on the internet they are located and to be able to "push" them a message.
Caveats:
What if they are no longer browsing your website? You see currently there is no way to log out automatically if you close your browser. The server needs to check after a fixed timeout if you have logged out (or you send a new nonce with every response to prevent the server from doing that check)
What about a system restart/crash etc? You'd lose all the IPs that you were keeping track of and you are back to square one - you have people connected to you but until you receive new requests you can't really "send" them data when they may be expecting it as per your model.
Let's take an example of facebook's news feeds or "Most recent" link close to the top right - sometimes while you are browsing your wall you see the number next to most recent has gone up or a new 'feed' has come to the top of your wall post! It's the client sending periodic requests to the server to find out what was updated rather than the other way round
You see, it keeps it simple and restful. You may feel it's inefficient for the client to "poll" the server to pull the data and you'd prefer push, but the design of the server gets simplified :)
I suggest ajax-pulling is the best way to go - you are distributing computation to the client and keeping it simple (KIS principle :)
Of course you can get around it, the question is, is it worth it?
Hope this helps :)
RFC 6202 might be a good read.
Since the Flash Player (or, more exactly, the URLLoader class) will not let you read HTTP response headers or cookies set by the server, and if you get hold of a session cookie through some workaround like reaching out to the browser and run JS, you can't send it to the server, because, among others, the Cookie header will be blocked.
Now I'm building a Flex client against an HTTP API for my server product. I control both sides, so I can get get around the above limitations, now I'm wondering how. I see the following options:
include the session token in the HTTP payload
include the token in the URL
build my own HTTP client (... with blackjack, and hookers ...) in AS, using the Socket class
I don't like (1), because I'm reimplementing functionality in my protocol that is already built into Struts, which I'm using to implement the server side. I then have to ensure that either both behave the same way, or turn off the usual way of session management and force other clients to use my protocol where they could just have the browser deal with it.
I don't like (2), because I understand that there are security concerns with this, although I'm not too sure which
I don't like (3), because it's 2010 and tons of HTTP clients have been written by smarter people than me.
So, are there other opportunities? Which of my "don't like"s do you reckon least severe? Are there ways to mitigate the problems I listed? For example, how insecure are session tokens in URLs really?
How about using the FlashVars parameter? It's designed explicitly to pass simple data into a Flash app, and it's trivial to embed the session token into the tag when the page is generated server-side. PHP-wise, it'd be something like
<embed href="movie.swf" flashvars="sessionID=<?= session_id(); ?>">blah blah blah</embed>
This way there's no session data in the movie's url that could leak via referers, and the data's already "there" so the app doesn't have to reach out and talk to the browser. And if someone's sniffing the source HTML page to get the data, they could have gotten the same information from the HTTP headers anyways.
There's more details here in the Adobe docs.
Some of your post may have eluded me, but do you know about Shared Objects:
"The SharedObject class is used to read and store limited amounts of data on a user's computer or on a server. Shared objects offer real-time data sharing between multiple client SWF files and objects that are persistent on the local computer or remote server. Local shared objects are similar to browser cookies and remote shared objects are similar to real-time data transfer devices.