I know I should probably use WebSockets or Server-Side Events, but what happened to the Comet Streaming techniques that involved writing chunked data from a server to either an iFrame or as a response to a xmlHttpRequest? I have stumbled upon multiple demos but none of them work as intended any more and since most of the material on this type of streaming is quite old, I'm wondering whether or not it is still doable in the year of 2015?
Just to be clear, I'm referring to the Comet techniques where the server keeps a connection open using chunked transfer encoding flushing new data on the fly. Incremental rendering in browsers should presumably make this data available, either as it comes in in the case of an iFrame (e.g. "Forever iFrame") or by reading the responseText property of the xmlHttpRequest object when its readyState returns 3 (e.g. "XHR streaming"). However, all browsers seem to buffer the data until the connection is closed, no matter how much bogus preamble I add before starting to send real data. Also, I'm not referring to the special case with Content-Type set to multipart/x-mixed-replace which works in Firefox only, but an approach which seems to have worked on most of the browsers a few years back.
Does anyone know if current browser behaviour has obsoleted these Comet streaming techniques?
Example demos:
Polling responseText on load through xmlHttpRequest object:
http://ajaxify.com/run/streaming/xmlHttpRequest/countdown/
http://ajaxify.com/run/streaming/xmlHttpRequest/
Slow loading of regular page, incremental rendering does NOT kick in for me:
http://ajaxify.com/run/streaming/
Polling iFrame content on slow iFrame load:
http://ajaxify.com/run/streaming/xmlHttpRequest/iframe/
These demos does not use chunked transfer encoding but resulting behaviour is the same, that is, incremental rendering isn't happening.
Article on Comet streaming:
http://cometdaily.com/2007/11/05/the-forever-frame-technique/
Short version conclusion:
All the Comet techniques still work, however, in my case the antivirus got in the way of things in an unforeseen manner so make sure to test things on different computers and in safe mode as well to be sure if it doesn't work!
Related
I do understand the basic idea of it. But most of the implementations I've seen have done nothing but confused me - I find myself incapable of fully understanding the concept of Comet and long polling... Simply put, I ask for simple explanation of these ideas. I am especially interested in an explanation of the hidden iframe polling technique. What gets executed, what gets requested etc.
Also, what are the advantages of it over the classic ajax approach? (besides the reduced traffic and more real-time feeling).
Thanks.
The technique is very nicely explained in the following article. The core idea resides on the chunked transfer encoding HTTP technique. A hidden iframe is included in the page which points to a server side script which uses chunked encoding. In chunked encoding the response is not sent entirely in one go and the stream closed. The server doesn't say in advance how much data is going to send so the browser keeps the channel open. Then when the server wants to push some data to the client it simply sends a chunk of response which represents a javascript function. The browser receives and executes this function on the client. This way the server can successfully PUSH information when some events occur like for example some data changes on the server, ...
Also, what are the real advantages of it over the classic ajax
approach? (besides the reduced traffic and more real-time feeling).
Aren't those advantages sufficient? Reduced traffic means more responsive application. Did you know that large sites like Google and Amazon conducted studies and explicitly throttled down their servers in order to increase the response time with a couple of milliseconds. I can't remember the exact but they were flagrant: they lost like 70% of their customers after doing that. Remember: the most important feature of a web application (and not only by the way) is its responsiveness.
So it's basically PULL (Ajax) vs PUSH (Comet). PUSH techniques scale better when the number of clients starts to increase.
I just got hammered on a Security Audit by Deloitte on behalf of SFDC. Basically we use flex and communicate via AMF. We use FluorineFX for this (as opposed to LCDS and Blaze). We are being told that because the AMF response is not encoded and that someone can manipulate the AMF parameters and insert Javascript that this is a XSS vulnerability. I'm struggling to understand how the AMF response back, which could echo the passed in JS in an error message, can be executed by the browser or anything else for that matter. I'm quite experienced with XSS with HTML and JS but seeing it get tagged with AMF was a bit of a surprise. I'm in touch with FluorineFx team and they are perplexed as well.
I'd be surprised to see an AMF library encode the response data, Fluorine surely does not. It would seem though that security applications like PortSwigger and IBM AppScan are including this type of test in their tool chest. Have you run into this vulnerability with AMF and can you explain how the XSS issue can manifest itself? Just curious. I need to either argue my way out of this if an argument exists or patch the hole. Given the AMF usage with Flex I thought you might have some insight.
Additional information ...
So A little more on this from the actual vendor, PortSwigger. I posed the question to them and net, net, they concede this type of attack is extremely complicated. Initially they are classifying this as a High Severity security issue but I think their tune is changing now. I thought I'd post the content of their response for you all as I think the perspective is interesting none-the-less.
--- From PortSwigger on the issue ---
Thanks for your message. I think the answer is that this is potentially a
vulnerability, but is not trivial to exploit.
You're right, the issue wouldn't arise when the response is consumed by an
AMF client (unless it does something dumb), but rather if an attacker could
engineer a situation where the response is consumed by a browser. Most
browsers will overlook the HTTP Content-Type header, and will look at the
actual response content, and if it looks at all like HTML will happily
process it as such. Historically, numerous attacks have existed where people
embed HTML/JS content within other response formats (XML, images, other
application content) and this is executed as such by the browser.
So the issue is not so much the format of the response, but rather the
format of the request required to produce it. It's not trivial for an
attacker to engineer a cross-domain request containing a valid AMF message.
A similar thing arises with XML requests/responses which contain XSS-like
behaviour. It's certainly possible to create a valid XML response which gets
treated by the browser as HTML, but the challenge is how to send raw XML in
the HTTP body cross-domain. This can't be done using a standard HTML form,
so an attacker needs to find another client technology, or browser quirk, to
do this. Historically, things like this have been possible at various times,
until they were fixed by browser/plugin vendors. I'm not aware of anything
that would allow it at the moment.
So in short, it's a theoretical attack, which depending on your risk profile
you could ignore altogether or block using server-side input validation, or
by encoding the output on the server and decoding again on the client.
I do think that Burp should flag up the AMF request format as mitigation for
this issue, and downgrade the impact to low - I'll get this fixed.
Hope that helps.
Cheers
PortSwigger
--- more info on audit ---
what portSwigger does is not necessarily mess with binary payload, what they do is mess with the actual AMF parameters that are posted to the handler to direct the request. For example here is a snippet from the audit and it shows part of the AMF response to a request ...
HTTP/1.1 200 OK
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
P3P: CP="CAO PSA OUR"
Content-Type: application/x-amf
Vary: Accept-Encoding
Expires: Tue, 06 Apr 2010 18:02:10 GMT
Date: Tue, 06 Apr 2010 18:02:10 GMT
Connection: keep-alive
Content-Length: 2595
......../7/onStatus.......
.SIflex.messaging.messages.ErrorMessage.faultCode.faultString
.faultDetail.rootCause.extendedData.correlationId.clientId.destination
.messageId.timestamp.timeToLive body.headers.#Server.Processing..kFailed
to locate the requested type
com.Analytics.ca.Services.XXX5c2ce<script>alert(1)</script>9ccff0bda62..
....I506E8A27-8CD0-598D-FF6E-D4490E3DA69F.Id95ab281-d83b-4beb-abff-c668b9fd42d5
..fluorine.I04165c8e-f878-447f-a19a-a08cbb7def2a.A.q..#............
. DSId.Aeb5eeabcbc1d4d3284cbcc7924451711.../8/onRes
...[SNIP]...
note the "alert" script in there ... what they did was appended some script enclosed JS to one of the parameters that are passed containing the method to call namely 'com.Analytics.ca.Services.XXX'. By doing so the JS came back in an error message but there are a lot of things that would have to happen for that JS to get anywhere close to executing. Seems an indirect threat at best.
-- Security Auditor's latest perspective --
I’ve discussed with the larger team and we all believe it’s a valid attack. As PortSwigger mentions in his first paragraph, while theoretically since you set the content-type to x-amf, and would hope it won’t render in the browser, most browsers will ignore this request and render it anyway. I think the vendors are relying heavily on the fact that the content-type is set; however popular browsers like IE and some versions of Safari will ignore this.
The attack can easily be triggered by exploiting CSRF or any other form of initiating an XSS attack.
It could not be a JavaScript injection, as what in the Flash Player would interpret JS? The flash community would be ecstatic if we had native JS or even json support in the player. There is no eval function for actionscript let alone javascript
Let's assume they meant you could inject it with actionscript. The AMF protocol does not send code, it sends data models in the form of primitive types or generic or typed objects. The worst thing that could happen is that they analyze your model and add additional data. This would be amazingly difficult to do as you would not be able to inject the data but would have to parse all the data, add the new data, parse it back and keep the AMF headers. Because AMF uses references in it's data serialization which means that when duplicate object types you would have had to of seen the first object. The reference is then an offset which means little chance of adding code but only changing values to existing parameters.
The remote object has a response handler that is checking for the data types and expects to bind those data types to ui components or whatever your code does. If those data types are wrong you will get an error. If the AMF response sequence number is wrong you will get an error. If anything is not perfectly formed in the amf datagram you will get an error.
Remote object automatically retry. If the "injecting" code takes to long Flex will resend a message and invalidate the one that took to long.
Just my two cents. As an AMF developer I have frequently wished it was easy to screw with the amf datagram for debugging and testing. Unfortunately you will get an error.
Wade Arnold
You seem to have answered your own queries here.
So you have a server side implementation that takes the arguments to an amf function call and includes the input data somewhere in the returned output.
I appreciate that this is largely a theoretical attack as it involves getting the payload to be rendered by the browser and not into an amf client. Other vulnerabilities in browsers/plugins may be required to even enable this scenario. Maybe a CSRF post via the likes of a gateway.php or similar would make this pretty easy to abuse, as long as the browser processed the output as html/js.
However, unless you need the caller to be able to pass-through angle brackets into the response, just html-encode or strip them and this attack scenario dissapears.
This is interesting though. Normally one would perform output-encoding solely for the expected consumer of the data, but it is interesting to consider that the browser could often be a special case. This really is one hell of an edge-case, but i'm all for people getting into the habit of sanitising and encoding their untrusted inputs.
This reminds me, in many ways, to the way that cross-protocol injection can be used to abuse the reflection capabilities of protocols such as smtp to acheive XSS in the browser. See http://i8jesus.com/?p=75
I can't explain how someone would take advantage of this "vulnerability".
But, can you solve the issue to their satisfaction by passing data over an HTTPS connection instead of straight HTTP? Assuming you have an SSL certificate installed on your server and HTTPS enabled, this should be a minor change in the services-config.xml file that you compile into your Flex Application.
I pinged an Adobe colleague of mine in hopes that he can offer more insight.
I think it is a valid attack scenario. A related attack is GIFAR, where the JVM is fooled to treat a gif file as a jar. Also, I don't think output encoding is the right way to solve the problem.
The premise of the attack is to fool the browser into thinking the AMF response is HTML or Javascript. This is possible because of a feature called MIME Type Detection, which is essentially the browser saying "Developers may not know about content-types, I will play god and (possibly incorrectly) figure out the MIME type".
In order for this to work, the following need to hold true -
The attacker should be able to make a GET or POST request to your AMF server using HTML techniques like <script> or <frame> or an <a> tag. Techniques like XmlHttpRequest or Flash or Silverlight don't count.
The attacker should be able to insert malicious content into the first 256 or so bytes of the response. Additionally, this malicious content should be able to trick the browser in thinking that the rest of the response is really javascript or html.
So, how do you prevent it?
It is best to ensure the attacker cannot make a request in the first place. A very simple and effective way is to add a http request header while making the AMF request, check its existence on the server and deny the request if absent. The value can be a hard-coded value and need not be secret. This works, because there is no known method of adding a custom request header via standard html techniques. You can do so via XmlHttpRequest or flash or silverlight, but then the browser will not interpret the content-type for you, so its okay.
Now, I don't know much about AMF, but if it is already adding a request header - then this attack scenario is not possible. If it isn't, its trivial to add one.
HTML escaping the content is not a good solution. Allegedly, there are various ways to trick the browser into thinking the response is actually HTML. In other words, the malicious input need not be well formed HTML. Try a google search on mime sniffing, you should be able to find various ways to trick the browser.
I don't know how possible it is to alter data within an AMF response stream, but you might want to ensure that your endpoints cannot be manipulated through communication with the browser and/or JavaScript. Check out this article under the Malicious data injection section.
I have a Flex client that loads data from a server to display a chart. This data may change, so the client regularly repeats the request. Since the result may require some work to retrieve, I'm going to have the server detect if the result has changed, and issue a 304 status if it hasn't.
I haven't seen any headers in the Flash Player's requests which would indicate that it's already handling conditional GETs. Also, the HTTPService API doesn't seem to provide anything, either. Does that mean, Flash can't do this, or how can I implement this myself?
With regards to cookies, which aren't supported in Flash, I have heard the suggestion to build my own HTTP client on top of the Socket class. This might solve this issue, too, but frankly, I'm really not keen on doing that.
As an alternative, I could just cache the result page and send it again, but so far, the API tries to utilize semantics that are already built into HTTP, and I'd like to keep it that way.
In my experience Flash has dealt properly with HTTP 304 responses, though I haven't tried to change application behavior based on whether content was new or cached.
You may be able to detect the 304 responses and change your behavior if you use URLLoader instead of HTTPService and listen for the httpStatus event.
Not sure how your cookie question is related. Take a look at CookieUtil for accessing cookies from Flash through Javascript.
Take a look at another SO post:
Is it feasible to create a REST client with Flex?
I believe this will clarify some things for you.
I'm sure Wave doesn't poll the server every millisecond to find out if the other user has typed something... so how can I see what the other person is typing as they type? And without hogging the bandwidth.
Persistent HTTP, Comet
Keep your HTTP connection alive and send characters as they are typed
*Edit in 2014: also, take a look at WebSocket and HTTP/1.1 Upgrade header. Browsers started implementing this around 2010, so I'm adding this to original answer.
They probably use Web Sockets, aka server-sent events: http://www.w3.org/TR/websockets The underlying protocol can be found (as a draft) at the IETF.
Update: it doesn't seem WebSockets has any implementation yet; and a video from Google I/O (go to 11:00) talks about a long lived HTTP GET request.
Server Push in GWT
Server push is the Wait, Respond, Close, Re-Open paradigm:
Wait: When the GWT code makes a call
to your server for some data that you
don't have yet, freeze (wait)
Respond: Once the requested data is
available, respond with it
Close: Then, close the connection.
Re-Open: Once your GWT code receives the response, immediately open up a new connection to query for the next event.
See Video Google Wave: Powered by GWT around at minute 55 (near the end)
Q: How you implement the persistent Connections, the long living http connections
A: Future Plan: HTML5 Web Sockets. Longer term. That's what we use at the moment.
Q: Is there a platform or library for this we can download and play with?
A: Not sure. Don't think so
P.S.: That's what he said. To me it did not make much sense ("future plans" vs "using at the moment"). Any native english speaker might want to verify if I transcribed it correctly?
Pure speculation but could it be using the Server Side DOM events from the HTML 5 spec?
the entire reason for WebSockets is to have the browser keep a bi-directional socket open to a server so that real time communications can be used. When someone types on the other end, in a wave client, it triggers an event that is sent to the server and the server in turn looks to see who should also receive the event and pass them the event, in this case the typed letter.
WebSocket and Comet are different.
Granville
Probably comet for now websocket in the future. Because it works in Firefox 3.5 and from what I've read the websocket is only available in the nightly builds of FF... I could be wrong though... as it appears to not work in IE at all.
I spent some time reverse-engineering the Google Wave client code (shameless plug for http://antimatter15.com/misc/read/ which is a read-only public client for google wave for all public waves without need of robots or gadgets which was a lot more useful a month ago when Google didn't launch the upgrades).
Anyway, Google uses the GWT framework with certain aspects of the Google Closure library (which is actually open source and documented) and they use the goog.net.BrowserChannel library, which from the comments is also used for chat functionality within gmail.
http://closure-library.googlecode.com/svn/docs/closure_goog_net_browserchannel.js.html
I would assume that they use ajax requests. Do an XMLHttpRequest, which is asynchronous, and when the server has something to send your browser the javascript callback that was registered gets the data and does whatever with it. So basically the browser requests the next event, handles it, repeats indefinitely.
Is HTTP partial GET a reliable mechanism? If it is, how come it seems like modern browsers still start from the beginning instead of resuming the download?
In my experience this feature is not ubiquitous across all web servers. Probably because it is not a widely used by web clients. Sort of like HTTP HEAD requests which may or may not be implemented. As always, YMMV depending on the clients and servers involved.
The download resumption mechanism is based on HTTP range request headers that specify what part of the content you want (see here). I have not messed with this much in the last few years, so you may be better served doing a little more Google research. Here is a link to a blog posting that talks about some the latest developments regarding this feature.
Whenever I download big files with wget, I might interrupt them and resume with -c. I don't remember ever getting a corrupted file. Safari allows you to resume (instead of restart) a stopped download, works fine there too.
Yes, when done properly (If-Match etag...), it is reliable.