I setup a simple nodejs http-server that stream-processes an html request. As the html is streamed, it extracts any inline content into a seperate element and calculates its hashes.
In the final step, a trailer-header is send containing the original csp + the new hashes. (This is the only csp send, it is combined in the server!)
However, the browser (all: Chrome, Firefox, Edge) does not honour the csp!
(According to canIuse all of these browsers support trailer-headers though.)
The above in pseudo-code (node-js like):
const server = http.createServer((reqest, response) => {
response.setHeader('Transfer-Encoding', 'chunked');
response.setHeader('content-type', 'text/html');
response.setHeader('Trailer', 'content-security-policy');
const stream = getHTMLSAXStream();
stream.on('data', function(element) {
// extract inlines and save, e.g. styles.push(element.style);
// then remove the attr: e.g. element.style = undefined;
// then on the stripped html: e.g. response.write(element.toHTML())
});
stream.on('end', function() {
const stylefile= `${styles.join("\n")}`;
const url = getUrlForString(stylefile); // make this file available on a temporary url
response.write(`<link rel="stylesheet" href="${url}">`)
response.addTrailers({ 'content-security-policy': mergeWithDefaultCSP("style-src: sha256-${sha256(stylefile)}") });
response.end(); // send response
})
As per MDN docs on trailers some headers are disallowed, however couldn't find a reason why the content-security-policy shouldn't be allowed.
More specifically, as per trailer header and csp spec:
A sender MUST NOT generate a trailer that contains a field necessary
for message framing (e.g., Transfer-Encoding and Content-Length),
routing (e.g., Host), request modifiers (e.g., controls and
conditionals in Section 5 of [RFC7231]), authentication (e.g., see
[RFC7235] and [RFC6265]), response control data (e.g., see Section 7.1
of [RFC7231]), or determining how to process the payload (e.g.,
Content-Encoding, Content-Type, Content-Range, and Trailer).
The CSP is not used for message framing, it is not used for routing, it is not used as a request modifier, not used for authentication and isn't used for processing the payload (only used after processing the payload, aka the html) - in short, I don't see a reason it shouldn't work!
Does anyone know more? Have I missed anything?
To get around this, currently I'm using the following workaround (I'd like to get rid of):
don't use hashes, whitelist by domain (e.g. all scripts are coming from the same domain)
use nounces instead of hashes (won't play well with cdns though)
More on the background, why am I doing this at all:
I have a cms that allows using raw html (incl. inline-styles and script tags) which I frequently use (else I'd need to deploy again, etc etc). On the other hand I'd like a good working CSP (e.g. when user-generated comments are loaded onto the page from an api with javascript (not in the backend, that would defeat the purpose!), just in case. Therefore I'd like to allow only my own inline-style and script tags, but no-others. The above addresses this adequatly.
No matter what the spec says, it’s up to browsers to support this and last I header browser support for trailing headers is very limited: Do any browsers support trailers sent in chunked encoding responses?
Additionally I don’t think it makes sense for CSP for two reasons:
HTML is often streamed (as you state you are doing) and the browser will render as the HTML comes in. To then retrospectively apply the CSP to already rendered content would be pointless - the damage has been done.
Multiple CSPs are additive and not replacing. I.e. it the most restrictive CSP that matters. So if you’ve a basic CSP and then want to add a nonce that is not possible AFAIK.
Related
I am creating a server using Go that allows the client to upload a file and then use a server function to parse the file. Currently, I am using two separate requests:
1) First request sends the file the user has uploaded
2) Second request sends the parameters to the server that the server needs to parse the file.
However, I have realised that due to the nature of the program, there can be concurrency problem if multiple users try to use the server at the same time. My solution to that was using mutex locks. However, I am receiving the file, sending a response, and then receiving the parameters and it seems that Go cannot send a response back when the mutex is locked. I am thinking about solving this problem by sending both the file and the parameters in one single HTTP request. Is there a way to do that? Thanks
Sample code (only relevant parts):
Code to send file from client:
handleUpload() {
const data = new FormData()
for(var x = 0; x < this.state.selectedFile.length; x++) {
data.append('myFile', this.state.selectedFile[x])
}
var self = this;
let url = *the appropriate url*
axios.post(url, data, {})
.then(res => {
//other logic
self.handleParser();
})
}
Code for handleParser():
handleNessusParser(){
let parserParameter = {
SourcePath : location,
ProjectName : this.state.projectName
}
// fetch the the response from the server
let self = this;
let url = *url*
fetch(url, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(parserParameter),
}).then( (response) => {
if(response.status === 200) {
//success logic
}
}).catch (function (error) {
console.log("error: ", error);
});
}
The question is not really about Go or reactjs or any particular software library.
To solve your problem you'd first need to understand how HTTP POST works,
hence I invite you to first read this intro on MDN.
In short:
There are multiple ways to encode the data sent in a POST request.
The way the receiver should deal with this data depends on how it's encoded by the sender.
The sender has to communicate the encoding with its request — usually via the Content-Type header field.
I won't go into the details of possible encodings — the referenced introductory material covers them, and you should do your own research on them, but to maybe recap what's written there, here is some perspective.
Back in the 80s and 90s the web was "static" and the dreaded era of JavaScript-heavy "web apps" did not yet come. "Static" means you could not run any code in the client's browser, and had to encode any communication with the server in terms of plain HTML.
An HTML document could have two ways to make the client rendering it to send something back to the server: a) embed an URL which would include query parameters; this would make the client to perform a GET request with these parameters sent to the server; b) embed an HTML "form" which, when "submitted", would result in performing a rather more complex POST request with the data taken from the filled in form.
The latter approach was the way to leverage the browser's ability to perform reasonably complex data processing — such as slurpling a file selected by the user in a specific form's control, encoding it appropriately and sending it to the server along with the other form's data.
There were two ways to encode the form's data, and they are both covered by the linked article, please read about them.
The crucial thing to understand about this "static web with forms" approach is that it worked like this: the server sends an HTML document containing a web form, the browser renders the document, the user fills the form in and clicks the "submit" button rendered by the browser; the browser collects the data from the form's controls, for entries of type "file" it reads and encodes the contents of those files and finally performs an HTTP POST request with this stuff encoded to the URL specified by the form. The server would typically respond with another HTML document and so on.
OK, so here came "web 2.0", and an "XHR" (XMLHttpRequest) was invented. It has "XML" in its name because that was the time when XML was perceived by some as a holy grail which would solve any computing problem (which it, of course, failed to do). That thing was invended to be able to send almost arbitrary data payloads; XML and JSON encoding were supported at least.
The crucial thing to understand is that this way to communicate with the server is completely parallel to the original one, and the only thing they share is that they both use HTTP POST requests.
By now you should possibly see the whole picture: contemporary JS libs allow you to contruct and perform any sort of request: they allow you to create a "web form"-style request or to create a JS object, and serialise it to JSON, and send the result in an HTTP POST request.
As you can see, any approach allows you to pass structured data containing multiple distinct pieces of data to the server, and the way to handle this all is a matter of agreement between the server and the client, that is, the API convention, if you want.
The difference between various approaches is that the web-form-style approach would take care of encoding the contents of the file for you, while if you opt to send your file in a JSON object, you'll need to encode it yourself — say, using base64 encoding.
Combined approaches are possible, too.
For instance, you can directly send binary data of a file as a POST request's body, and submit a set of parameters along with the request by encoding them as query-parameters of the URL. Again, it's up to the agreement between the client and the server about how the latter encodes the data to be sent and the former decodes them.
All-in-all, I'd recommend to take a pause and educate yourself on the stuff I have outlined above, and then have another stab at solving the problem, but this time — with reasonably complete understanding about how the stuff works under the hood, and how you intend to wield it.
I am trying to upload a file to Dropbox using XmlHttpRequest, in a QML application. I am getting the following error:
Error in call to API function "files/upload": Bad HTTP "Content-Type"
header: "application/octet-stream;charset=UTF-8". Expecting one of
"application/octet-stream", "text/plain; charset=dropbox-cors-hack".
I seems that no matter which value I set to the Content-Type request header, it always adds
;charset=UTF-8
to the end.
My test data is simply a string, of which I would like a text file be generated to the user's dropbox folder.
request.send("TestString abcdefg")
How to deal with this issue?
Edit, with regards to the duplicate suspicion:
This is a similar issue, however, this particular case, if it can be addressed at all, needs to be addressed within the capabilities of Qt, on the QML javascript side, which has a set of different capabilities than javascript programming for the web, as it is a custom js engine, as far as I understand.
I'm trying to access a REST service I don't control. First problem is that the service doesn't include a Access-Control-Allow-Origin header, which is a problem that, if I understand correctly, immediately limits me to JSONP.
Also, by default, this service sends XML rather than JSON, though it's capable of sending JSON. I think it should respond to my Accept header, the people responsible for the service say it looks at my Content-Type. That would mean I'd need to do a POST rather than a GET (though get makes more sense when I'm just getting some static data, right?).
Stubborn as I am, I'm trying my Accept header first. Since Angular only accepts JSON, I'd expect it to use the Accept: application/json header by default, but it doesn't, and it ignores my attempts to set it manually:
app.config(['$httpProvider', function($httpProvider){
console.log($httpProvider.defaults.headers.common);
delete $httpProvider.defaults.headers.common['X-Requested-With'];
$httpProvider.defaults.headers.post['Accept'] = 'application/json, text/javascript';
$httpProvider.defaults.headers.post['Content-Type'] = 'application/json; charset=utf-8';
$httpProvider.defaults.headers.post['Access-Control-Max-Age'] = '1728000';
$httpProvider.defaults.headers.common['Access-Control-Max-Age'] = '1728000';
$httpProvider.defaults.headers.common['Accept'] = 'application/json, text/javascript';
$httpProvider.defaults.headers.common['Content-Type'] = 'application/json; charset=utf-8';
$httpProvider.defaults.useXDomain = true;
}]);
I do this again in the actual resource:
return $resource('http://foo.com/getStuff', {}, {
fetch: {
method:'JSONP',
params: params,
headers: {
'Accept':'application/json, text/javascript',
'Content-Type':'application/json; charset=utf-8'
},
isArray:false,
callback: 'JSON_CALLBACK'
}
});
But still, the request headers contain Accept: */*.
My question is: WHY? Why does Angular ignore my headers? And how do I get it to use the proper headers anyway?
And also: is there a way to use JSONP in a POST?
Edit: Originally I used Angular 1.0.7, but I just tried it with 1.2.3 and got the same results. Headers are ignored, yet everybody claims that this is the way to do it.
I also tried doing it directly with $http, rather than with $resource, with the same results.
Edit 2: Here's a JSFiddle. It's anonymized and doesn't use my real server, but using Firebug/developer tools, you can verify that it sends Accept: */* on both calls, despite my many attempts to set application/json headers. And that is my real problem here. On my real server, I'm getting an XML result because of that, despite my real server's ability to send JSON.
(Whether the real server supports jsonp is less relevant at the moment. This dummy server clearly doesn't, but that's okay. I just care about the headers.)
Edit 3: I've tried both solutions suggested below:
$http.defaults.headers.common['Accept'] = 'application/json, text/javascript';
$http.defaults.transformRequest.push(function (data, headersGetter) {
headersGetter().Accept = "application/json, text/javascript";
return data;
});
I've tried both statements separately. In the controller, and then in the service just before the http call itself. Still doesn't work.
Can someone give me a JsFiddle where this is shown to work?
Edit 4: I notice that when I use GET rather than JSONP, the Accept header is correct. But then the response is rejected because it doesn't have the correct header.
What kind of headers should a JSONP call have? Because there's a lot more headers in the JSONP call, but nothing that identifies it as JSONP. Does the server have to have explicit JSONP support for this to work? I suddenly realize I don't know nearly enough about jsonp.
I think your answer is here. According to the wiki, A JSONP call is executed through injection of a <script> tag to load the script from the host server, which responds by calling your callback, passing the data. A <script> tag generates a regular browser request (not an XmlHttpRequest), and the browser will send its own Accept header (it also sends its own User-Agent header, for example).
I would hope there is an easier client-side way to do this, but I think the only way may be the one suggested in the referenced post:
So, if you want to be able to set request headers for cross domain calls
you will have to setup a server side script on your domain that will
delegate the call to the remote domain (and set the respective
headers) and then send the AJAX request to your script.
EDIT: here is a (rejected) jQuery bug report about this same problem.
Some more background info:
In angular, callbacks are managed automagically, so if your say this:
$http({
method: "JSONP",
url: "http://headers.jsontest.com?callback=JSON_CALLBACK",
}).success(function(data) {
console.log('Return value:');
console.log(data);
}).error(function(data) {
console.log('Error!');
console.log(data);
})
a <script> tag will be created that looks more or less like this:
<script type="application/javascript"
src="http://headers.jsontest.com/?callback=angular.callbacks._1">
</script>
The content of the response to http://headers.jsontest.com/?callback=angular.callbacks._1 will be:
angular.callbacks._1({key1: "value1", key2: "value2"});
angular.callbacks._1 will contain your success function, and it will be called with the data.
While what you have is supposed to work according to the docs, my experience has been a bit different. To get around this issue, we did the following:
Create a "base controller" that gets added to the page either on the body or html tag.
In that controller, make the assignment using $http instead of $httpProvider. Because your base controller loads when the initial page loads, it is there for all other controllers and services that will run in your app.
I don't know why this works and the proscribed method does not, and I'd love to see an answer to your question that is better than this work-around, but at least this can get you moving forward with development again.
The following works for me - however, I do that during "runtime" with $http and I am not using $httpProvider during bootstrapping.
function SomeCtrl($http) {
$http.defaults.transformRequest.push(function (data, headersGetter) {
headersGetter().Accept = "application/json, text/javascript";
return data;
});
}
Edit
Here is a working jsFiddle version. Check the request which is done with Developer Tools/Firebug and see that "application/json, text/javascript" is requested.
"Accept" is not listed in the forbidden headers in the documentation here:
http://livedocs.adobe.com/flash/9.0/ActionScriptLangRefV3/flash/net/URLRequestHeader.html
but cant seem to set it in a GET request. Help!
You are allowed to specify the "accept" header, but only on a POST request with one or more variables.
The documentation comments mention this:
For browser-based Flash/AS3 applications, the only way to successfully set
or modify request headers on a URLRequest object is to set its method to
POST as well as send at least one variable of data along with the request
(i.e. with URLVariables). Otherwise the headers will silently remain
unchanged.
for some reason, IE6/7 is caching the ajax call that returns a json result set back.
My page makes the call, and returns a json result which I then inject into the page.
How can I force IE6/7 to make this call and not use a cached return value?
You might want to add
Cache-Control: no-cache
to your HTML response headers when you're serving the JSON to tell the browser to not to cache the response.
In ASP.NET (or ASP.NET MVC) you can do it like this:
Response.Headers.Add("Cache-Control", "no-cache");
you can change your settings in ie, but the problem most likely lies on your server. You can't go out and change all your users' browser settings. But if you want to at least check it on your browser, go to Internet Options->General (Tab)->Browsing History(section)->Settings (button)->"Every time I visit the webpage"
Make sure you set it back, though, at some point.
To fix it on the server, have a look at http://www.mnot.net/cache_docs/
Using curl (w/ cygwin) for debugging is your great way to figure out what's actually being sent across the wire.
If cache-control doesn't work for you (see DrJokepu's answer), according to the spec the content from any URL with a query string should be non-cacheable, so you might append a pointless query parameter to your request URL. The value doesn't matter, but if you really want to be thorough you can append the epoch value, e.g.:
var url = "myrealurl?x=" + (new Date()).getTime();
But this is a hack; really this should be solved with proper caching headers at the server end.
In the controller action that returns a JsonResult, you need to specify in your headers to avoid caching:
ControllerContext.HttpContext.Response.AddHeader("Cache-Control", "no-cache");