I want to return a TSV file from a web call in Hunchentoot (SBCL), but want the user to just save the raw result blatted to the page, rather than use a separate file and download link (which is hard because of local firewall complexities).
I can't figure out how to output the page without any headers at all, i.e., to make it just plain raw text. (I know that the browser would make a mess w/o headers in the DOM, but don't care; the goal is just to have the user save the page, not read it.)
I've tried various combinations of
(setf (hunchentoot:content-type*) "text/plain")
and
(cl-who:with-html-output-to-string
(*standard-output* nil :prologue nil)
and setting the content-type* inside, outside, and around the with... but I always get header junk.
Writing a string directly
I tried defining a handler as follows:
(define-easy-handler (text :uri "/text") ()
(setf (content-type*) "text/csv")
"a,b,c")
When I visit the page locally, the browser automatically downloads a text file without even displaying (this is probably a setting we can change in Chrome, I don't know).
When I enable the browser developer mode, here are the response headers I receive as part of the HTTP protocol:
HTTP/1.1 200 OK
Server: ...
Date: ...
Content-Type: text/csv; charset=utf-8
Content-Length: 5
Connection: keep-alive
But the file itself is just the string a,b,c.
If I change the content-type to "text/plain", then the browser successfully displays the text, and nothing else (the HTTP headers are the same).
Remarks
You don't need to use the cl-who macros if you do not intend to build an HTML document, in fact its better not to. In any case, you can supply your own REPLY-CLASS when initializing the acceptor (see https://edicl.github.io/hunchentoot/#replies) and have a very low-level control about what you emit as a reply, headers included. But I don't think this is necessary in your case. I don't clearly understand where your problem comes from, but sending back a plain text is something the framework is supposed to be able to do out of the box. Please add more details if you can.
Is the correct answer not to use the Content-Disposition header?
Related
What is the process by which sinatra's send_file decides what content-type to use?
For example, it seems that it works by the extension of the file passed to send_file, so if it is send_file blah.txt . then when I http to the route, I will get/ the response header will be, content-type: text/plain, so any html in the txt file will be interpreted by the web browser as plain text. Whereas if the file is blah.html then the server will respond with content-type: text/html.(and any html in the file is rendered as such)
And of course the route name is irrelevant so you could go to http://127.0.0.1:4567/zzz.html and it could lead to send_file a.txt and a.txt may contain html tags but since it's a .txt file send_file will cause sinatra to respond with content-type: text/plain and the browser won't render any html sent and will show it as plain text. I may be wrong but that seems to be what my quick tests indicate. Where I tried different routes, different filename extensions(.txt, and .html), sometimes files with html in them sometimes not, seeing whether the browser renders the html or not, and seeing what the content-type header was, with wget -d.
So then my question related to that is, is there a list that sinatra's send_file function uses, that relates file extension to content-type? I would like to see that list. And if not, then what is the process it is using.
Note- I understand there is a way to pass in a content-type Sinatra: How to respond with an image with headers "content-type" => "image/jpeg"
but i'm asking how/ by what method, send_file determines content-type when no content-type is passed in.
This is the send_file method in the Sinatra framework (currently v2.0.5), notice it hands off finding out the content type straight away if none has been set:
if opts[:type] or not response['Content-Type']
content_type opts[:type] || File.extname(path), :default => 'application/octet-stream'
end
The content_type method will either return immediately or hand off to mime_type, which is a delegate of Rack's mime_type method (currently v2.0.7). This uses a well known list of extensions to check against.
def mime_type(ext, fallback='application/octet-stream')
MIME_TYPES.fetch(ext.to_s.downcase, fallback)
end
The list begins on line 49:
MIME_TYPES = {
".123" => "application/vnd.lotus-1-2-3",
".3dml" => "text/vnd.in3d.3dml",
".3g2" => "video/3gpp2",
".3gp" => "video/3gpp",
# <snip>
As you can see from the content_type snippet, the default it falls back on is application/octet-stream.
I am trying to upload a file to Dropbox using XmlHttpRequest, in a QML application. I am getting the following error:
Error in call to API function "files/upload": Bad HTTP "Content-Type"
header: "application/octet-stream;charset=UTF-8". Expecting one of
"application/octet-stream", "text/plain; charset=dropbox-cors-hack".
I seems that no matter which value I set to the Content-Type request header, it always adds
;charset=UTF-8
to the end.
My test data is simply a string, of which I would like a text file be generated to the user's dropbox folder.
request.send("TestString abcdefg")
How to deal with this issue?
Edit, with regards to the duplicate suspicion:
This is a similar issue, however, this particular case, if it can be addressed at all, needs to be addressed within the capabilities of Qt, on the QML javascript side, which has a set of different capabilities than javascript programming for the web, as it is a custom js engine, as far as I understand.
i have written a flex component to allow the user to select an image from the local filesystem and then POST it to a CQ5 DAM.
there are 2 CQ5 instances with which i'm working. the image posts fine to one instance, but not the other. specifically, in the 2nd instance, the renditions are not getting created when using the component.
one difference i've noted is that the working images, when i look at them in crxde, have a jcr:primaryType of dam:Asset. the non-working ones are nt:File.
from Flex, I use URLLoader to POST with a multipart form. the request (in part) looks like this:
POST /content/dam/test/foo.createasset.html HTTP/1.1
Host: xxxxxxxx:4502
Content-type: multipart/form-data; boundary=doudrbitutcfasnbhlpogirdctuxem
--doudrbitutcfasnbhlpogirdctuxem
Content-Disposition: form-data; name="file"
home.png
--doudrbitutcfasnbhlpogirdctuxem
Content-Disposition: form-data; name="Filename"
home.png
--doudrbitutcfasnbhlpogirdctuxem
Content-Disposition: form-data; name="home.png"; filename="home.png"
Content-Type: application/octet-stream
*** image data ***
--doudrbitutcfasnbhlpogirdctuxem
Content-Disposition: form-data; name="Upload"
Submit Query
--doudrbitutcfasnbhlpogirdctuxem--
that does save the image at: /content/dam/test/foo/home.png
i've tried adding a variable to the form:
./jcr:contentType dam:Asset
but that didn't cause the contentType to change. instead, the file didn't show up in CQ5 at all.
i know next to nothing about CQ5. i've seen some (old) examples of code POSTing right to where they want the asset to go, instead of hitting foo.createAsset.html as i've done. i could not get the more-straightforward POST working, and instead used CQ5 DAM to upload and image and captured through Charles, then tried to replicate that.
the CQ5 version that works is 5.5.0.
the version that does not is 5.4.0.
i'm sure that there are other configuration differences as well. in addition, the client is unwilling to upgrade from 5.4.0.
am i on the right track? close?
edit to clarify server setup:
CQ 5.5.0 --> installed locally, this one is an author server. my component works when POSTing to this server. meaning, the uploaded image is marked as dam:Asset and the renditions are generated.
CQ 5.4.0 --> a dev instance used by many. this is an author and publish server. my component does not 100% work when POSTing to this server. however, if i use the DAM admin interface to upload an image, it does properly mark the image as dam:Asset and generate the renditions.
edit #2: WORKING
it turns out that the dev/5.4 instance handles file uploading differently. my multi-part POST code mostly worked, but instead of using createAsset.html, i'm uploading to /tmp/fileupload.
then i issue a 2nd POST, using application/x-www-form-urlencoded, to issue a move command.
for those wishing to do the same, the move code looks like this:
var service:HTTPService = new HTTPService();
var url:String = instanceUrl + "/tmp/fileupload";
service.url = url;
var headerData : Object = new Object();
headerData['Cache-Control'] = 'no-store';
headerData['Authorization'] = getAuthString();
service.headers = headerData;
service.contentType = "application/x-www-form-urlencoded; charset=UTF-8";
service.method = URLRequestMethod.POST;
var urlVar:URLVariables = new URLVariables();
var command:String = "/var/dam/" + destPath + "/" + filename + "#MoveFrom";
var arg:String = "/tmp/fileupload/" + filename;
urlVar[command] = arg;
urlVar["_charset_"] = "utf-8";
var token:AsyncToken = service.send(urlVar);
not knowing CQ5, i can only assume the dev server is set up to run some workflow steps when it receives the #MoveFrom; those are the steps that ensure the uploaded file is of type dam:Asset and that the desired renditions are created.
If uploading from the DAM admin page via a browser works on the 5.4.0 instance, I would suggest analysing the HTTP request that this makes, to reproduce the same request from your Flex client. There's probably a subtle difference between the 5.4.0 and 5.5.0 HTTP APIs that explains this.
as a followup, below are the broad steps i took to get this working.
my overall goal was to write a Flex component that, for a specified VO, allowed the user to upload an image from their local filesystem (i used FileReference for this) into the component, then upload that image to CQ5 and publish it. after it was published, i then read it back into the component to display it.
i won't put the full code solution here, as it's involved and belongs to my client. in addition to my component, i wrote a utility for cq5 DAM operations, and an http service with built-in retries (which ended up being necessary because even though cq will give me a 200 when i request a resource, subsequent operations on that resource may fail, because cq doesn't seem to think it's there). Note that in all retry instances, i have a max retry count. the default value is 10, and default retry interval is 250ms.
please understand i know very little of CQ; most of what i learned was reverse engineering through trying things in the tool and watching Charles. also understand the steps below may be very specific to the install of CQ5 i'm working with.
so here are my overall steps. unless indicated otherwise, all requests are on port 4502:
a destination directory is determined from data in the VO and a POST is issued to create it. this is done with Content-type=application/x-www-form-urlencoded. the url is the full path of the folder i want to create, with no trailing slash.
repeat a GET on the created directory until we get a 200. the url here does have a trailing slash.
the image is POSTed to a temp area, [instance]/tmp/fileupload, as multipart form data. To help with this, i used an MIT-licensed AS class called MultipartURLLoader (https://code.google.com/p/in-spirit/). I used Content-type=multipart/form-data; boundary=[boundary]. CQ seemed very picky about the contents of the form data. mine is set up like this:
file: [name of file]
Filename: [name of file]
[name of file]: [file data]
Upload: Submit Query
another POST is issued, with a move command, to move the image from the temp area to the directory created in step 1. the url is [instance]/tmp/fileupload, and Content-type=application/x-www-form-urlencoded; charset=UTF-8. The form data is set up like this:
/var/dam/[destination_path]/[filename]#MoveFrom: /tmp/fileupload/[filename]
charset: utf-8
repeat step 4 until we get a 200. when new destination folders are indeed created, the first POST to #MoveFrom usually results in a 500, saying the destination folder is not there. perhaps there's another way to ask CQ if the destination is ready? i don't know.
we now need to publish the file, but first we issue a series of GETs on it to ensure it's there, with this url: [instance]/content/dam/[destination]/[filename].assets.json. once it's there, CQ will respond with some JSON that we use next.
check to see if the file has already been published. it may be the case that the user has already uploaded an image with the same name to the same location. the JSON response has a results node, which i check to see if it's 1. if it is, then i look at "pages[0].replication" to see if it has a node called "action". if it does, i see if the value is "ACTIVATE". if it is, it's already published. in every other case, i try to publish it.
POST a command to activate (publish it). the url is [instance]/bin/replicate.json. Content-type=application/x-www-form-urlencoded; charset=UTF-8. The form looks like this:
path: /content/dam/[destination]/[filename]
cmd: Activate
charset: utf-8
for my purposes, i wanted to then retrieve the published image to re-display it in my component. i waited for the 200 from the publish, then tried my GET. the url i used here had no port number, and no trailing slash: [instance:80]/content/dam/[destination]/[filename]. The first call almost always gave me a 404, so i kept trying until i got the 200.
that's it. i hope this is helpful to someone.
note: just saw that "charset" is in italics in the form specification. note that i'm using (underscore)charset(underscore).
I would like to create a HTTP response, using multipart/mixed, but I'm not sure which browsers support it; and if it's as convenient as it sounds, from the client's point of view.
To be honest, I do not need specifically that content type. I just want to transmit more than one file in the same response; maybe there's another content-type more used.
I've tested it, with a home-made server and a simple response. Not sure if the response is well-formed because no browser understands it 100% OK. But here are the results:
Firefox 67.0.1 (64-bit): Renders only the last part, others are ignored.
IE 11.503: Saves all the content in a single file (including the boundaries), nothing is rendered.
Chrome May 2019: Saves all the content in a single file, nothing is rendered.
Safari 4: Saves all the content in a single file, nothing is rendered.
Opera 10.10: Something weird. Starts rendering the first part as plain/text, and then clears everything. The loading progress bar hangs on 31%.
Here's the complete response, if anyone finds any error, please tell me and I'll try again:
HTTP/1.1 200 OK
Date: Tue, 01 Dec 2009 23:27:30 GMT
Vary: Accept-Encoding,User-Agent
Content-Length: 681
Content-Type: Multipart/mixed; boundary="sample_boundary";
Multipart not supported :(
--sample_boundary
Content-Type: text/css; charset=utf-8
Content-Location: http://localhost:2080/file.css
body
{
background-color: yellow;
}
--sample_boundary
Content-Type: application/x-javascript; charset=utf-8
Content-Location: http://localhost:2080/file.js
alert("Hello from a javascript!!!");
--sample_boundary
Content-Type: text/html; charset=utf-8
Content-Base: http://localhost:2080/
<html>
<head>
<link rel="stylesheet" href="http://localhost:2080/file.css">
</head>
<body>
Hello from a html
<script type="text/javascript" src="http://localhost:2080/file.js"></script>
</body>
</html>
--sample_boundary--
In my experience, multipart responses work in Firefox but not in Internet Explorer. This was 2 years ago, using the browsers of the time.
I have had HTTP multipart responses working for a stream of JPEG images. For example, Axis IP cameras use for their motion JPEG stream for Firefox. For Internet explorer, Axis require the use of a plugin.
If Firefox-only support meets your requirements, then I recommend setting the content-length header in each part of the multi-part response. It might help to make the boundary string identical in the original HTTP header and the multi-part response (the '--' is missing in the HTTP header).
Two ideas:
Formatting: I think "multipart" should be in lower case, and I don't think a semicolon is expected at the end of the Content-type header (although it's doubtful that it will make a difference, it's possible that it might).
Have you tried replace mode? Just use: Content-type: multipart/x-mixed-replace -- everything else should stay the same.
Multi part it yourself
(A good option)
A multipart response can be made manually!
So one can write a no multipart response! Let's say in chunked mode! There it make sense!
So you are streaming the data!
Send all as blunt text!
Make your own separators! Between each part!
In the browser! Extract and parse the data! Split to get each part separately!
And parse each appart! Depending on what type of data it hold!
So if a part is json! You parse it as so!
Quick illustration! Let say we want to send a csv file! Or some other type of file! Along that we want to send too a json object!
And that by streaming it by chunk
Here a code that illustrate that in express:
const data = {
initCapital: fileState.mappingObj.initialCapital
};
res.write(JSON.stringify(data, undefined, 0));
res.write('>>>>'); // parts separator
fileState.readStream.pipe(res); // continue streaming the file (chunk by chunk)
And in the client
export function parseBackTestFetchedData(data: string) {
const [_data, csvData] = data.split('>>>>');
return {
data: JSON.parse(_data),
backTestStatesData: parseCsv(csvData)
};
}
That way! it doesn't matter who the client is!
for some reason, IE6/7 is caching the ajax call that returns a json result set back.
My page makes the call, and returns a json result which I then inject into the page.
How can I force IE6/7 to make this call and not use a cached return value?
You might want to add
Cache-Control: no-cache
to your HTML response headers when you're serving the JSON to tell the browser to not to cache the response.
In ASP.NET (or ASP.NET MVC) you can do it like this:
Response.Headers.Add("Cache-Control", "no-cache");
you can change your settings in ie, but the problem most likely lies on your server. You can't go out and change all your users' browser settings. But if you want to at least check it on your browser, go to Internet Options->General (Tab)->Browsing History(section)->Settings (button)->"Every time I visit the webpage"
Make sure you set it back, though, at some point.
To fix it on the server, have a look at http://www.mnot.net/cache_docs/
Using curl (w/ cygwin) for debugging is your great way to figure out what's actually being sent across the wire.
If cache-control doesn't work for you (see DrJokepu's answer), according to the spec the content from any URL with a query string should be non-cacheable, so you might append a pointless query parameter to your request URL. The value doesn't matter, but if you really want to be thorough you can append the epoch value, e.g.:
var url = "myrealurl?x=" + (new Date()).getTime();
But this is a hack; really this should be solved with proper caching headers at the server end.
In the controller action that returns a JsonResult, you need to specify in your headers to avoid caching:
ControllerContext.HttpContext.Response.AddHeader("Cache-Control", "no-cache");