I have written downloading a file in a simple manner:
#ResourceMapping(value = "content")
public void download(ResourceRequest request, ResourceResponse response) {
//...
SerializableInputStream serializableInputStream = someService.getSerializableInputStream(id_of_some_file);
response.addProperty(HttpHeaders.CACHE_CONTROL, "max-age=3600, must-revalidate");
response.setContentType(contentType);
response.addProperty(HttpHeaders.CONTENT_TYPE, contentType);
response.addProperty(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename*=UTF-8''"
+ URLEncoder.encode(fileName, "UTF-8"));
OutputStream outputStream = response.getPortletOutputStream();
byte[] parcel = new byte[4096];
while (serializableInputStream.read(parcel) > 0)
outputStream.write(parcel);
outputStream.flush();
serializableInputStream.close();
outputStream.close();
//...
}
The SerializableInputStream is described here - JavaDocs. It allows an InputStream to be serialized and, for instance, passed over remoting.
I read from input and write it to the output, not all bytes at once. But unfortunately the portlet isn't "streaming" the contents - the file (e.g. an image) is sent to the browser only after reading the entire input stream - this is how it looks like. I see the file being read from the database (from live logs), but I don't see any "growing" image on the screen.
What am I doing wrong? Is it possible to really stream a file in Liferay 6.0.6 and Spring Portlet MVC?
Where are you doing this? I fear that you're doing this instead of rendering your portlet's HTML (e.g. render phase). Typically the portlet content is embedded in an HTML page, thus you need the resource phase, which (roughly) behaves like a servlet.
Also, the code you give does not match the actual question you ask: You use a comment //read from input stream (file), write file to os and ask what to do differently in order to not have the full content in memory.
As the comment does not have anything in memory and you could loop through reading from the input file while writing to the output stream: What's the underlying question? Do you have problems with implementing download-streaming in a portal environment or difficulties (i.e. using too much memory) reading from a file while writing to a stream?
Edit: Thanks for clarifying. Have you tried to flush the stream earlier? You can do that whenever you want - e.g. every loop (though that might be a bit too much). Also, keep in mind that the browser as well as the file itself must handle it in a way that you expect: If an image is not encoded "incrementally" a browser might not show it that way.
Have you tried this with huge files as well? It might be that the automatic flushing is just not triggered because your files are too small for it to be triggered...
Also, I think that filename*=UTF-8'' looks strange. Might be valid encoding, but I've never seen this
Related
I use the Apache commons fileupload 1.4 library in my java project.
I have a html part with a classic form with a file input and some hidden fields.
I have a problem with uploading files of around >500ko only with Firefox >= 52
It works well with files of 10mo in Chrome or Internet Explorer.
But with Firefox, I have a timeout after waiting several minutes after submitting the form.
After some debugging, I see that the code responsible of the timeout is :
List<FileItem> items = (new ServletFileUpload(new DiskFileItemFactory())).parseRequest(request);
The part with cause wait is "parseRequest".
I try to debug the content of request with debugger in IntelliJ, but there is no way to copy entire content value of this request object in raw format.
It's working in these cases :
- Firefox : version <= 52 or file size < 500ko (around, it's not really precise)
- Internet Explorer
- Chrome
There is no file size limit, it seems that depends on the request size, because the parsing request part is taking too much time...
I get the HTTP request with a Firefox extension in two cases.
One generating uploading a file of 3mo which doesn't works (the request file is huge, 3x the size of the uploaded file) :
https://code.empreintesduweb.com/13561.html
One generated uploading a file of 200ko which works (the request file is small) :
https://code.empreintesduweb.com/13560.html
In fact, the main difference is that in Chrome or IE, I don't have the raw content of the uploaded file in the request headers :
The part with :
obj
stream
....
endstream
endobj
Only appear with Firefox...
You can try setting the maximum file size, maybe the file size exceeds the maximum threshold .According to the documentation :
Uploaded items should be retained in memory as long as they are reasonably small.
Larger items should be written to a temporary file on disk.
Very large upload requests should not be permitted.
The built-in defaults for the maximum size of an item to be retained in memory, the maximum permitted size of an upload
request, and the location of temporary files are acceptable.
Try the following :
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
try {
// Set factory constraints
DiskFileItemFactory factory = new DiskFileItemFactory();
factory.setSizeThreshold(yourMaxMemorySize);
ServletContext servletContext = this.getServletConfig().getServletContext();
File repository = (File) servletContext.getAttribute("javax.servlet.context.tempdir");
factory.setRepository(repository);
List<FileItem> items = new ServletFileUpload(factory).parseRequest(request);
for (FileItem item : items) {
if (item.isFormField()) {
// Process regular form field (input type="text|radio|checkbox|etc", select, etc).
String fieldName = item.getFieldName();
String fieldValue = item.getString();
// ... (do your job here)
} else {
// Process form file field (input type="file").
String fieldName = item.getFieldName();
String fileName = FilenameUtils.getName(item.getName());
InputStream fileContent = item.getInputStream();
// ... (do your job here)
}
}
} catch (FileUploadException e) {
throw new ServletException("Cannot parse multipart request.", e);
}
// ...
}
Here, we are providing a temp location for the file since the file is large.
A few things that are worth to try here:
Explicit the encoding: https://stackoverflow.com/a/10488411/4279120
Decompose your call and add iteration and try catch, ex. : https://www.programcreek.com/java-api-examples/?api=org.apache.commons.fileupload.FileItemIterator
Take a look at the MultipartConfig, it seems to provide such attributes as maxFileSize and maxRequestSize (see: https://www.codejava.net/java-ee/servlet/java-file-upload-example-with-servlet-30-api#maxFileSize%28%29)
Manually define the header of your Request if you can. It seems that adding "X-File-Name" and "X-File-Size", can also help, but this is a little old: AJAX File Upload with XMLHttpRequest
We may also help you better if you provide some more informations, like the versions of apache / java / servlet, and a few more code (especially the definition of request)
Some ressources that could be helpful:
XMLHttpRequest
Sending_files_using_a_FormData_object
How to set a header for a HTTP GET request, and trigger file download?
try this to set session timeout using setMaxInactiveInterval method
request.getSession().setMaxInactiveInterval(1200);
parameter Specifies the time, in seconds, between client requests before
the servlet container will invalidate this session. An interval value
of zero or less indicates that thesession should never timeout.
Thanks for all your answer.
Finally, I successfully resolve this issue, but in fact... not really.
I notice that there was some specific things in my form.
I had two inputs, one standard file input, and another which receive the file content encoded in base64 by some weird js before any upload.
So I was having one time the raw content of the file, and also the file in base64. Why ?! I don’t know.
But I delete all this, I create a new simple and clean form with a standard input file.
I use the stream API from ServletFileUpload, and it works, and takes only few seconds for big files.
So I don’t understand everything (why the problem was only on some browser for example), but I find a solution ;)
Thank you !
everyone!
I have a trouble. I tried to save excel file in jsf web application.
I generated file by my utils and trying to get "save" window, but I failed.
Here is my code:
<div>
<h:commandButton value="Apply" actionListener="#{hornPhonesBean.generateReport}"/>
</div>
and:
public void generateReport(ActionEvent event) {
System.out.println("GENERATE REPORT FROM = " + this.dateFrom + "; TO = " + this.dateTo);
try {
XSSFWorkbook workbook = (XSSFWorkbook) HornReportGenerator.getWorkbook(null, null);
String fileName = "1.xlsx";
FacesContext fc = FacesContext.getCurrentInstance();
ExternalContext ec = fc.getExternalContext();
// Some JSF component library or some Filter might have set some headers in the buffer beforehand. We want to get rid of them, else it may collide.
ec.responseReset();
// Check http://www.w3schools.com/media/media_mimeref.asp for all types. Use if necessary ExternalContext#getMimeType() for auto-detection based on filename.
ec.setResponseContentType("application/vnd.ms-excel");
// Set it with the file size. This header is optional. It will work if it's omitted, but the download progress will be unknown.
//ec.setResponseContentLength(contentLength);
// The Save As popup magic is done here. You can give it any file name you want, this only won't work in MSIE, it will use current request URL as file name instead.
ec.setResponseHeader("Content-Disposition", "attachment; filename=\"" + fileName + "\"");
OutputStream output = ec.getResponseOutputStream();
workbook.write(output);
output.flush();
output.close();
fc.responseComplete(); // Important! Otherwise JSF will attempt to render the response which obviously will fail since it's already written with a file and closed.
System.out.println("END");
} catch (Exception e) {
e.printStackTrace();
}
}
I read suggestions here and from another forums - everyone says I shouldnt use , but I didn't use it at all.
Then I thought that the problem could be in the
<ice:form>,
where I kept the
<h:commandButton>,
and I changed to
<h:form>,
but it didn't help.
Maybe the problem in the request - it has header Faces-Request partial/ajax. But I am not sure.
Please give me some ideas - I already spent 4 hours for this crazy jsf download issue)
Maybe the problem in the request - it has header Faces-Request partial/ajax. But I am not sure.
This suggests that the request is an ajax request. You can't download files by ajax. Ajax requests are processed by JavaScript which has for obvious security reasons no facilities to programmatically pop a Save As dialogue nor to access/manipulate client's disk file system.
Your code snippet does however not show that you're using ajax. Perhaps you oversimplified it too much or you're using ICEfaces which silently auto-enables ajax on all standard JSF command components.
In any case, you need to make sure that it's not sending an ajax request.
See also:
How to provide a file download from a JSF backing bean?
ICEfaces libary in classpath prevents Save As dialog from popping up on file download
My initial purpose was to verify the HTTP chunked transfer. But accidentally found this inconsistency.
The API is designed to return a file to client. I use HEAD and GET methods against it. Different headers are returned.
For GET, I get these headers: (This is what I expected.)
For HEAD, I get these headers:
According to this thread, HEAD and GET SHOULD return identical headers but not necessarily.
My question is:
If Transfer-Encoding: chunked is used because the file is dynamically fed to the client and Tomcat server cannot know its size beforehand, how could Tomcat know the Content-Length when HEAD method is used? Does Tomcat just dry-run the handler and count all the file bytes? Why doesn't it simply return the same Transfer-Encoding: chunked header?
Below is my RESTful API implemented with Spring Web MVC:
#RestController
public class ChunkedTransferAPI {
#Autowired
ServletContext servletContext;
#RequestMapping(value = "bootfile.efi", method = { RequestMethod.GET, RequestMethod.HEAD })
public void doHttpBoot(HttpServletResponse response) {
String filename = "/bootfile.efi";
try {
ServletOutputStream output = response.getOutputStream();
InputStream input = servletContext.getResourceAsStream(filename);
BufferedInputStream bufferedInput = new BufferedInputStream(input);
int datum = bufferedInput.read();
while (datum != -1) {
output.write(datum);
datum = bufferedInput.read();
}
output.flush();
output.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
ADD 1
In my code, I didn't explicitly add any headers, then it must be Tomcat that add the Content-Length and Transfer-Encoding headers as it sees fit.
So, what are the rules for Tomcat to decide which headers to send?
ADD 2
Maybe it's related to how Tomcat works. I hope someone can shed some light here. Otherwise, I will debug into the source of Tomcat 8 and share the result. But that may take a while.
Related:
HTTP HEAD and GET different result
Content-Length header with HEAD requests?
Does Tomcat just dry-run the handler and count all the file bytes?
Yes, the default implementation of javax.servlet.http.HttpServlet.doHead() does that.
You can look at helper classes NoBodyResponse, NoBodyOutputStream in HttpServlet.java
The DefaultServlet class (the Tomcat servlet that is used to serve static files) is more wise. It is capable of sending the correct Content-Length value, as well as serving GET requests for a subset of the file (the Range header). You can forward your request to that servlet, with
ServletContext.getNamedDispatcher("default").forward(request, response);
Although it seems strange, it might make sense to send the size only in response to a HEAD request and chunked in response to a GET request, depending on the type of data that has to be returned by the server.
While your API seems to provide a static file, you also talk about dynamically created files or data, so I will be talking in general here (also for webservers in general).
First let's have a look at the different usages for GET and HEAD:
With GET the client is requesting the whole file or data (or a range of the data), and wants it as fast as possible. So there is no specific reason for the server to send the size of the data first, especially when it could start sending faster/sooner in chunked mode. So the fastest possible way is preferred here (the client will have the size after the download anyway).
With HEAD on the other hand, the client usually wants some specific information. This could just be a check on existance or 'last-changed', but it could also be used if the client wants a certain part of the data (with a range request, including a check to see if range requests are supported for that request), or just needs to know the size of the data up front for some reason.
Lest's look at some possible scenarios:
Static file:
HEAD: there's no reason to not include the size in the response-header because that information is available.
GET: most of the time the size will be inluded in the header and the data sent in one go, unless there are specific performance reasons to send it in chunks. On the other hand it seems you are expecting chunked transfer for you file, so this could make sense here.
Live logfile:
Ok, somewhat strange, but possible: downloading a file where the size could change while downloading.
HEAD: again, the client probably wants the size, and the server can easily provide the size of the file at that specific time in the header.
GET: since loglines could be added while downloading, the size is unknown up front. Only option is to send chunked.
Table with fixed-sized records:
Let's imagine a server needs to send back a table with fixed-length records coming from multiple sources/databases:
HEAD: size is probably wanted by the client. The server could quickly do a query for count in each database, and send the calculated size back to the client.
GET: instead of doing a query for count in each database first, the server better starts sending the resulting records from each database in chunks.
Dynamically generated zip-files:
Maybe not common, but an interesting example.
Imagine you want to provide dynamically generated zip-files to the user based on some parameters.
Let's first have a look at the structure of a zip-file:
There are two parts: first there's a block for each file: a small header followed by the compressed data for that file. Then there's a list of all the files inside the zip-file (including sizes/positions).
So the prepared blocks for each file could be pre-generated on disk (and the names/sizes stored in some data structure.
HEAD: the client probably wants to know the size here. The server can easily calculate the size of all the needed blocks + the size of the second part with the list of the files inside.
If the client wants to extract a single file, it could directly ask for the last part of the file (with a range-request) to have the list, and then with a second request ask for that single file. Although the size is not necessarily needed to get the last n bytes, it could be handy if for example if you wanted to store the different parts in a sparse file with the same size of the full zip-file.
GET: no need to do the calculations first (including generating the second part to know its size). It would be better and faster to just start sending each block in chunks.
Fully dynamically generated file:
In this case it wouldn't be very efficient to return the size to a HEAD request of course, since the whole file would need to be generated just to know its size.
I want to limit the allowed uploaded file types to images, pdfs, and docs. What is the recommended way to approach this?
I assume checking the file extension alone is not enough, since an attacker can change the file extension as he wishes.
I also thought about checking against MIME Type using PostedFile.ContentType.
I still don't know if this is adding any further functionality than checking against file extensions alone, and if an attacker have and ability to change this information easily.
This is basically for a course management system for students to upload assignments and teachers to download and view them.
Thanks.
I agree with validating the extension as show by pranay_stacker, and checking against PostedFile.ContentType will provide another layer of security. But, it still relies on a the Content-Type header set by the client and therefore susceptible to attack.
If you want to guarantee the file types then you need to upload the file and check the first 2 bytes. Something along the lines of (untested)
string fileclass = "";
using(System.IO.BinaryReader r = new System.IO.BinaryReader(fileUpload1.PostedFile.InputStream))
{
byte buffer = r.ReadByte();
fileclass = buffer.ToString();
buffer = r.ReadByte();
fileclass += buffer.ToString();
r.Close();
}
if(fileclass!="3780")//.pdf 208207=.doc 7173=.gif 255216=.jpg 6677=.bmp 13780=.png
{
errorLiteral.Text = "<p>Error - The upload file must be in PDF format.</p>"
return;
}
This is very rough and not robust, hopefully someone can expand on this.
To be 99% sure, you'll have to check magic numbers of a uploaded files, just like UNIX file utility does.
What's the best way to stream files using ASP.NET?
There appear to be various methods for this, and I'm currently using the Response.TransmitFile() method inside an http handler, which sends the file to the browser directly. This is used for various things, including sending FLV's from outside the webroot to an embedded Flash video player.
However, this doesn't seem like a reliable method. In particular, there's a strange problem with Internet Explorer (7), where the browser just hangs after a video or two are viewed. Clicking on any links, etc have no effect, and the only way to get things working again on the site is to close down the browser and re-open it.
This also occurs in other browsers, but much less frequently. Based on some basic testing, I suspect this is something to do with the way files are being streamed... perhaps the connection isn't being closed properly, or something along those lines.
After trying a few different things, I've found that the following method works for me:
Response.WriteFile(path);
Response.Flush();
Response.Close();
Response.End();
This gets around the problem mentioned above, and viewing videos no longer causes Internet Explorer to hang.
However, my understanding is that Response.WriteFile() loads the file into memory first, and given that some files being streamed could potentially be quite large, this doesn't seem like an ideal solution.
I'm interested in hearing how other developers are streaming large files in ASP.NET, and in particular, streaming FLV video files.
I would take things outside of the "aspx" pipeline. In particular, I would write a ran handler (ashx, or mapped via config), that does the minimum work, and simply writes to the response in chunks. The handler would accept input from the query-string/form as normal, locate the object to stream, and stream the data (using a moderately sized local buffer in a loop). A simple (incomplete) example shown below:
public void ProcessRequest(HttpContext context) {
// read input etx
context.Response.Buffer = false;
context.Response.ContentType = "text/plain";
string path = #"c:\somefile.txt";
FileInfo file = new FileInfo(path);
int len = (int)file.Length, bytes;
context.Response.AppendHeader("content-length", len.ToString());
byte[] buffer = new byte[1024];
Stream outStream = context.Response.OutputStream;
using(Stream stream = File.OpenRead(path)) {
while (len > 0 && (bytes =
stream.Read(buffer, 0, buffer.Length)) > 0)
{
outStream.Write(buffer, 0, bytes);
len -= bytes;
}
}
}
Take a look at the following article Tracking and Resuming Large File Downloads in ASP.NET which will give you more in depth than just open a stream and chuck out all the bits.
The http protocol supports ranged byte requests and resumeable downloads, and many streaming clients (like video players or Adobe pdf) can and will try to chunk these up, saving bandwidth and giving your users a better experience.
Not trivial, but it's time well spent.
Try opening the file as a stream, then using Response.OutputStream.Write(). For example:
Edit: My bad, I forgot that Write takes a byte buffer. Fixed
byte [] buffer = new byte[1<<16] // 64kb
int bytesRead = 0;
using(var file = File.OpenRead(path))
{
while((bytesRead = file.Read(buffer, 0, buffer.Length)) != 0)
{
Response.OutputStream.Write(buffer, 0, bytesRead);
}
}
Response.Flush();
Response.Close();
Response.End();
Edit 2: Did you try this? It should work.
After trying lots of different combinations, including the code posted in the various answers, it seems like setting Response.Buffer = true before calling TransmitFile did the trick and the web application is now a lot more responsive in Internet Explorer.
In this particular case, the SWF extension is also mapped to ASP.NET, and we're using a custom handler in our web application to read the files from disk and then send them to the browser using Response.TransmitFile(). We've got a flash-based video player to play video files which are also SWF's, and I think having all of this activity go through the handler without buffering is what may have been causing strange things to happen in IE.