I see numerous examples of using Apache Camel to send or receive data over a TCP connection, as long as the data is text. For example setting up an endpoint such as netty:tcp://localhost:5000?textline=true. The important thing to note here is the usage of the option textline=true. This works great when you are dealing with text, not so much if it's just bytes.
How do I do the same thing, except that instead of text, I'm sending blobs of binary data (it's not serialized Java objects)? From what I could find, it would seem that I need to specify a codec to use (instead of the default text codec when textline=true is used), but I don't know how to do that.
I can't find any good examples of how to do this either on the Camel site nor anywhere else.
UPDATE [Thu Jul 19 10:43:10 EDT 2018]:
I have managed to attain a certain degree of success. I created a simple DataFormat and plugged it into the stream:
public class ByteArrayFormat implements DataFormat {
#Override
public void marshal(Exchange exchange, Object object, OutputStream ostream) throws Exception {
byte[] bytes = exchange.getContext().getTypeConverter().mandatoryConvertTo(byte[].class, object);
ostream.write(bytes);
}
#Override
public Object unmarshal(Exchange exchange, InputStream istream) throws Exception {
byte[] bytes = exchange.getContext().getTypeConverter().mandatoryConvertTo(byte[].class, istream);
return bytes;
}
}
I am not 100% sure this works, as I may have been seeing only unmarshal in action. Any suggestions or corrections are welcome.
I then modified my route as follows:
public void configure() {
from("netty4:tcp://localhost:5000?allowDefaultCodec=false&sync=false").unmarshal(new ByteArrayFormat())
.process(processor).to("file://test");
}
The testing "processor" does nothing more than grab the input stream from the exchange and hexdump the data to the console. The "to" just places what it gets in a file so I can look at it there as well.
I then did a simple test where I piped the contents of a binary file (via nc) and watched what happened. As far as I can tell, I received the data of the file as bytes.
There is a major caveat though. The result was broken into 1024 byte blocks (which I assume is a limitation imposed by the network stack), so the original file of 3862 bytes came over as four "receives" of 1024 byte blocks (really, 3 x 1024 + 790). Any ideas or suggestions on how to "stitch" this back together would be helpful.
Related
I use the Apache commons fileupload 1.4 library in my java project.
I have a html part with a classic form with a file input and some hidden fields.
I have a problem with uploading files of around >500ko only with Firefox >= 52
It works well with files of 10mo in Chrome or Internet Explorer.
But with Firefox, I have a timeout after waiting several minutes after submitting the form.
After some debugging, I see that the code responsible of the timeout is :
List<FileItem> items = (new ServletFileUpload(new DiskFileItemFactory())).parseRequest(request);
The part with cause wait is "parseRequest".
I try to debug the content of request with debugger in IntelliJ, but there is no way to copy entire content value of this request object in raw format.
It's working in these cases :
- Firefox : version <= 52 or file size < 500ko (around, it's not really precise)
- Internet Explorer
- Chrome
There is no file size limit, it seems that depends on the request size, because the parsing request part is taking too much time...
I get the HTTP request with a Firefox extension in two cases.
One generating uploading a file of 3mo which doesn't works (the request file is huge, 3x the size of the uploaded file) :
https://code.empreintesduweb.com/13561.html
One generated uploading a file of 200ko which works (the request file is small) :
https://code.empreintesduweb.com/13560.html
In fact, the main difference is that in Chrome or IE, I don't have the raw content of the uploaded file in the request headers :
The part with :
obj
stream
....
endstream
endobj
Only appear with Firefox...
You can try setting the maximum file size, maybe the file size exceeds the maximum threshold .According to the documentation :
Uploaded items should be retained in memory as long as they are reasonably small.
Larger items should be written to a temporary file on disk.
Very large upload requests should not be permitted.
The built-in defaults for the maximum size of an item to be retained in memory, the maximum permitted size of an upload
request, and the location of temporary files are acceptable.
Try the following :
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
try {
// Set factory constraints
DiskFileItemFactory factory = new DiskFileItemFactory();
factory.setSizeThreshold(yourMaxMemorySize);
ServletContext servletContext = this.getServletConfig().getServletContext();
File repository = (File) servletContext.getAttribute("javax.servlet.context.tempdir");
factory.setRepository(repository);
List<FileItem> items = new ServletFileUpload(factory).parseRequest(request);
for (FileItem item : items) {
if (item.isFormField()) {
// Process regular form field (input type="text|radio|checkbox|etc", select, etc).
String fieldName = item.getFieldName();
String fieldValue = item.getString();
// ... (do your job here)
} else {
// Process form file field (input type="file").
String fieldName = item.getFieldName();
String fileName = FilenameUtils.getName(item.getName());
InputStream fileContent = item.getInputStream();
// ... (do your job here)
}
}
} catch (FileUploadException e) {
throw new ServletException("Cannot parse multipart request.", e);
}
// ...
}
Here, we are providing a temp location for the file since the file is large.
A few things that are worth to try here:
Explicit the encoding: https://stackoverflow.com/a/10488411/4279120
Decompose your call and add iteration and try catch, ex. : https://www.programcreek.com/java-api-examples/?api=org.apache.commons.fileupload.FileItemIterator
Take a look at the MultipartConfig, it seems to provide such attributes as maxFileSize and maxRequestSize (see: https://www.codejava.net/java-ee/servlet/java-file-upload-example-with-servlet-30-api#maxFileSize%28%29)
Manually define the header of your Request if you can. It seems that adding "X-File-Name" and "X-File-Size", can also help, but this is a little old: AJAX File Upload with XMLHttpRequest
We may also help you better if you provide some more informations, like the versions of apache / java / servlet, and a few more code (especially the definition of request)
Some ressources that could be helpful:
XMLHttpRequest
Sending_files_using_a_FormData_object
How to set a header for a HTTP GET request, and trigger file download?
try this to set session timeout using setMaxInactiveInterval method
request.getSession().setMaxInactiveInterval(1200);
parameter Specifies the time, in seconds, between client requests before
the servlet container will invalidate this session. An interval value
of zero or less indicates that thesession should never timeout.
Thanks for all your answer.
Finally, I successfully resolve this issue, but in fact... not really.
I notice that there was some specific things in my form.
I had two inputs, one standard file input, and another which receive the file content encoded in base64 by some weird js before any upload.
So I was having one time the raw content of the file, and also the file in base64. Why ?! I don’t know.
But I delete all this, I create a new simple and clean form with a standard input file.
I use the stream API from ServletFileUpload, and it works, and takes only few seconds for big files.
So I don’t understand everything (why the problem was only on some browser for example), but I find a solution ;)
Thank you !
I am working on setting up a basic network system in Processing:
import processing.net.*;
Server myServer;
Client myClient;
However, I'm having trouble with my server and client side communication set up. My client is interpreting all incoming messages as Strings, and the issue is that whenever multiple messages are sent from the server in the same frame, they become added into a single string, which my program cannot interpret. After testing, I found that sending multiple messages from a single client to the server are given the same treatment.
My client reader looks like this:
while (myClient.available() > 0) {
String dataIn = myClient.readString();
As of now, I don't know if the problem is in the reader (combining the strings), or in the fact that I'm using write() multiple times in a single frame (and the data is being sent as a single string).
I am wondering if the messages can somehow be sent/read separately or, if not, there is some method to test if a message has already been sent (that works for both the client and server side) so that I can set up a queue to keep track of messages to be sent.
Well, I decided to forgo the idea of checking if a message had already been sent as I did not see any functions that would be able to help do that. Instead, I went ahead and created an arrayList of strings for the server called serverQ to act as a queue for messages to be sent:
ArrayList <String> serverQ = new ArrayList<String>();
I also added a function writeSQ(String) that would place any input string into the queue:
void writeSQ(String s) {
serverQ.add(s);
}
I then proceeded to replace every usage of myServer.write(String) with writeSQ(String s). At the end of my ServerUpdate function, I added a section that would empty the queue, sending the next string to all clients, one frame at a time:
// send data
if (serverQ.size() > 0) {
myServer.write(serverQ.get(0));
serverQ.remove(0);
}
}
However, for some reason, messages still got compounded, so I speculated that it may be due to the closeness (frequency) of the sent messages (each frame); so I set a boolean serverSent to alternate the messages sent to every other frame. The new code looks like this:
// send data
if (serverQ.size() > 0) {
if (!serverSent) {
myServer.write(serverQ.get(0));
serverQ.remove(0);
serverSent = true;
} else
serverSent = false;
}
This worked perfectly and the messages were interpreted individually by clients. I added that exact same support code to the clients (changing everything from server to client when needed) and after a good amount of testing, can confirm that this new support works properly both ways.
My initial purpose was to verify the HTTP chunked transfer. But accidentally found this inconsistency.
The API is designed to return a file to client. I use HEAD and GET methods against it. Different headers are returned.
For GET, I get these headers: (This is what I expected.)
For HEAD, I get these headers:
According to this thread, HEAD and GET SHOULD return identical headers but not necessarily.
My question is:
If Transfer-Encoding: chunked is used because the file is dynamically fed to the client and Tomcat server cannot know its size beforehand, how could Tomcat know the Content-Length when HEAD method is used? Does Tomcat just dry-run the handler and count all the file bytes? Why doesn't it simply return the same Transfer-Encoding: chunked header?
Below is my RESTful API implemented with Spring Web MVC:
#RestController
public class ChunkedTransferAPI {
#Autowired
ServletContext servletContext;
#RequestMapping(value = "bootfile.efi", method = { RequestMethod.GET, RequestMethod.HEAD })
public void doHttpBoot(HttpServletResponse response) {
String filename = "/bootfile.efi";
try {
ServletOutputStream output = response.getOutputStream();
InputStream input = servletContext.getResourceAsStream(filename);
BufferedInputStream bufferedInput = new BufferedInputStream(input);
int datum = bufferedInput.read();
while (datum != -1) {
output.write(datum);
datum = bufferedInput.read();
}
output.flush();
output.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
ADD 1
In my code, I didn't explicitly add any headers, then it must be Tomcat that add the Content-Length and Transfer-Encoding headers as it sees fit.
So, what are the rules for Tomcat to decide which headers to send?
ADD 2
Maybe it's related to how Tomcat works. I hope someone can shed some light here. Otherwise, I will debug into the source of Tomcat 8 and share the result. But that may take a while.
Related:
HTTP HEAD and GET different result
Content-Length header with HEAD requests?
Does Tomcat just dry-run the handler and count all the file bytes?
Yes, the default implementation of javax.servlet.http.HttpServlet.doHead() does that.
You can look at helper classes NoBodyResponse, NoBodyOutputStream in HttpServlet.java
The DefaultServlet class (the Tomcat servlet that is used to serve static files) is more wise. It is capable of sending the correct Content-Length value, as well as serving GET requests for a subset of the file (the Range header). You can forward your request to that servlet, with
ServletContext.getNamedDispatcher("default").forward(request, response);
Although it seems strange, it might make sense to send the size only in response to a HEAD request and chunked in response to a GET request, depending on the type of data that has to be returned by the server.
While your API seems to provide a static file, you also talk about dynamically created files or data, so I will be talking in general here (also for webservers in general).
First let's have a look at the different usages for GET and HEAD:
With GET the client is requesting the whole file or data (or a range of the data), and wants it as fast as possible. So there is no specific reason for the server to send the size of the data first, especially when it could start sending faster/sooner in chunked mode. So the fastest possible way is preferred here (the client will have the size after the download anyway).
With HEAD on the other hand, the client usually wants some specific information. This could just be a check on existance or 'last-changed', but it could also be used if the client wants a certain part of the data (with a range request, including a check to see if range requests are supported for that request), or just needs to know the size of the data up front for some reason.
Lest's look at some possible scenarios:
Static file:
HEAD: there's no reason to not include the size in the response-header because that information is available.
GET: most of the time the size will be inluded in the header and the data sent in one go, unless there are specific performance reasons to send it in chunks. On the other hand it seems you are expecting chunked transfer for you file, so this could make sense here.
Live logfile:
Ok, somewhat strange, but possible: downloading a file where the size could change while downloading.
HEAD: again, the client probably wants the size, and the server can easily provide the size of the file at that specific time in the header.
GET: since loglines could be added while downloading, the size is unknown up front. Only option is to send chunked.
Table with fixed-sized records:
Let's imagine a server needs to send back a table with fixed-length records coming from multiple sources/databases:
HEAD: size is probably wanted by the client. The server could quickly do a query for count in each database, and send the calculated size back to the client.
GET: instead of doing a query for count in each database first, the server better starts sending the resulting records from each database in chunks.
Dynamically generated zip-files:
Maybe not common, but an interesting example.
Imagine you want to provide dynamically generated zip-files to the user based on some parameters.
Let's first have a look at the structure of a zip-file:
There are two parts: first there's a block for each file: a small header followed by the compressed data for that file. Then there's a list of all the files inside the zip-file (including sizes/positions).
So the prepared blocks for each file could be pre-generated on disk (and the names/sizes stored in some data structure.
HEAD: the client probably wants to know the size here. The server can easily calculate the size of all the needed blocks + the size of the second part with the list of the files inside.
If the client wants to extract a single file, it could directly ask for the last part of the file (with a range-request) to have the list, and then with a second request ask for that single file. Although the size is not necessarily needed to get the last n bytes, it could be handy if for example if you wanted to store the different parts in a sparse file with the same size of the full zip-file.
GET: no need to do the calculations first (including generating the second part to know its size). It would be better and faster to just start sending each block in chunks.
Fully dynamically generated file:
In this case it wouldn't be very efficient to return the size to a HEAD request of course, since the whole file would need to be generated just to know its size.
Here is my code:
CIwHTTP http;
std::string output="";
char buffer[1024];
int32 httpCallback(void* sys_data, void* user_data) {
http.ReadData(buffer,http.ContentLength());
output += buffer;
return 0;
}
http.Get(url.c_str(), httpCallback, 0);
The content-length header is properly set in the API. For some reason only part of the API output is received. Sometimes it gets the entire API string and sometimes it returns different portions of the string. It seems random. Help!
You are passing ContentLength() to ReadData, but your buffer has only 1024 bytes. Most likely you have stack overflow - pun intended.
You can either call ReadData in a loop with 1024 until it returns zero, or dynamically allocate buffer on the heap.
IwHTTP::Get() only performs the callback once the headers for the response have been received.
You then need to use IwHTTP::ReadContent() to actually read the remainder of the response in a series of callbacks, as hinted at in one of the other comments.
Please see the IwHTTP Example in our API reference documentation for more details.
Hope this helps!
I have written downloading a file in a simple manner:
#ResourceMapping(value = "content")
public void download(ResourceRequest request, ResourceResponse response) {
//...
SerializableInputStream serializableInputStream = someService.getSerializableInputStream(id_of_some_file);
response.addProperty(HttpHeaders.CACHE_CONTROL, "max-age=3600, must-revalidate");
response.setContentType(contentType);
response.addProperty(HttpHeaders.CONTENT_TYPE, contentType);
response.addProperty(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename*=UTF-8''"
+ URLEncoder.encode(fileName, "UTF-8"));
OutputStream outputStream = response.getPortletOutputStream();
byte[] parcel = new byte[4096];
while (serializableInputStream.read(parcel) > 0)
outputStream.write(parcel);
outputStream.flush();
serializableInputStream.close();
outputStream.close();
//...
}
The SerializableInputStream is described here - JavaDocs. It allows an InputStream to be serialized and, for instance, passed over remoting.
I read from input and write it to the output, not all bytes at once. But unfortunately the portlet isn't "streaming" the contents - the file (e.g. an image) is sent to the browser only after reading the entire input stream - this is how it looks like. I see the file being read from the database (from live logs), but I don't see any "growing" image on the screen.
What am I doing wrong? Is it possible to really stream a file in Liferay 6.0.6 and Spring Portlet MVC?
Where are you doing this? I fear that you're doing this instead of rendering your portlet's HTML (e.g. render phase). Typically the portlet content is embedded in an HTML page, thus you need the resource phase, which (roughly) behaves like a servlet.
Also, the code you give does not match the actual question you ask: You use a comment //read from input stream (file), write file to os and ask what to do differently in order to not have the full content in memory.
As the comment does not have anything in memory and you could loop through reading from the input file while writing to the output stream: What's the underlying question? Do you have problems with implementing download-streaming in a portal environment or difficulties (i.e. using too much memory) reading from a file while writing to a stream?
Edit: Thanks for clarifying. Have you tried to flush the stream earlier? You can do that whenever you want - e.g. every loop (though that might be a bit too much). Also, keep in mind that the browser as well as the file itself must handle it in a way that you expect: If an image is not encoded "incrementally" a browser might not show it that way.
Have you tried this with huge files as well? It might be that the automatic flushing is just not triggered because your files are too small for it to be triggered...
Also, I think that filename*=UTF-8'' looks strange. Might be valid encoding, but I've never seen this