I need to send a file to an HTTP server with the curl utility. However, as my server application only needs to process a single file, I'd like to avoid using some large HTTP framework with a wide range of functionality, so I'd like to stick to TCP/UDP protocols with some simple HTTP parser.
A file to send to the server may be quite large and I doubt if it is reasonable to send this file as a single TCP packet, so I'm dreaming of splitting this file into UDP packets and sending them one by one. But on the client side, all this must be done with a simple curl command, like curl --data-binary #filename 127.0.0.1:80.
Is it possible to split this request into several packets using WinSock API? For example, the server reads the name of a file, detects its size, allocates as many packets as needed and starts receiving UDP packets. Or maybe should I look at other ways of solving it?
Related
SSEs are advertised as a unidirectional communication tool to be used from server to client. I have a requirement to broadcast data to all clients and so i was wondering how SSEs behave on a low level. I cannot seem to find any low level information about SSEs online.
Primarily i would like to know if, after sending the data, does the server wait for a response from the client to confirm it has received the data before finishing the "send". That would mean that doing a broadcast using a for loop would be quiet dangerous and slow in which case websockets might be the better options.
Perhaps the implementation depends entirely on the language and framework? Is it not standardized?
Broadcast usually uses UDP which does not wait for a response. - - Broadcasting ip:port by socket server
.. says
UDP Packet: First four bytes as a magic number, next four bytes an IPv4 address (and you might want to add other things like a server name).
The magic number is just in case there is a collision with another application using the same port. Check both the length of the packet and the magic number.
Server would broadcast the packet at something like 30 second time intervals. (Alternatively you could have the server send a response only when a client sends a request via broadcast.)
So the client app would have to send a request back to the server app.
Different protocols would get different responses according the the underlying technology. eg HTTP uses responses extnsivly.
SSE and WebSockets are both over TCP, so there could be a wait before the socket could be used to send further data.
However, each client is a dedicated socket. So server-side you would be using threads or async coding (depending on the server-side language and its conventions). So looping through all the sockets to send a message to each client would be fine and quick.
I write a simple server application. In that application, I created a server socket and put it into the listen state with listen call.
After that, I did not write any code to accept the incoming connection request. I simply waited for the termination with pause call.
I want to figure out practically that how many bytes are buffered in the server side if the connection is not accepted. Then I want to validate the number with the theory of the TCP.
To do that,
First, I started my server application.
Then I used "dd" and "netcat" to send the data from client to server. Here is the command:
$> dd if=/dev/zero count=1 bs=100000 | nc 127.0.0.1 45001
Then I opened wireshark and wait for the zero-window message.
From the last properly acknowledged tcp frame. the client side can successfully send 64559 byte data to the server.
Then I execute the above dd-netcat command to create another client and send data again.
In this case, I got the following wireshark output:
From the last successfully acknowledged tcp frame, I understand that the client application can successfully sent 72677 bytes to the server.
So, it seems that the size of the related buffer can change in runtime. Or, I misinterpret the output of the wireshark.
How can I understand the size of the related receive buffer? What is the correct name to refer that receive buffer in terminology? How can I show the default size of the related receive buffer?
Note that the port number of the tcp server is "45001".
Thank you!
I want to write an application layer sniffer (SMTP/ftp/http).
Based on my searchs, first (and perhaps hardest!) step is to reassemble the tcp stream of the sniffed connections.
Indeed, what I need is something like the "follow TCP stream" option of wireshark, but I need a tool which do it on live interface and automatically. As I know, Tshark can extract TCP streams data from the saved pcap files automatically (link) but not from live interfaces. Can Tshark do it on live interfaces???
As I know, TCPflow can do exactly what I want, however, it can not handle IP defragmentation and SSL connections (I want to analyse the SSL content in the case I have the server private key).
Finally, I also try bro network monitor. Although it provides the list of TCP connections (conn.log), I was not able to get TCP connections contents.
Any suggestion about mentioned tools or any other useful tool is welcome.
Thanks in advance, Dan.
perl Net::Inspect library might help you. It also comes with a tcpudpflow which can write tcp and udp flows into separate files, similar to tcpflow. It works on pcap files or can do live captures. The library handles IP fragmenting. It also comes with a httpflow tool to extract HTTP requests and responses (including decompression, chunked encoding..). It does not currently handle SSL.
As the author of this library I don't think that extracting TCP flows is the hardest part, the HTTP parser (exluding decompression, including chunked mode) is nearly twice as big than IP and TCP combined.
This example works for reassembling application data of a single protocol:
tshark -Y "tcp.dstport == 80" -T fields -d tcp.port==80,echo -e echo.data
It captures live http data, reassembles it, and outputs it as raw hex.
I can add a small script to parse the hex into ascii if you like.
I want to analyse the SSL content in the case I have the server private key
TL;DR: This can't be done with a capturing tool alone.
Why not: Because each SSL session generates a new secret conversation key, and you can't decrypt the session without this key. Having the private server key is not enough. The idea behind this is that if someone captures your SSL traffic, saves it, and then a year later he "finds" the private server key, then he still won't be able to decrypt your traffic.
While reading the FTP protocol specification from : (http://www.pcvr.nl/tcpip/ftp_file.htm). I came across this "FTP differs from the other applications that we've described because it uses two TCP connections to transfer a file". My question is, can FTP have multiple TCP connection for multiple parallel file transfer, for example can I transfer two files in parallel over two TCP connections, is this a matter of customization or standardization?
While it would be theoretically possible to make an FTP server support multiple, concurrent transfers, it's not supported by the RFC or any known implementation.
The block is a simple one in that the control connection, after receiving a transfer request, does not return a final status or accept new commands until the data transfer is completed. Thus, though you could queue up another transfer request it wouldn't actually be processed by the server until the current one completes.
If you want multiple file transfers, just log into the FTP server multiple times using different programs or command-line windows and have each initiate a transfer.
No it can't. FTP uses a control connection for sending commands and a data connection that exists for the duration of the file transfer or directory listing retrieval, that's it.
For more information you can consult RFC 959, which defines the specs of the FTP protocol.
Is a http end point suppose to respond to requests from a particular client in order that they are received?
What about if it doesn't make sense to in the case of requests handled by cluster behind a proxy or in requests handled with NIO where one request is finished faster than the other?
Is there a standard way of associating a unique id with each http request to associate with the response? How is this handled in clients like http componenets httpclient or curl?
The question comes down to the following case:
Suppose, I am downloading a file from a server and the request is not finished. Is a client capable of completing other requests on the same keep-alive connection?
Whenever a TCP connection is opened, the connection is recognized by the source and destination ports and IP addresses. So if I connect to www.google.com on destination port 80 (default for HTTP), I need a free source port which the OS will generate.
The reply of the web server is then sent to the source port (and IP). This is also how NAT works, remembering which source port belongs to which internal IP address (and vice versa for incoming connections).
As for your edit: no, a single http connection can execute one command (GET/POST/etc) at the same time. If you send another command while you are retreiving data from a previously issued command, the results may vary per client and server implementation. I guess that Apache, for example, will transmit the result of the second request after the data of the first request is sent.
I won't re-write CodeCaster's answer because it is very well worded.
In response to your edit - no. It is not. A single persistent HTTP connection can only be used for one request at once, or it would get very confusing. Because HTTP does not define any form of request/response tracking mechanism, it simply would not be possible.
It should be noted that there are other protocols which use a similar message format (conforming to RFC822), which do allow for this (using mechanisms such as SIP's cSeq header), and it would be possible to implement this in a custom HTTP app, but HTTP does not define any standard mechanism for doing this, and therefore nothing can be done that could be assumed to work everywhere. It would also present a problem with the response for the second message - do you wait for the first response to finish before sending the second response, or try and pause the first response while you send the second response? How will you communicate this in a way that guarantees messages won't become corrupted?
Note also that SIP (usually) operates over UDP, which does not guarantee packet ordering, making the cSeq system more of a necessity.
If you want to send a request to a server while another transaction is still in progress, you will need to create a new connection to the server, and hence a new TCP stream.
Facebook did some research into this while they were building their CDN, and they concluded that you can efficiently have 2 or 3 open HTTP streams at any one time, but any more than that reduces overall transfer time because of the extra packet overhead cost. I would link to the blog entry if I could find the link...