I am working on linux.
I have a HTTP client which requests some data from the HTTP server. The HTTP client is written in C and HTTP server is written in perl.
I want to simulate TCP re-transmission timeouts at the client end.
I assume that closing the socket gracefully would not result in client to re-transmit the requests.
So I tried the following scenario:
Exit the Server as soon as it gets the HTTP GET request. However, I noticed that once the application exits, the socket is still closed gracefully. I see that the server initiates FIN.ACK messages towards the client even though the application has not called "close" on the socket. I have noticed this behaviour on a simple TCP server and client written in C program as well.
Server does not send any response to the client's GET request. In this case I notice that there is still FIN, ACK sent by the server.
Seems that in these cases the OS (linux) takes care of closing the socket with the peer.
Is there any way to suppress this behaviour (using ioctl or setsockopt options) or any other way to simulate the TCP re-transmission timeouts.
You could try setting firewall rules that block the packets going from the server to the client, which would cause the client to re-transmit the quests. On Linux, this would probably be done using iptables, but different distributions have different methods of controlling it.
This issue was previously discussed here
Related
Story: In a client-server system I use a long time connection(an http(s) request from client to server with a long time timeout) in order to use notify client to do some actions(Most of data transfer is from client to server but some commands send to client in response of this http(s) request).
Problem: If client cancel the connection server can understand that but if internet connection of client loses(e.g, unplug the LAN cable or it loses the WLAN/GPRS antenna) neither client nor server understand this. Connection still remains until (some time spends and) somebody writes something in it which is too late.
PS: 0) I googled with AKC/NACK, Keep-alive, ping-pong and heartbeat key words for http(s) request and could not find a protocol which it periodically check the status of request.
1) In this you can find an argument for curl command which sets a time interval to send a props(also I monitored this with wireshark). But if still if you unplug the cable neither curl command nor server can understand the connection lost.
curl -k --keepalive-time 5 https://exampel.com/v1/v/f9a64e73/notification
2) Also here explains that there is an http header which is used to use a connection multiple time.
In server side and with nginx web-server we can enable TCP keep-alive probe with so_keepalive=on as listen input argument. Find more information in this link.
I understand that http2 uses one tcp connection to serve multiple requests, for example, if I request index.html which contains a.css and a.js, these three requests will be done in one tcp connection.
What happens if user clicks index2.html? does this request still use the same previous tcp connection? If so, will the browser keep the connection open until user closes the browser? And on the server side, does the server keep many connections open all the time?
When using HTTP/2, browsers typically open only one connection per domain.
In your example, index2.html will be sent on the same TCP connection that was used for index.html, a.css and a.js.
In HTTP/2 requests are multiplexed on the same TCP connection, so that the browser can send them concurrently, without waiting for a previous request to be responded to.
Both browsers and servers have an idle timeout for TCP connections.
If the connection is idle for long enough, it will be closed by either party - the one that has the shorter idle timeout, to save resources.
For example, you may open a connection to a wikipedia.org, perform a few requests, and then leave that tab and work on something else.
After a while (typically 30 seconds) the browser will close the TCP connection to wikipedia.org.
On the server side, the server will keep the connections from various clients open, until they are either closed by the client or until the server-side idle timeout fires, at which point it's the server that initiated the close of the TCP connection.
With HTTP/2, the number of connections that a server has to maintain is vastly less than it was with HTTP/1.1.
With HTTP/2, a server has to maintain just 1 TCP connection per client; with HTTP/1.1, the server had to maintain typically 2-8 TCP connections per client.
What happens if user clicks index2.html? does this request still use the same previous tcp connection?
Yes. On top of that, multiple browser tabs/windows also share a single HTTP/2 connection.
If so, will the browser keep the connection open until user closes the browser?
Below from RFC - connection management
For best performance, it is expected that clients will not close
connections until it is determined that no further communication with
a server is necessary (for example, when a user navigates away from a
particular web page) or until the server closes the connection.
Clients SHOULD NOT open more than one HTTP/2 connection to a given
host and port pair.
And on the server side, does the server keep many connections open all the time?
Servers are encouraged to maintain open connections for as long as
possible but are permitted to terminate idle connections if necessary.
When either endpoint chooses to close the transport-layer TCP
connection, the terminating endpoint SHOULD first send a GOAWAY
(Section 6.8) frame so that both endpoints can reliably determine
whether previously sent frames have been processed and gracefully
complete or terminate any necessary remaining tasks.
More info on connection error below.
RFC connection-error-handling
A connection error is any error that prevents further processing of
the frame layer or corrupts any connection state. An endpoint that
encounters a connection error SHOULD first send a GOAWAY frame with
the stream identifier of the last stream that it successfully received
from its peer. The GOAWAY frame includes an error code that indicates
why the connection is terminating. After sending the GOAWAY frame for
an error condition, the endpoint MUST close the TCP connection. It is
possible that the GOAWAY will not be reliably received by the
receiving endpoint. In the event of a connection error, GOAWAY only
provides a best-effort attempt to communicate with the peer about why
the connection is being terminated.
An endpoint can end a connection at any time. In particular, an
endpoint MAY choose to treat a stream error as a connection error.
Endpoints SHOULD send a GOAWAY frame when ending a connection,
providing that circumstances permit it.
HTTP is client - server communication where client always initiates the connection and server responds.
In the client server communication with HTTP 1.1 the following steps takes place:
1. Client sends the request to the server.
2. Server sends the response to the client with the response message and the status code.
My question is how is the data transfer handled in the protocol? I know HTTP is stateless and also it is either everything or nothing mechanism but how do you prove this? How is the handshake between server and client?
For example: When the server sends the response back to the client, what happens if 50% of the data is sent and then there is connection loss...then what will happen in this scenario? Will the client wait for remaining 50% of the message or it will start new transfer where server tries to send 100% of the message again? (In synchronous communication)
HTTP relies on a TCP connection, so in your example if 50% of the data is correctly sent but others packets (yes, you should think in terms of packets) are lost, the data will be sent again following the rules defined in TCP protocol
As the title suggests, should Ping Frames only be sent from a server or it is better to have both endpoints send them? As mentioned in the Websocket RFC:
NOTE: A Ping frame may serve either as a keepalive...
So by by having one endpoint sending a ping request it should keep the connection open, right?
The second part of above line is this:
or as a means to verify that the remote endpoint is still responsive.
I'm new to the concept of websockets but if the connection closes from the server won't the client be notified?
Consider the case where the server just goes away, maybe it crashes. Who or what will notify the client of this? Or say a network link close to the server is down for so long that by the time it comes back up, the server has totally forgotten about this client. Who or what would tell the client?
There are three possibilities:
The client does not need to detect loss of the connection. In this case, there's nothing special you need to do.
The client has some way to detect loss of the connection already. For example, if the connection is idle for some period of time, the client could send an application-level query and timeout if it gets no response or if the query fails.
The client needs to detect loss of the connection but has no existing way to do this. In this case, using pings makes sense.
In a typical query/response protocol, the client usually doesn't need to ping the server because there's some query it can send that has the same effect. Unless the protocol layered above websocket supports some way for the server to query the client, the server often has only two choices: use pings to detect lost connections or disconnect idle clients.
Both variants has ways of implementation. For example in case if server sends ping to client, then client can get information that server disconnected by have a loop with deadline timer which is reset every time when ping is received. If the timer reaches dealine, then it's mean that server disconnected.
I've observed a HTTP 1.1 Server implementation, which terminates a client connection as soon as it detects a client-side connection shutdown of its outgoing channel (or rather, either before or after sending a proper http response). Is this a conforming HTTP 1.1 implementation?
RFC 2616 Section 8.1.4 seems to suggest this is to be the proper behaviour:
When a client or server wishes to time-out it SHOULD issue a graceful
close on the transport connection. Clients and servers SHOULD both
constantly watch for the other side of the transport close, and
respond to it as appropriate.
...
Servers SHOULD NOT close a connection in the middle of transmitting a response, unless a network or client failure is suspected.
Am I interpreting it right? Is there a more explicit reference about half-closed connection handling in the context of HTTP 1.1?
As far as i know, thats is all we need to know about Half-closed connections.
The server will only close the connection if it detects that the client closed it (it can ben when the server is about to write to the socket) or at the end of the request, if it does not support connection: keep-alive.
The client can disconnect any time, but it should tell the server why is it disconnecting (time_out, request cancel). But it is not very used by those who write sockets components. They just close the socket when they need to force a time_out.
But the client implementation is not the problem. You should worry about server implementation since suffer a lot with those unexpected disconnects.
EDIT
Maybe those links can help you.
Transmission Control Protocol - Functional Specification
TRANSMISSION CONTROL PROTOCOL