HTTP request, strange socket behaviour - http

Question

I experience strange behavior when doing HTTP requests through sockets, here the request:
POST https://example.com:443/service/XMLSelect HTTP/1.1
Content-Length: 10926
Host: example.com
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 1.0.3705)
Authorization: Basic XXX
SOAPAction: http://example.com/SubmitXml
Later on there goes body of my request with given content length.
After that I receive something like:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: text/xml;charset=utf-8
Transfer-Encoding: chunked
Date: Tue, 30 Mar 2010 06:13:52 GMT
So everything seem to be fine here. I read all contents from network stream and successfully receive response. But my socket which I'm doing polling on switches it's modes like that:
write ( i write headers and request here )
read ( after headers sent i begin to receive response )
write ( STRANGE BEHAVIOUR HERE. WHY? here i send nothing really )
read ( here it switches to read back again )
last two steps can repeat several times. So I want to ask what leads for socket's mode change? And in this case it's not a big problem, but when I use gzip compression in my request ( no idea how it's related ) and ask server to send gzipped response to me like this:
POST https://example.com:443/service/XMLSelect HTTP/1.1
Content-Length: 1076
Accept-Encoding: gzip
Content-Encoding: gzip
Host: example.com
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 1.0.3705)
Authorization: Basic XXX
SOAPAction: http://example.com/SubmitXml
I receive response like that:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Encoding: gzip
Content-Type: text/xml;charset=utf-8
Transfer-Encoding: chunked
Date: Tue, 30 Mar 2010 07:26:33 GMT
2000
�
I receive a chunk size and GZIP header, it's all okay. And here's what is happening with my poor little socket meanwhile:
write ( i write headers and request here )
read ( after headers sent i begin to receive response )
write ( STRANGE BEHAVIOUR HERE. And it finally sits here forever waiting for me to send something! But if i refer to HTTP I don't have to send anything more! )
What can it be related to? What it wants me to send? Is it remote web server's problem or do I miss something?
PS All actual service references and login/passwords replaced with fake ones :)

Answer 1

A socket becomes writable whenever there's space in the socket send buffer. The OS can't really know if your application has more data to send, but knows about its internal structures, like socket buffers. You have to explicitly add/remove the socket to the write fd_set for select(2) (enable/disable EPOLLOUT event for epoll(4)). This is usually handled with a state machine, like in libevent. Also polling works best with non-blocking sockets.
Hope this helps.