I am trying to upload data from an Arduino to data.sparkfun.com, but somehow it always fails. To make sure that the HTTP request I am sending is correct, I would like to send it from a computer to the server and see if it uploads the correct values.
According to some examples, the request should be formulated like this:
GET /input/publicKey?private_key=privateKey&dht1_t=24.23&dht1_h=42.4&dht2_t=24.48&dht2_h=41.5&bmp_t=23.3&bmp_p=984021 HTTP/1.1\n
Host: 54.86.132.254\n
Connection: close\n
\n
How do I send this request to the server from my computer? Do I just type in the terminal? Im not sure where to start.
Have a look at curl which should be able to handle your needs.
Even easier and more low level is netcat (here is an example on SO)
Related
I'd like to send a incomplete http request, or some kind of request that will temporarily block my server for a while. I wrote the server myself in C, and it is currently designed to only accept one client at a time. I want to test that this is indeed the case.
Would it be possible to send something simple, similar to GET /HTTP/1.0? I'm just doing all my testing in my terminal, not using anything else so far.
Yes, you can do this with netcat.
nc -c servername 80 <<<"GET / HTTP/1.0"
This will send the GET line and then wait for the server to respond. But the server should be waiting for headers and the blank line that ends them, so it will never respond. So nc will wait forever, keeping the connection open.
I came across the term HTTP. I have done some research and wanted to ensure that I correctly understood the term.
So, is it true that HTTP, in simple words, a letter containing information in the language that both client and server can understand.
Then, that letter is sent to the server thanks to TCP/IP which serves as a car that takes that letter to the server.
Then, after the letter is delivered to the server, the server reads the content of the letter and if it is GET request, the server takes the necessary data and ATTACHES that data to the letter and sends back to the client via again TCP/IP. But if it was POST request then the client ATTACHES the DATA to the letter and sends it to the server so that it saves that data in the database.
Is that true?
Basically, it is true.
However, the server can decide what to do if it is a GET or POST or any other request(it doesn't need to e.g. append it to a file).
I will show you some additional information/try to explain it in my words:
TCP is another communication protocol protocol. It allows a client to open a connection to a server and they can communicate afterwards.
HTTP(hyper text transfer protocol) builds up on TCP.
At first, the client opens a connection to the server.
After that, the client sends the HTTP Request. The first line contains the type of the request, the path and the version. For example, it could be GET / HTTP/1.1.
The next part of the request contains the Request parameters. Every parameter is a line. The parameters are sent like the following: paramName: paramValue
This part of the request ends with an empty line.
If it is a POST Request, query parameters are added next. If it is a GET Request, these query parameters are added with the path(e.g. /index.html?paramName=paramValue)
After rescieving the Request, the server sends a HTTP Response back to the client.
The first line of the response contains the HTTP version, the status code and the status message. For example, it could be HTTP/1.1 200 OK.
Then, just like in the request, the response parameters are following. For example Content-Length: 1024.
The response parameters also end with an empty line.
The last part of the response is the body/content. For example, this could be the HTML code of the website you are visiting.
Obviously, the length of the content/body of the response has to match the Content-Length parameter(in bytes).
After that, the connection will be closed(normally). If the client to e.g. request resources, it will send another request. The server has NO POSSIBILITY to send data to the client after that unless the client sends another request(websockets can bypass this issue).
GET is meant to get the content of a site A web browser will send a GET request if you type in a URL. POST can be used to update a site but in fact, the server can decide that. POST can be also used if the server doesn't want query parameters to be shown in the address bar.
There are other methods like PATCH or DELETE that are used by some APIs.
Some important status codes (and status messages) are:
200 OK (everything went well)
204 No content (like ok but there is no body in the response)
400 Bad Request (something is wrong with the Request)
404 Not found (the requested file(the path) was not found on the server)
500 Internal server error (An error occured while processing the request)
Every status code beginning with 1 is related to inform the client of something.
If it is starting with 2, everything went right.
Status code beginning with 3 forward the client to another site.
If it starts with 4, there is a error on the client side.
Codes starting with 5 represent an error that occured on the server side.
TCP is a network protocol that establishes a connection with the server over a network (or the Internet) and allows two-way communication. The HTTP will traffic inside this TCP tunnel. TCP is a very useful protocol that helps keep things sane, it ensures data packets are read in the correct order and that packets that went missing during transmission are sent again.
Sometimes there will be another protocol layer between HTTP and TCP, called SSL. It is responsible for encrypting the data that traffics over TCP, so that it is transmitted safely over unsafe networks. This is know as HTTPS, and is just HTTP but using this additional layer.
Although almost always true, HTTP doesn't necessarily uses TCP. UPnP requests use HTTP over UDP, a network protocol that uses standalone packets instead of a connection.
HTTP is a plain text protocol, meaning it's designed in such a way that a human can understand it without using any tools. This is very convenient for learning.
If you're using Firefox or Chrome, you can press Ctrl-Shift-C to open the Developer Tools, and under the Network tab you will see every HTTP request your browser is making, see exactly what's the request, what the server answered etc, and get a better view of how this protocol works.
Explaining it in details is... too extensive for this answer. But as you will see it's not that complicated.
All,
My requirement is fairly simple. I have to perform a simple HTTP POST to an IP:port combination. I used simple socket programming to do that and I have been successful in sending across my request to them and also get back response from them. The only problem being that the response is always a HTTP 400: Bad Request followed by my HTTP POST message. I am not sure if the problem is with the client or the server. My only guess being that there might be a problem with my data that I am sending. This is what my POST looks like
POST /<Server Tag> HTTP/5.1
Content-Length: xxx
--Content--
and the response from the server looks something like this
HTTP/1.1 400 Bad Request
Content-Length: xxx
--Same content that I sent them--
I was not sure If I could put in the IP of the server here so kept myself to using . I am pretty sure that the problem would not be there since I get back some response from the server and confident about the connection. Can someone help me ?
PS: Some pointers about my POST:
1) HTTP 5.1 was requested by the server and I am not sure if that is correct
2) I have played around with the number of line spaces after the content length. I have tried giving one and two lines. Not sure if that would make a difference. On wireshark though I see a difference with the number of line spaces as with a single line space the protocol is specified as TCP but with two it changes to HTTP. The response is always received on HTTP protocol. Some explanation on the difference would also help
Thanks
edit: the other thing that confuses me is that the response has a HTTP 1.1 and not a 5.1 that I had sent. I have also tried changing my post to 1.1 with no success
edit2: Based on suggestion form fvu and others, I used WebClient to Upload my request. Still got back a 400. The header that was generated by the WebClient looks like this
POST <server tag> HTTP/1.1
Host: <IP:PORT>
Content-Length: 484
Expect: 100-continue
Connection: Keep-Alive
The issue I see with this might be that the server was not expecting all the details in the header. The server has requested only the Content-Length from us. Would that be a problem?
Thanks
You can use a debugging proxy to view a client request and a server response to figure out what your client socket program needs to do.
But first you need to create a simple web page that a browser displays, allows you to do a POST from the browser to the web server, and get a simple response back from the server.
HTTP/5.1 is either wrong or misused by the programmer of the server application
You should get a valid example from the server api to check your protocol implementation first.
I'm pulling data from a server but need to know the type of data before I pull it. I know I can look at content-type in the response header, and I've looked into using
curl --head http://x.com/y/z
however some servers do not support the "HEAD" command (I get a 501 not implemented response).
Is it possible to somehow do a GET with curl, and immediately disconnect after all headers have been received?
Check out the following answer:
https://stackoverflow.com/a/5787827
Streaming. UNIX philosphy and pipes: they are data streams. Since curl and GET are unix filters, ending the receiving pipe (dd) will terminate curl or GET early (SIGPIPE). There is no telling whether the server will be smart enough to stop transmission. However on a TCP level I suppose it would stop retrying packets once there is no more response. #sehe
Using this method you should be able to download as many bytes as you want, and then cancel the request. You could also work some magic to terminate after receiving a blank line, which would mean the end of the header.
This question is similar to
Starting to use OpenSSL
but more specific and detailed so I think it's fair to ask.
Suppose I have an simple http server that does the following in a successful GET scenario
creates a listening socket
a client connects
reads the data through recv
parses the GET request, now it knows which resource to return
writes the response through send
close the socket
This server is written in c++ on linux.
My question is, What does it take to convert this server into an minimal https server? (in particular using OpenSSL, but answers in a general sense are welcome.)
Here's my understanding (question marks mean I'm have no idea)
initialize the library
read the server certificate and private key and other configurations
create a normal listening socket(?)
a client connects
do the handshaking through a library function(?)
handshaking done
do I need a special step before I start receiving and sending data?
read data through library function(?)
does the data look exactly like an HTTP GET at this point?
if it does, parse the GET and get the resource
write return data through library function(?)
close the connection through a library function(?)
In summary, I'm hoping that it only requires adding some extra steps to the current code and does not affect the HTTP parsing. Is this assumption correct?
Many thanks to anybody who could fill in the blanks.
Look through "Network Security with OpenSSL", as it covers this. Even if you don't have the book, you can look through the code.