I'm struggling with this question and did not found information in google, that's why I can't understand what are they talking about.
Please follow,
I was given a https address and a port, this address run a (as they called and I copy here) "TPC/IP using SSL. It's a "real time" protocol, that will give you real time data of the channels you subscribe"
So, I was given a document where they specify how to connect to some channels. It reads:
This protocol is based on JSON format. The default port for this application is XXXXX, and the connection will be established using a SSL TCP-IP connection.
Commands sent and received have the following format:
command-id:[message-id[+]]:[channel]:[data]
command-id = Valid commands are LOGIN, JOIN, LEAVE, PING , ACK , ERROR, REPLY, CMD , JSON (mandatory)
[message-id] = Identification for message (optional on client)
[channel] = channel name for command (optional on client)
[data] = Json formated data <-- (this data I copy an example below) (mandatory)
All the commands use a \r\n (CR + LF) at the end of each line
Example of [data] = {“user”:”XX”, , ”password”:”YYY”, app”:”ZZZ”, “app_ver”:”zzz”
”protocol”:”xxx”,”protocol_ver”:”xxxx”}
Also, I provide an example of a complete command:
LOGIN:::{"user":"myname","password":"mypassword","app":"Manual Test", "app_ver":"1.0.0" ,
"protocol":"CustomProtocolABCD", "protocol_ver":"1.0.0"}
Here is what I tried:
Postman, forming the commands, message and data with Parameters, with Body, URL Encoding , everything. I only get "Error: Parse Error: Expected HTTP/" from postman
CURL, tried this also and it prompt me with this odd message "Expected HTTP 1.1 but response is HTTP 0.9". Ok I force this to --http0.9 and I finally get a response with a similar shape:
ERROR:1::{"reason":"Wrong number of parameters"}
Here is the question, How should I test it to send the right amount of parameters the server is expecting? I will provide here below mi CURL with credentials erased of course.
curl https://eu.xxxxxxxxxx.com:11001 -H 'Content-Type: application/json' -k --verbose -d 'command-id:LOGIN' -d 'message-id:100' -d 'channel:' --http0.9 -d 'data:{"user":"XXXXXXX","password":"xxxxxxx","app":"ManualTest","app_ver":"1.0.0","protocol":"CustomProtocolABCD","protocol_ver":"1.0.0"}' -i
NOTE: The password contain a "%" symbol, I don't know if this is generating a problem in the encode but I'm very lost here.
Can someone help me pointing any documentation about this kind of communication? I'm really supposed to make an app consuming this information in a embedded device (Qt Creator) but I can't test the endpoint to receive initial json data and then program the app.
Any help is welcomed. Thanks and excuse my english if I made some mistake.
Thanks again
Your data is not a valid JSON string.
This is the post data you are posting:
command-id:LOGIN&message-id:100&channel:&data:{"user":"XXXXXXX","password":"xxxxxxx","app":"ManualTest","app_ver":"1.0.0","protocol":"CustomProtocolABCD","protocol_ver":"1.0.0"}
This part of the post data is valid JSON:
'{"user":"XXXXXXX","password":"xxxxxxx","app":"ManualTest","app_ver":"1.0.0","protocol":"CustomProtocolABCD","protocol_ver":"1.0.0"}
But your header says it the whole body of the packet is Content-Type: application/json
Your actual content is mixed data where the one field data is JSON.
This is the non-JSON part:
command-id:LOGIN&message-id:100&channel:&data:
That looks like a mishmash of post data and JSON combined.
I can only guess but I would think data should look like this:
-d '{"command-id":"LOGIN","message-id":100,"channel":{"user":"XXXXXXX","password":"xxxxxxx","app":"ManualTest","app_ver":"1.0.0","protocol":"CustomProtocolABCD","protocol_ver":"1.0.0"}}'
Which translates to this:
obj(
'command-id' => 'LOGIN',
'message-id' => 100,
'channel' =>
obj(
'user' => 'XXXXXXX',
'password' => 'xxxxxxx',
'app' => 'ManualTest',
'app_ver' => '1.0.0',
'protocol' => 'CustomProtocolABCD',
'protocol_ver' => '1.0.0',
),
)
But I am thinking curl is not a valid protocol for this TCP/IP packet.
You might be able to use curl if you send a body with no header.
Where the body is an SSL encoded packet.
What programming language[s] do you use?I think I could do this in PHP with sockets and a little better documentation.
The solution is to use
openssl s_client -c connect host.com:11111
Additionally I had to use \" instead of only " to overcome json encoding in my server.
It was indeed a SSL socket over TCP IP
In Qt I'm using QSslSocket class
Additional information for whoever needs it, this kind of sockets use to have a ping routine where you need to ping the server with a custom command to keep the connection alive.
In Qt I'm programming threads, and one of them will be used to ping the server once connection is established.
Related
I'm working on a very simple and straightforward reverse proxy in rust without any external libraries. I'm come to my first roadblock. I've noticed that when I try to parse an https request into utf8 it fails. I printed the request as a lossy string. Here is the output:
�f�^���;�r�;�d��N7# ^�8�6 �m�xpPk�
����B]���Fi��֚*G]"�+�/̨̩�,�0�
� ����/5�rus
I was thinking this has something to do with ssl because on the client side, it says something along the lines of "Secure Connection has Failed". I've looked into decoding ssl requests or whatever this is and have found nothing useful. Any ideas would be greatly appreciated.
I have tried parsing the request using several different solution from other platforms. They consisted of relying on base64 and other ssl related crates meant for decoding text.
For more context, below is a general example for how I go about getting the output from above:
use std::{
io::{Read, Result},
net::TcpListener,
};
fn main() -> Result<()> {
let server = TcpListener::bind("localhost:443")?;
for mut stream in server.incoming().filter_map(Result::ok) {
let mut buf = [0; 256];
let bytes = stream.read(&mut buf)?;
let utf8_lossy = String::from_utf8_lossy(&buf); // this contains the non-utf8 wumbo jumbo
let utf8 = String::from_utf8(buf.to_vec()).unwrap(); // this fails
}
Ok(())
}
When an https client connects with the server, they establish a secure socket to protect data transfer between them. The data that is being passed over this socket is not necessarily text, and cannot be interpreted as such.
The process of establishing a socket is a multi-step protocol, where the client sends a ClientHello message, to which you should reply with a ServerHello which contains your certificate. The client then replies with it's keys and some cipher information, before the socket is finally ready to be used for data. All of these initalization steps are happening with a binary protocol, that cannot be interpreted as text. That is the reason you're not seeing any sensible output.
Once that socket is setup, only then does http data begin to flow over the connection. This is likely what you're expecting to see, as it contains the familiar 'HTTP/1.1 GET', etc.
Openssl, the library you mentioned using, has a way to setup a socket that will perform handshakes for you. See the docs.
I am trying to create a TCP Server which will store incoming TCP Packets as binary, for a Key/Value Store. I already have an Erlang client which can send TCP packets to my Erlang Server, however for the sake of completeness, I want to allow the user to send TCP packets from a command line using clients such as NetCat. The user would adhere to a spec of how to format the data in the TCP Packet such that the Server will be able to understand it. For Example
$ nc localhost 8091
add:key:testKey
Key Saved!
add:value:testValue
Value Saved!
get:key:testKey
Value: testValue
The user interacts with the server by using the add:key/value: and get:key:. What is after that should be taken literally and passed to the server. Meaning a situation like this could be possible, if the user so wanted to.
$ nc localhost 8091
add:key:{"Foo","Bar"}
Key Saved!
add:value:["ferwe",324,{2,"this is a value"}]
Value Saved!
get:key:{"Foo","Bar"}
Value: ["ferwe",324,{2,"this is a value"}]
However, this doesn't seem possible to do as what actually happens is as follows...
I will pre-fill the erlang key/value store (using ETS) using my erlang client with a key of {"Foo","Bar"} and a value of ["ferwe",324,{2,"this is a value"}]. A tuple and list respectively (in this example) as this key/value store has to be able to accommodate ANY erlang compliant data type.
So in the example, currently there is 1 element in the ETS table:
Key
Value
{"Foo","Bar"}
["ferwe",324,{2,"this is a value"}]
I then want to retrieve that entry using NetCat by giving the Key, so I type in NetCat...
$ nc localhost 8091
get:key:{"Foo","Bar"}
My Erlang Server, receives this as <<"{\"Foo\",\"Bar\"}\n">>
My Erlang Server is set up to receive binary which is not an issue.
My question is therefore, can NetCat be used to send unencoded Packets which doesn't escape the quote marks.
Such that my Server is able to receive the Key and just <<"{"Foo","Bar"}">>
Thank you.
My question is therefore, can NetCat be used to send unencoded Packets which doesn't escape the quote marks.
Yes, netcat sends exactly what you give it, so in this case it sends get:key:{"Foo","Bar"} without escaping the quote marks.
Such that my Server is able to receive the Key and just <<"{"Foo","Bar"}">>
<<"{"Foo","Bar"}">> is not a syntactically correct Erlang term. Do you want to get the tuple {"Foo","Bar"} instead, in order to look it up in the ETS table? You can do it by parsing the binary:
Bin = <<"{\"Foo\",\"Bar\"}\n">>,
%% need to add a dot at the end for erl_parse
{ok, Tokens, _} = erl_scan:string(binary_to_list(Bin) ++ "."),
{ok, Term} = erl_parse:parse_term(Tokens),
ets:lookup(my_table, Term).
I have been struggling to replicate an issue we are facing in Production. The clients are sending multiple headers with the same name via a cookie and we are trying to troubleshoot the same via CURL.
The intent is to send TWO header values for the same header name so that the application (below as myhost) can intercept it via this curl attempt. However, when I attempt something like this, the server, the "x-targetted-group" value doesn't resolve. IF I send TWO headers using -H "X-targetted-group:Group1" - "x-targetted-group:Group2", the server only gets the first one. How can i send both ?
curl -i -H "Accept: application/json" -H "x-targetted-group:Group1,Group2" https://myhost:8990/"
curl won't let you. So answer is you can't. Later version of wget won't either.
If you want to experiment with odd possibly malformed HTTP requests, you can just craft your own - it's all just plain text. Example using netcat:
> cat request.txt # I.e. the contents of the file request.txt is:
GET /
Accept: application/json
X-targetted-group: Group1
X-targetted-group: Group2
> nc myhost 8990 <request.txt
The HTTP spec says lines have to end in CRLF (\r\n) so the above might not be accepted by your server unless the text file request.txt uses CRLF line termination (there is an option for saving like that in text editors ..).
Aside: What HTTP spec says about multiple headers with the same name (they are allowed):
Multiple message-header fields with the same field-name MAY be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list [i.e., #(values)]. It MUST be possible to combine the multiple header fields into one "field-name: field-value" pair, without changing the semantics of the message, by appending each subsequent field-value to the first, each separated by a comma. The order in which header fields with the same field-name are received is therefore significant to the interpretation of the combined field value, and thus a proxy MUST NOT change the order of these field values when a message is forwarded.
I use to perform a lot of bad queries syntax attacks on HTTP servers. By definition curl or wget wont let you do much bad syntax work.
You should try to use low level netcat + printf.
With printf you write your HTTP query, and netcat will manage the socket connection (for ssl connections you can replace netcat with openssl_client).
That would look like (for a basic query):
printf 'GET /my/url?foo=bar HTTP/1.1\r\n'\
'Host: www.example.com\r\n'\
'\r\n'\
| nc -q 2 127.0.0.1 80
And for a more complex one (repeated header & old ops-fold header syntax, not also how to write a %character in printf):
printf 'GET /my/url?foo=bar&percent_char=%% HTTP/1.1\r\n'\
'Host: www.example.com\r\n'\
'x-foo-header: value1\r\n'\
'x-foo-header: value2\r\n'\
'x-foo-header: value3, value4\r\n'\
'x-foo-header:\t\tval5\r\n'\
' val6\r\n'\
'User-agent: tests\r\n'\
'\r\n'\
| nc -q 2 127.0.0.1 80
Once you get used of it it's a pleasure, no limitations.
This is a limitation of the HTTP protocol itself. You are not allowed to send multiple headers with the same name unless they are sent in the same key as a comma separated list of values. Take a look at this answer.
I am trying to upload data from an Arduino to data.sparkfun.com, but somehow it always fails. To make sure that the HTTP request I am sending is correct, I would like to send it from a computer to the server and see if it uploads the correct values.
According to some examples, the request should be formulated like this:
GET /input/publicKey?private_key=privateKey&dht1_t=24.23&dht1_h=42.4&dht2_t=24.48&dht2_h=41.5&bmp_t=23.3&bmp_p=984021 HTTP/1.1\n
Host: 54.86.132.254\n
Connection: close\n
\n
How do I send this request to the server from my computer? Do I just type in the terminal? Im not sure where to start.
Have a look at curl which should be able to handle your needs.
Even easier and more low level is netcat (here is an example on SO)
I'm pulling data from a server but need to know the type of data before I pull it. I know I can look at content-type in the response header, and I've looked into using
curl --head http://x.com/y/z
however some servers do not support the "HEAD" command (I get a 501 not implemented response).
Is it possible to somehow do a GET with curl, and immediately disconnect after all headers have been received?
Check out the following answer:
https://stackoverflow.com/a/5787827
Streaming. UNIX philosphy and pipes: they are data streams. Since curl and GET are unix filters, ending the receiving pipe (dd) will terminate curl or GET early (SIGPIPE). There is no telling whether the server will be smart enough to stop transmission. However on a TCP level I suppose it would stop retrying packets once there is no more response. #sehe
Using this method you should be able to download as many bytes as you want, and then cancel the request. You could also work some magic to terminate after receiving a blank line, which would mean the end of the header.