I am trying to POST with URL encoded data. Based on the web server logs, I am not actually sending any data from Twilio (request size is always 131 bytes, no matter what I type in the Studio widget box).
What does a working form body look like? Do I need to encode it myself? How do I escape an "=" that is not part of the key-value structure?
When making an HTTP request with the widget, when it is set to make Form URL encoded requests you can set the HTTP parameter keys and values which will automatically encode the values. There are known as URL parameter as the encoding is Form URL encoding. The parameters are encoded as if they were in a URL, but they are sent as the POST body.
I am new to backend service and I was tracing API calls from a banking website. Normally, I have seen parameters in POST requests being encoded with base64 encoding. However, I came across a type of request where the date was encrypted with a type of encoding that I am unable to figure out.
For date: 19/12/2018 the encoding is: U2FsdGVkX1/o1qw9zIiZBHLAbGck6j15wwUZ/z/zLqw=
and for date: 18/12/2019 the encoded string is: U2FsdGVkX1/mo5+FfuqqqbUtCsdFObB8eKvyosc4b8E=
I am aware of only base64 encoding, but since I am unable to decode this with base64, it seems this is using something else. Appreciate if anyone can help and share some knowledge about the different ways in parameters can be sent to backend in a secret manner.
The encoding/encryption seems to be happening from the frontend side and I feel this could also be an encrypted string with seed sent in a separate parameter. Appreciate if someone can atleast share a list of possible algorithms that I can look into to understand the request sent and create my own requests.
I'm sending some pre-encrypted strings (AES) to a REST API, this is data that has been encrypted by the client. I am storing the data, and never decrypting it on the server side.
I will hand it back to the client later.
My initial thought was to use base64 encoding to ensure the data survives the form post. Is this the correct way to do things?
You do not need any special encoding to POST data to a server, as there is no restriction to what characters you can use.
If you choose to send the encrypted data as binary data, you should specify content-type header to be application/octet-stream.
I get some string data from a webservice in utf-8. How do I convert it in an aspx vb to a readable format? The website is german.
UTF-8 is readable. ASP.NET should be able to read it just fine. If it's transmitted with a Content-Type whose charset parameter is set to something other than UTF-8 you might need to instruct ASP.NET to force the decoding to UTF-8. Use Fiddler and figure out how the HTTP request looks like and pay special attention to the Content-Type parameter.
If you have a different output-encoding than UTF-8, you should still be able to output the characters correctly if you decode them with the correct encoding. What is your output encoding? What encoding is the web service you're communicating with using? Figure the answer to these questions (using Fiddler) and your solution should be obvious.
When a web server gets a POST of a form, parsing it into param-value(s) pairs is quite straightforward. However, if the values contain non-English chars that have been encoded by the browser, it must know the charset used in order to decode them.
I've examined the requests sent by two posts. One was done from a page using UTF-8, and one from a page using Windows-1255. The same text was encoded differently. AFAIK, the Content-type header could contain a charset after the application/x-www-form-urlencoded, but it wasn't (using Firefox).
In a servlet, when you use request.getParameter(), you're supposed to get the decoded value. How does the servlet container do that? Does it always bet on UTF-8, use some heuristics, or is there some deterministic way I'm missing?
From the Serlvet 3.0 Spec, section 3.10 Request Data Encoding (emphasis mine)
Currently, many browsers do not send a char encoding qualifier with the ContentType header, leaving open the determination of the character encoding for reading
HTTP requests. The default encoding of a request the container uses to create the
request reader and parse POST data must be “ISO-8859-1” if none has been specified
by the client request. However, in order to indicate to the developer, in this case, the
failure of the client to send a character encoding, the container returns null from
the getCharacterEncoding method.
If the client hasn’t set character encoding and the request data is encoded with a
different encoding than the default as described above, breakage can occur. To
remedy this situation, a new method setCharacterEncoding(String enc) has
been added to the ServletRequest interface. Developers can override the
character encoding supplied by the container by calling this method. It must be
called prior to parsing any post data or reading any input from the request. Calling
this method once data has been read will not affect the encoding.
In practice, I find that setting the charset in a response influences the charset used in the subsequent POST. To be extra sure, you can write a Servlet Filter that calls the setCharacterEncoding on every request object before it is used.
You may also find this thread useful - Detecting the character encoding of an HTTP POST request
The apropriate header for specifying charsets is Accept-Charset.
Latest Chrome for linux, e.g., spits:
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
on each request.
Section 14.2 from http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html states:
The Accept-Charset request-header field can be used to indicate what character sets are acceptable for the response. This field allows clients capable of understanding more comprehensive or special- purpose character sets to signal that capability to a server which is capable of representing documents in those character sets.
(...)
If no Accept-Charset header is
present, the default is that any
character set is acceptable. If an
Accept-Charset header is present, and
if the server cannot send a response
which is acceptable according to the
Accept-Charset header, then the server
SHOULD send an error response with the
406 (not acceptable) status code,
though the sending of an unacceptable
response is also allowed.
So if you receive such a header from a client, the value with highest q can be the encoding you're receiving from it.