Sending POST with groovy and data is already URL-encoded - http

I'm trying to send a POST using Groovy HTTPBuilder but the data I want to send is already URL-encoded so I want HTTPBuilder to POST it as is. I tried the following:
def validationString = "cmd=_notify-validate&" + postData
def http = new HTTPBuilder(grailsApplication.config.grails.paypal.server)
http.request(Method.POST) {
uri.path = "/"
body = validationString
requestContentType = ContentType.TEXT
response.success = { response ->
println response.statusLine
}
}
But it gives me a NullPointerException:
java.lang.NullPointerException
at groovyx.net.http.HTTPBuilder$RequestConfigDelegate.setBody(HTTPBuilder.java:1200)

Since you're using pre-encoded form values you cannot use the default map-based content type encoder. You must specify the content type so the EncoderRegistry knows how to handle the body.
You may create the HttpBuilder with a content type that specifies the body is a URL-encoded string:
def http = new HTTPBuilder(url, ContentType.URLENC)
Or make the request passing the content type explicitly:
http.request(Method.POST, ContentType.URLENC) {
// etc.
For reference, here's how I figured it out--I didn't know before I read the question.
Looked at the request method's API docs to see what the closure was expected to contain. I've used HTTPBuilder only in passing, so I wanted to see what, specifically, the body "should" be, or "may" be, and if the two were different.
The four-arg version of the request method links to the RequestConfigDelegate class and said the options were discussed in its docs.
The RequestConfigDelegate.setBody method, which is what the body setter is, states the body "[...] may be of any type supported by the associated request encoder. That is, the value of body will be interpreted by the encoder associated with the current request content-type."
The request encoder link is to the EncoderRegistry class. It has an encode_form method taking a string and states it "assumes the String is an already-encoded POST string". Sounds good.
The request content-type link was to an HttpBuilder inner class method, RequestConfigDelegate.getRequestContentType, which in turn had a link to the ContentType enum.
That enum has a URLENC value that led me to believe it'd be the best first guess.
I tried the HTTPBuilder ctor taking a content type, and that worked.
Circled back to the request methods and noticed there's a version also taking a content type.
I'd guess the total time was ~5-10 minutes, much shorter than it took to type up what I did. Hopefully it'll convince you, though, that finding this kind of stuff out is possible via the docs, in relatively short order.
IMO this is a critical skill for developers to groom, and make you look like a hero. And it can be fun.

Related

Tracing an HTTP request in Go in a one structured log line

I have learned you can "decorate" the HTTP transport so that you can details of the request, however I can't quite figure out how you log the URL in the same line.
https://play.golang.org/p/g-ypQN9ceGa
results in
INFO[0000] Client request dns_start_ms=0 first_byte_ms=590 response_code=200 total_ms=590 url=
INFO[0000] 200
I'm perpetually confused if I should be using https://golang.org/pkg/context/#WithValue to pass around the context in a struct, especially in light where https://blog.golang.org/context-and-structs concludes with pass context.Context in as an argument.
Go through the behaviour of how the request is constructed in request.go from net/http. You see that the RequestURI field is never set there. Quoting from same reference,
Usually the URL field should be used instead.
It is an error to set this field in an HTTP client request
So, I would suggest you to use request.URL instead.
It is a parsed from the request uri. It should have the data you need.
You can construct the output as following:
f := log.Fields{
"url": fmt.Sprintf("%s %s%s", r.Method, r.URL.Host, r.URL.Path),
}
Also, in my experience, it is far more easier to use context.WithValue and pass the context as an argument.
Replace r.RequestURI by r.URL.String() in your code to log the full, valid URL (https://golang.org/pkg/net/url/#URL.String). RequestURI is empty on the client side (https://golang.org/pkg/net/http/#Request), as the output from your code is showing.
I don't see how context.Context relates to your question, but I believe https://blog.golang.org/context-and-structs is considered "best practice".

How does a Body get applied to an HTTP request?

In a previous post (link below) I was able to figure out how to apply a body to an HTTP request using VB.NET and the HttpWebRequest class object. However, I'm trying to understand how this is working technically. In other words:
How is the body actually being added to the http request because I don't see it being assigned to a property of the request object?
Here's the code snippet.
Dim vHTTPREQUESTBODY As String = "the body of the request"
Dim vBYTEVERSIONOFBODY As Byte() = Encoding.ASCII.GetBytes(vHTTPREQUESTBODY)
vHTTPREQUEST.ContentLength = vBYTEVERSIONOFBODY.Length
Dim vDATASTREAM As Stream = vHTTPREQUEST.GetRequestStream()
vDATASTREAM.Write(vBYTEVERSIONOFBODY, 0, vBYTEVERSIONOFBODY.Length)
vDATASTREAM.Close()
Any help appreciated and more information below for those interested.
More Information
Why the question:
What is confusing to me is that you cannot clearly see where/how the body is added to the request before it is sent. As in older VB objects doing the same, there would be simple “Body” property for that object to set a value to. For example, in line 3 of the above, you can see the “vHTTPREQUEST.ContentLength” property set. However, you don’t see something like that for the body as in “vHTTPREQUEST.BodyOrBlahBlahBlah”. Yet, somehow the above does add the body string you set in the “vHTTPREQUESTBODY” variable to the request.
Can anyone explain how? I've search online and nothing specifically addresses this.
This is my guess:
Take a look at line 5. If you look up the “Write” method of the “Stream” object it basically states what is happening is that you are adding the bytes you placed in the first parameter of that method to the end of the current stream. So... I’m assuming what is happening is that the stream is in the computer’s memory at that point of the code, and when you use the “Write” method you are adding your body in the form of bytes to that stream that is already in memory. Therefore, you don’t need to explicitly assign it to a request of the property. Again that is a guess. I was hoping for confirmation or the true explanation or a resource/reference where I can read about this.
Previous Post:
Here's a link to previous post

Are PUT and POST requests required/expected to have a request body?

I'm writting a RESTful api, and at I'm thinking about the process of a user creating a key. I have the following possibilities:
GET request to /new/<keyname> - although it's very easy I think I won't use this, because I heard GET is for retrieving and/or listing information;
POST request to /<keyname> - This seemed to me easy and simple enough, but does not pass any data in the request body. Can I do it this way ? Is this weird ?
POST request to /keys passing in the request body "keyname=SomeKey" - Is this the correct way ?
I looked at this API from joyent and in all their PUT and POST requests they pass some data in the request body. Is this expected ? Is it really wrong not to require a request body in a PUT and POST request ?
I asked this question on the Http-WG. This was the most precise answer I got http://lists.w3.org/Archives/Public/ietf-http-wg/2010JulSep/0276.html
In summary, POST does not require a body. I would expect the same justification can be applied to PUT.
RFC2616 is the base RFC for HTTP 1.1
In the most general form, an HTTP message is this (note the optional body):
generic-message = start-line
*(message-header CRLF)
CRLF
[ message-body ]
start-line = Request-Line | Status-Line
Reading further gives this:
9.5 POST
The POST method is used to request that the origin server accept the
entity enclosed in the request as a new subordinate of the resource
identified by the Request-URI in the Request-Line. ...
and
9.6 PUT
The PUT method requests that the enclosed entity be stored under the
supplied Request-URI. ...
The fundamental difference between the POST and PUT requests is
reflected in the different meaning of the Request-URI. The URI in a
POST request identifies the resource that will handle the enclosed
entity. That resource might be a data-accepting process, a gateway to
some other protocol, or a separate entity that accepts annotations.
In contrast, the URI in a PUT request identifies the entity enclosed
with the request -- the user agent knows what URI is intended and the
server MUST NOT attempt to apply the request to some other resource.
Both POST and PUT include the phrase entity enclosed in the request.
Based on my reading, I believe that a body is desired (a non-normative description, I know) for both POST and PUT.
In the context of REST, POST is create and PUT is update. I can imagine creating an empty object (perhaps a placeholder for future information), but I don't imagine much use of an empty update.
It is not required. You can send a POST/PUT request without a body and instead use query string parameters. But be careful if your parameters contain characters that are not HTTP valid you will have to encode them.
For example if you need to POST 'hello world' to and end point you would have to make it look like this: http://api.com?param=hello%20world
Probably the best way is your third option: POST to /keys with keyname=SomeKey.
Here's why: You may wish to add another function to your API, for example create_new_user. It would then be difficult to tell the difference between a user trying to POST a key called create_new_user and a user trying to use the create_new_user function.
You are correct in saying that you should not be using GET to do this operation as the GET operation "SHOULD NOT have the significance of taking an action
other than retrieval." (RFC 2616).
To answer your question in one line. Yes it is expected to have Body/Content in body, but it is not required(Mandatory).
According to okHttp3 (an HTTP library for android): the following methods need a body: POST, PUT, PATCH, PROPPATCH (WebDAV) and REPORT (source). It even crashes if you try to do a request with the given methods without a body.

Is it considered bad practice to perform HTTP POST without entity body?

I need to invoke a process which doesn't require any input from the user, just a trigger. I plan to use POST /uri without a body to trigger the process. I want to know if this is considered bad from both HTTP and REST perspectives?
I asked this question on the IETF HTTP working group a few months ago. The short answer is: NO, it's not a bad practice (but I suggest reading the thread for more details).
Using a POST instead of a GET is perfectly reasonable, since it also instructs the server (and gateways along the way) not to return a cached response.
POST is completely OK. In difference of GET with POST you are changing the state of the system (most likely your trigger is "doing" something and changing data).
I used POST already without payload and it "feels" OK. One thing you should do when using POST without payload: Pass header Content-Length: 0. I remember problems with some proxies when I api-client didn't pass it.
If you use POST /uri without a body it is something like using a function which does not take an argument .e.g int post (void); so it is reasonable to have function to your resource class which can change the state of an object without having an argument. If you consider to implement the Unix touch function for a URI, is not it be good choice?
Yes, it's OK to send a POST request without a body and instead use query string parameters. But be careful if your parameters contain characters that are not HTTP valid you will have to encode them.
For example if you need to POST 'hello world' to and end point you would have to make it look like this: http://api.com?param=hello%20world
Support for the answers that POST is OK in this case is that in Python's case, the OpenAPI framework "FastAPI" generates a Swagger GUI (see image) that doesn't contain a Body section when a method (see example below) doesn't have a parameter to accept a body.
the method "post_disable_db" just accepts a path parameter "db_name" and doesn't have a 2nd parameter which would imply a mandatory body.
#router.post('/{db_name}/disable',
status_code=HTTP_200_OK,
response_model=ResponseSuccess,
summary='',
description=''
)
async def post_disable_db(db_name: str):
try:
response: ResponseSuccess = Handlers.databases_handler.post_change_db_enabled_state(db_name, False)
except HTTPException as e:
raise (e)
except Exception as e:
logger.exception(f'Changing state of DB to enabled=False failed due to: {e.__repr__()}')
raise HTTPException(HTTP_500_INTERNAL_SERVER_ERROR, detail=e.__repr__())
return response

How do we pass parameters when doing HTTP Post?

I am working on an app where we have to pass specific web api parameters to a web app using HTTP POST.
eg:
apimethod name
parameter1 value
parameter2 value
So do I use a string or URLEncodedPostData to send that data?
It would be good if u help me with a code eg.
I am using something like this but it doesnt post the data to the server.
Though the response code is ok/200 and I also get get a parsed html response when i read the httpresponse input stream. But the code doesnt post anything. So unable to get the expected response.
_postData.append("method", "session.getToken");
_postData.append( "developerKey", "value");
_postData.append( "clientID", "value");
_httpConnection = (HttpConnection) Connector.open(URL, Connector.READ_WRITE);
String encodedData = _postData.toString();
_httpConnection.setRequestMethod(HttpConnection.POST);
_httpConnection.setRequestProperty("User-Agent", "BlackBerry/3.2.1");
_httpConnection.setRequestProperty("Content-Language", "en-US");
_httpConnection.setRequestProperty("Content-Type","application/x-www-form-urlencoded");
_httpConnection.setRequestProperty("Content-Length",(new Integer(encodedData.length())).toString());
os = _httpConnection.openOutputStream();
os.write(requeststring.getBytes());`
The code you posted above looks correct - although you'll want to do a few more things (maybe you did this already but didn't include it in your code):
Close the outputstream once you've written all the bytes to it
Call getResponseCode() on the connection so that it actually sends the request
POSTed parameters are usually sent in the response BODY, which means URL-encoding them is inappropriate. Quote from the HTTP/1.1 protocol:
Note: The "multipart/form-data" type has been specifically defined
for carrying form data suitable for processing via the POST
request method, as described in RFC 1867 [15].
The post method allows you to use pretty arbitrary message bodies — so it is whatever format the server wants.

Resources