How to collect the HTTP response status for a script?
Below is a sample code which will start a server and allow two routes for interaction.
using Genie
import Genie.Router: route
import Genie.Renderer.Json: json
Genie.config.run_as_server = true
route("/try/", method=GET) do
(:message => "Welcome") |> json
end
route("/test/", method=POST) do
data = jsonpayload()
<body>
end
Genie.startup()
How to collect the response status like 200, 500 or others as a string variable?
Open connection to your server using HTTP and look for the status field:
julia> using HTTP
julia> response = HTTP.get("http://127.0.0.1:8000/try")
HTTP.Messages.Response:
"""
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Server: Genie/1.18.1/Julia/1.6.1
Transfer-Encoding: chunked
{"message":"Welcome"}"""
julia> response.status
200
If you rather want to control the status yourself you can add on the server side:
route("/tryerror/", method=GET) do
Genie.Responses.setstatus(503)
end
And now let us test it for 503:
julia> response = HTTP.get("http://127.0.0.1:8000/tryerror")
ERROR: HTTP.ExceptionRequest.StatusError(503, "GET", "/tryerror", HTTP.Messages.Response:
"""
HTTP/1.1 503 Service Unavailable
Content-Type:
Server: Genie/1.18.1/Julia/1.6.1
Transfer-Encoding: chunked
""")
Related
I have a strange situation. I want to return the content type application/json; charset=utf-8 from an http handler.
func handleTest() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Accept") != "application/json" {
w.WriteHeader(http.StatusNotAcceptable)
return
}
w.WriteHeader(http.StatusOK)
w.Header().Set("Content-Type", "application/json; charset=utf-8")
json.NewEncoder(w).Encode(map[string]string{"foo": "bar"})
}
}
When I check for this in my unit tests it is correct. This test does not fail.
func TestTestHandler(t *testing.T) {
request, _ := http.NewRequest(http.MethodGet, "/test", nil)
request.Header.Set("Accept", "application/json")
response := httptest.NewRecorder()
handleTest().ServeHTTP(response, request)
contentType := response.Header().Get("Content-Type")
if contentType != "application/json; charset=utf-8" {
t.Errorf("Expected Content-Type to be application/json; charset=utf-8, got %s", contentType)
return
}
}
But when I try with curl (and other clients) it comes out as text/plain; charset=utf-8.
$ curl -H 'Accept: application/json' localhost:8080/test -v
* Trying 127.0.0.1:8080...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /test HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.68.0
> Accept: application/json
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Tue, 28 Dec 2021 13:02:27 GMT
< Content-Length: 14
< Content-Type: text/plain; charset=utf-8
<
{"foo":"bar"}
* Connection #0 to host localhost left intact
I have tried this with curl, insomnia and python. In all 3 cases the content type came out as text/plain; charset=utf-8.
What is causing this problem and how can I fix it?
From the http package docs:
WriteHeader sends an HTTP response header with the provided status code.
and
Changing the header map after a call to WriteHeader (or Write) has no effect unless the modified headers are trailers.
So you are setting the "Content-Type" header after the header has already been sent out to the client. While mocking this likely works because the buffer where the headers are stored can be modified after the WriteHeader call. But when actually using a TCP connection you can't do this.
So simply move your w.WriteHeader(http.StatusOK) so it happens after the w.Header().Set(...)
I have written lua code and integrated in nginx to upload files into s3 bucket. If the upload fails or signature is malformed, i get an 403 Forbidden which is send to the client along with very verbose Error XML.
I wanted to intercept/ return default error msg instead of the verbose error msg from s3 which reveals Access Key ID.
I tried adding snippet below but it has no effect what so ever:
error_page 403 /forbidden;
location = /forbidden {
internal;
content_by_lua_block {
local cjson = require "cjson"
ngx.header.content_type = "application/json; charset=utf-8"
ngx.status = 403
ngx.say(cjson.encode({ status = "Forbidden. Check the Request" }))
return ngx.exit(ngx.status)
}
}
When i see the reponse of curl cmd to upload, There are two status codes, 100 and then 403.
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 403 Forbidden
< Date: Thu, 28 Sep 2017 04:26:31 GMT
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Connection: keep-alive >
How do i handle this to return default response for all 403 status code response.
I'm trying to make a GET call in R using httr and I keep getting an authorization 401 error.
R code:
testfunction2 <- function()
{
set_config(verbose())
locus_url <- "https://api.locusenergy.com/v3/clients/5599"
r <- GET(url = "https://api.locusenergy.com/v3/clients/5599",
query=list(authorization="Bearer c935845d8fc1124757e66ce04d2c75d0"),
Accept="application/json")
}
The results:
> print(testfunction2())
-> GET /v3/clients/5599
authorization=Bearer%20c935845d8fc1124757e66ce04d2c75d0 HTTP/1.1
-> User-Agent: libcurl/7.39.0 r-curl/0.9.1 httr/1.0.0
-> Host: api.locusenergy.com
-> Accept-Encoding: gzip, deflate
-> Cookie: AWSELB=D91FBFE1087EF6EBC125A126777051237474A8A060B6095B8E3C16151308453F8556B2A2E90CB2178F365FAA8AA8C29B124D15CA3EB859CFE615428E8D55C393ABB5B436BF
-> Accept: application/json, text/xml, application/xml, */*
->
<- HTTP/1.1 401 Unauthorized
<- Content-Type: application/json
<- Date: Sun, 16 Aug 2015 05:02:27 GMT
<- Server: Apache-Coyote/1.1
<- transfer-encoding: chunked
<- Connection: keep-alive
<-
I expect it to return a 200 code (rather than 401, that implies authorization error.)
I know the token is correct because it works if I use the Postman (google add-in) and Python. The token won't work for you because I changed it since I can't share it.
Python Code:
import http.client
conn = http.client.HTTPSConnection("api.locusenergy.com")
headers = {
'authorization': "Bearer 935845d8fc1124757e66ce04d2c75d0"
}
conn.request("GET", "/v3/clients/5599", headers=headers)
res = conn.getresponse()
data = res.read()
print(data)
results from Python
b'{"statusCode":200,"partnerId":4202,"tz":"US/Arizona","firstName":"xxx","lastName":"xxxx","email":"xxxx#aol.com","id":5599}'
So, again the question is what am I doing wrong in R or can you give me any hints? This won't be reproducible for you because the token expired and I can't share it.
Could it be the space in the authorization? authorization="Bearer 935845d8fc1124757e66ce04d2c75d0"? Are there any hints in the verbose output of the get call in R?
For reference, this is the site's API page:
https://developer.locusenergy.com/
The site requires OAUTH2 authentication to return the token. I didn't include that code but I verified the token works with Python and Postman.
Right now you are passing your authorization values in the query string of the httr code and not in the http header as you are doing in the python code. Instead use
GET(url = "https://api.locusenergy.com/v3/clients/5599",
accept_json(),
add_headers(Authorization="Bearer c935845d8fc1124757e66ce04d2c75d0")
)
I am trying to simply download a page with python.
http://webapps.rrc.state.tx.us/CMPL/viewPdfReportFormAction.do?method=cmplP4FormPdf&packetSummaryId=97770
If i get the response code from the server i get 200
import urllib2
url = 'http://webapps.rrc.state.tx.us/CMPL/viewPdfReportFormAction.do?method=cmplP4FormPdf&packetSummaryId=97770'
file_pointer = urllib2.urlopen(url)
print file_pointer.getcode()
However if i get the url i get the redirect page
file_pointer.geturl()
I have tried urllib, urllib2,requests, and mechanize all separately and can not get any to work. I am obviously missing something because other people in the office have code that works. SOS
Also here is more information provided by requests
import requests
url = 'http://webapps.rrc.state.tx.us/CMPL/viewPdfReportFormAction.do?method=cmplP4FormPdf&packetSummaryId=97770'
proxy = { 'https': '200.35.152.93:1212'}
response = requests.get(url, proxies=proxy)
send: 'GET /CMPL/viewPdfReportFormAction.do?method=cmplP4FormPdf&packetSummaryId=97770 HTTP/1.1\r\nHost: webapps.rrc.state.tx.us\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-requests/2.7.0 CPython/2.7.10 Windows/7\r\n\r\n'
reply: 'HTTP/1.1 302 Found\r\n'
header: Date: Wed, 26 Aug 2015 19:33:12 GMT
header: Server: Apache/2.2.15 (Red Hat)
header: Location: http://www.rrc.state.tx.us/site-policies/railroad-commission-of-texas-site-policies/?method=cmplP4FormPdf&packetSummaryId=97770
header: Content-Length: 405
header: Connection: close
header: Content-Type: text/html; charset=iso-8859-1
send: 'GET /site-policies/railroad-commission-of-texas-site-policies/?method=cmplP4FormPdf&packetSummaryId=97770 HTTP/1.1\r\nHost: www.rrc.state.tx.us\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-requests/2.7.0 CPython/2.7.10 Windows/7\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: private
header: Content-Type: text/html; charset=utf-8
header: server: one
header: Date: Wed, 26 Aug 2015 19:33:11 GMT
header: Content-Length: 41216
The problem is that this specific site is looking for your User Agent header, and since you're a python client, it disallows you to get the PDF and redirect you.
Therefore you need to mask your user agent.
Look at the following example:
url = 'http://webapps.rrc.state.tx.us/CMPL/viewPdfReportFormAction.do?method=cmplP4FormPdf&packetSummaryId=97770'
req = urllib2.Request(url)
req.add_unredirected_header('User-Agent', 'Mozilla/5.0')
file_pointer = urllib2.urlopen(req)
print file_pointer.getcode()
print file_pointer.geturl();
Okay so all one has to do with the requests module is to disable redirection.Here is my working code that is also using a proxy server.
import requests
url = 'http://webapps.rrc.state.tx.us/CMPL/viewPdfReportFormAction.do?method=cmplP4FormPdf&packetSummaryId=97770'
proxy = { 'https': '200.35.152.93:1212'}
r = requests.get(url, proxies=proxy,allow_redirects=False)
print r.url
I'm just trying to get the response body when the post request fails using RCurl. When I'm running it through R, I only get the response error and it fails out.
Error: Unprocessable Entity
when I post the same request with the UI interface for testing, it gives
{
"reason": [
"Can not create Data with Name: DataTest. Data Name should be unique."
],
"singleReason": "Can not create Data with Name: DataTest. Data Name should be unique."
}
Any help on how to get response bodies on 4XX errors is appreciated. TIA.
Post request
postdata.json <- '{"name":"DataTest","description":"Test Payload","algorithm":{"name":"DataTest","version":"0.1.0"}}'
post.result <- httpPOST(url=SERVER, postfields = postdata.json, verbose = T,
httpheader=c(Authorization=access.token, 'Content-Type'='application/json', Accept='application/json'))
RStudio output
* About to connect() to SERVER port 80 (#0)
* Trying SERVER... * connected
* Connected to SERVER port 80 (#0)
> POST /api/test HTTP/1.1
Host: SERVER
Authorization: AUTHENTICATION
Content-Type: application/json
Accept: application/json
Content-Length: 171
< HTTP/1.1 422 Unprocessable Entity
< Content-Type: application/json;charset=UTF-8
< Date: Thu, 13 Nov 2014 16:31:42 GMT
< Server: Apache-Coyote/1.1
< Content-Length: 215
< Connection: keep-alive
<
* Connection #0 to host SERVER left intact
Show Traceback
Rerun with Debug
Error: Unprocessable Entity
n.b. I'm okay with it failing and returning an Error. That's expected. I'm just trying to get the response body associated with the error code.
Here's one approach with httr:
library(httr)
postdata.json <- '{"name":"DataTest","description":"Test Payload","algorithm":{"name":"DataTest","version":"0.1.0"}}'
res <- POST(SERVER,
body = postdata.json,
add_headers(
Authorization = access.token,
'Content-Type' = 'application/json',
Accept = 'application/json'
)
)
http_status(res)
content(res)
I was able to get the response body using RCurl by adding a basicTextGatherer() and a writefunction.
reader <- basicTextGatherer()
post.result <- httpPOST(url=SERVER, postfields = postdata.json, verbose = T,
writefunction = reader$update,
httpheader=c(Authorization=access.token,
'Content-Type'='application/json',
Accept='application/json')
)
return(body=reader$value())
but Hadley's solution is simpler to work with and doesn't error out.