HTTP2 post request in pycurl - nginx

I am trying to hit a nginx server that is behind an alb. I am triggering a simple HTTP2 POST request using pycurl. My request looks like this'
import io
import pycurl
curl = pycurl.Curl()
curl.setopt(curl.URL, "https://alb-xxxx.us-east-1.elb.amazonaws.com/endpoint")
curl.setopt(pycurl.HTTP_VERSION, pycurl.CURL_HTTP_VERSION_2_PRIOR_KNOWLEDGE)
curl.setopt(pycurl.POST, 1)
curl.setopt(pycurl.VERBOSE, 1)
curl.setopt(pycurl.SSL_VERIFYPEER, 0)
curl.setopt(pycurl.SSL_VERIFYHOST, 0)
with io.BytesIO() as buf:
curl.setopt(curl.WRITEFUNCTION, buf.write)
curl.perform()
print(buf.getvalue())
curl.close()
The error i am getting is
error Traceback (most recent call last)
<ipython-input-91-0756005b6c1b> in <module>()
16 with io.BytesIO() as buf:
17 curl.setopt(curl.WRITEFUNCTION, buf.write)
---> 18 curl.perform()
19 print(buf.getvalue())
20 curl.close()
error: (92, 'HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)')
the error occurrence is random. some times I get correct response and sometimes I get error like this. But when I try to do the same in curl I am getting correct response always. This is my curl command.
curl -v -k --http2-prior-knowledge -X POST https://alb-xxx.us-east-1.elb.amazonaws.com/endpoint
My post request response is a string({"field":"value"}). My NGINX configuration file looks like this
location =/endpoint {
if ($request_method = POST) {
return 200 '{"field":"value"}';
}
}
I don't know why I am getting correct response in curl but getting a different response in pycurl. Am I doing anything wrong in pycurl ?
Pycurl version -> 7.43.0.6
Python version -> 3.8.5
Thanks. Any help is appreciated.

Related

How to resolve octave machine learning submission error?

How to resolve submission error:
curl: (6) Couldn't resolve host 'www-origin.coursera.org'
m = 15
error] submission with curl() was not successful
!! Submission failed: Grader sent no response
Function: submitWithConfiguration>validateResponse
FileName: C:\Users\admin\Desktop\ex7\lib\submitWithConfiguration.m
LineNumber: 158
This error is probably caused because your computer is not able to connect to the internet at the moment.
This is the function working behind collecting the data from the online grader.
function response = validateResponse(resp)
% test if the response is json or an HTML page
isJson = length(resp) > 0 && resp(1) == '{';
isHtml = findstr(lower(resp), '<html');
if (isJson)
response = resp;
elseif (isHtml)
% the response is html, so its probably an error message
printHTMLContents(resp);
error('Grader response is an HTML message');
else
error('Grader sent no response');
end
end
Now the statement: "Grader sent no response" is printed when the response is null.
And the response can be null when the computer is not connected.
Hope this is the reason behind your error, if not then let me know.
Matlab?
This steps may helpful:
open file: submitWithConfiguration.m ; and: goto line 131 and 134;
then change:
line131: json_command = sprintf('echo jsonBody=%s | curl -k -X POST -d #- %s', body, url);
line134: json_command = sprintf('echo ''jsonBody=%s'' | curl -k -X POST -d #- %s', body, url);
to:
line131: json_command = sprintf('echo jsonBody=%s | curl -k -X POST -s -d #- %s', body, url);
line134: json_command = sprintf('echo ''jsonBody=%s'' | curl -k -X POST -s -d #- %s', body, url);
(both add -s)
it looks like the code is fine and the problem is that your computer can't connect to the internet.
You can simply solve this by using a VPN.
Good luck!

Riak db returns 500 Internal server error

I seem to be having an issue with our riak-kv db v2.2.3, we are currently running 5 node cluster, out of nowhere last week suddenly requests to our db started returning 500 internal server error for all/almost every request (also might be valuable to mention our db has nearly doubled in size over the past 2 weeks.) At first, we thought the issue was in the code making the requests however after sshing into one of the nodes in the cluster and attempting a simple request to list buckets we saw this:
Command:
curl -i http://localhost:8098/buckets?buckets=true
Response:
HTTP/1.1 500 Internal Server Error
Vary: Accept-Encoding
Server: MochiWeb/1.1 WebMachine/1.10.9 (cafe not found)
Date: Sun, 10 Dec 2017 06:21:20 GMT
Content-Type: text/html
Content-Length: 1193
<html><head><title>500 Internal Server Error</title></head><body>
<h1>Internal Server Error</h1>The server encountered an error while
processing this request:<br><pre>{error,
{error,
{badmatch,{error,mailbox_overload}},
[{riak_kv_wm_buckets,produce_bucket_list,2,
[{file,"src/riak_kv_wm_buckets.erl"},{line,225}]},
{webmachine_resource,resource_call,3,
[{file,"src/webmachine_resource.erl"},{line,186}]},
{webmachine_resource,do,3,
[{file,"src/webmachine_resource.erl"},{line,142}]},
{webmachine_decision_core,resource_call,1,
[{file,"src/webmachine_decision_core.erl"},{line,48}]},
{webmachine_decision_core,decision,1,
[{file,"src/webmachine_decision_core.erl"},{line,562}]},
{webmachine_decision_core,handle_request,2,
[{file,"src/webmachine_decision_core.erl"},{line,33}]},
{webmachine_mochiweb,loop,2,
[{file,"src/webmachine_mochiweb.erl"},{line,72}]},
{mochiweb_http,headers,5,
[{file,"src/mochiweb_http.erl"},{line,105}]}]}}</pre><P><HR>
<ADDRESS>mochiweb+webmachine web server</ADDRESS></body></html>
After some more investigation into the issue, I pulled the logs on one of the riak nodes and saw this:
2017-12-10 03:54:58.654 [error]
<0.24342.271>#yz_solrq_helper:send_solr_ops_for_entries:301 Updating a
batch of Solr operations failed for index <<"attachment">> with error
{error,{other,{ok,"500",[{"Content-Type","application/json;
charset=UTF-8"},{"Transfer-Encoding","chunked"}],<<"
{\"responseHeader\":{\"status\":500,\"QTime\":1}, error": {
"msg ": "Exception writing document id 1*default*Attachment*07d8e24-dc32s-11e7-q9640-a7f8b4edb446*623 to the index; possible analysis error.",
"trace": ""org.apache.solr.common.So lrException: Exception writing document id 1*default*Attachment*07d8e24-dc32s-11e7-q9640-a7f8b4edb446*623 to the index; possible analysis error.
at org.apache.solr.update. DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:169)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
at or g.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd (DistributedUpdateProcessor.java:952)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:692)
at org.apache.so lr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:141)
at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLo ader.java:106)
at org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:68)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java :99)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at org.apache.solr.handler.RequestHandlerBase.handleRequ est(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilte r.java:777)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFi lter.java:207)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(Servle tHandler.java:455)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHand ler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(Contex tHandler.java:1075)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHa ndler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedH andler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at org.eclipse.jetty.server.handler.HandlerCo llection.handle(HandlerCollection.java:154)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.han dle(Server.java:368)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.BlockingHttpConnec tion.handleRequest(BlockingHttpConnection.java:53)
at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at org.eclipse.jetty.s erver.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
at org.ecli pse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at org.eclips e.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java :608)
at org.eclip..." >>
}
}
}
After some googling some people said this error is
Caused by: java.lang.OutOfMemoryError: Java heap space
So I modified my riak config from:
search.solr.jvm_options = -d64 -Xms4g -Xmx4g -XX:+UseStringCache -XX:+UseCompressedOops
To:
search.solr.jvm_options = -d64 -Xms8g -Xmx8g -XX:+UseStringCache -XX:+UseCompressedOops
However I am still getting the same error any help is much appreciated.

Openresty torch GPU module loading issue

I am using openresty with lua. When I was loading torch without cutorch,cunn(GPU) it was working fine. But it is unable to load cutorch module.
Here is my nginx.conf
error_log stderr notice;
daemon off;
events{}
http {
lua_code_cache on;
#lua_package_cpath '/usr/local/cuda/include/?.so;;';
init_by_lua_file 'evaluate.lua';
server {
listen 6788;
lua_code_cache on;
lua_need_request_body on;
client_body_buffer_size 10M;
client_max_body_size 10M;
location /ff {
# only permit POST requests
if ($request_method !~ ^(POST)$ ) {
return 403;
}
content_by_lua '
local body = ngx.req.get_body_data()
if body == nil then
ngx.say("empty body")
else
local resp =FeedForward(body)
ngx.say(cjson.encode({result=resp}))
end
';
}
}
}
Here is my evaluate.lua code
-- load the required lua models
torch = require("torch")
nn = require("nn")
gm = require("image")
cutorch = require "cutorch"
cunn = require "cunn"
cjson = require("cjson")
-- model
modelPath='./model.t7'
ninputs = 3
model = nn.Sequential()
model:add(nn.Linear(ninputs,2))
-- let’s save a dummy model
-- to demonstrate the functionality
torch.save(modelPath, model)
-- load a pretrained model
net = torch.load(modelPath)
net:cuda()
net:evaluate()
-- our end point function
-- this function is called by the ngx server
-- accepts a json body
function FeedForward(json)
print("starting")
-- decode and extract field x
local data = cjson.decode(json)
local input = torch.Tensor(data.x)
local response = {}
-- example checks
if input == nil then
print("No input given")
elseif input:size(1) ~=ninputs then
print("Wrong input size")
else
-- evaluat the input and create a response
local output = net:forward(input:cuda()):float()
-- from tensor to table
for i=1,output:size(1) do
response[i] = output[i]
end
end
-- return response
return response
end
I am trying to run the model using following commands
/usr/local/openresty/nginx/sbin/nginx -p "$(pwd)" -c "nginx.conf"
It's starting up fine but when I am sending curl request like
curl -H "Content-Type: application/json" -X POST -d '{"x":[1,2,3]}' http://localhost:6788/ff
I am getting following error.
2016/09/29 12:59:59 [notice] 10355#0: *1 [lua] evaluate.lua:28: FeedForward(): starting, client: 127.0.0.1, server: , request: "POST /ff HTTP/1.1", host: "localhost:6788"
THCudaCheck FAIL file=/tmp/luarocks_cutorch-scm-1-700/cutorch/lib/THC/generic/THCStorage.cu line=40 error=3 : initialization error
2016/09/29 12:59:59 [error] 10355#0: *1 lua entry thread aborted: runtime error: unknown reason
stack traceback:
coroutine 0:
[C]: in function 'resize'
/home/ubuntu/torch/install/share/lua/5.1/cutorch/Tensor.lua:14: in function 'cuda'
/rootpath/evaluate.lua:41: in function 'FeedForward'
content_by_lua(nginx.conf:31):7: in function <content_by_lua(nginx.conf:31):1>, client: 127.0.0.1, server: , request: "POST /ff HTTP/1.1", host: "localhost:6788"
Without cutorch the model is running fine like If I remove
net:cuda()
and replace line
local output = net:forward(input:cuda()):float()
by
local output = net:forward(input):float()
It's working fine. Also I tried to run evaluate.lua using th and it's working fine there with cutorch and cunn packages.

How to use python requests to PUT a pmml model to openscoring

I'm trying to use python requests to PUT a .pmml model to a local openscoring server.
This works (from directory containing DecisionTreeIris.pmml):
curl -X PUT --data-binary #DecisionTreeIris.pmml -H "Content-type: text/xml" http://localhost:8080/openscoring/model/DecisionTreeIris
This doesn't:
import requests
file = '/Users/weitzenfeld/IntelliJProjects/openscoring/openscoring-server/etc/DecisionTreeIris.pmml'
r = requests.put('http://localhost:8080/openscoring/model/DecisionTreeIris', files={'file': open(file, 'rb')})
r.text
returns:
u'<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>\n<title>Error 415 </title>\n</head>\n<body>\n<h2>HTTP ERROR: 415</h2>\n<p>Problem accessing /openscoring/model/DecisionTreeIris. Reason:\n<pre> Unsupported Media Type</pre></p>\n<hr /><i><small>Powered by Jetty://</small></i>\n</body>\n</html>\n'
I also tried:
r = requests.put('http://localhost:8080/openscoring/model/DecisionTreeIris', files={'file': open(file, 'rb')}, headers={'Content-type': 'text/xml', 'Accept': 'text/xml'})
r.text
which returns:
u'<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>\n<title>Error 406 </title>\n</head>\n<body>\n<h2>HTTP ERROR: 406</h2>\n<p>Problem accessing /openscoring/model/DecisionTreeIris. Reason:\n<pre> Not Acceptable</pre></p>\n<hr /><i><small>Powered by Jetty://</small></i>\n</body>\n</html>\n'
Note that my python attempt is the same as in the accepted answer to this question: Using Python to PUT PMML.
Also, someone with >1500 rep should consider making an 'openscoring' tag.
You should check the annotations of the method org.openscoring.service.ModelResource#deploy(String, HttpServletRequest) for valid request/response MIME types.
The first request fails because the server only accepts application/xml and text/xml payloads. The second request fails, because the server emits application/json payloads, but your client is only willing to accept text/xml payloads.
Solution was to put data, not the file handler:
r = requests.put('http://localhost:8080/openscoring/model/DecisionTreeIris', data=open(file, 'rb'), headers={'Content-type': 'text/xml', 'Accept': 'text/xml'})

Determine difference in http request between Python2 and Python3

I am attempting to use Python3 to send metrics to Hosted Graphite. The examples given on the site are Python2, and I have successfully ported the TCP and UDP examples to Python3 (despite my inexperience, and have submitted the examples so the docs may be updated), however I have been unable to get the HTTP method to work.
The Python2 example looks like this:
import urllib2, base64
url = "https://hostedgraphite.com/api/v1/sink"
api_key = "YOUR-API-KEY"
request = urllib2.Request(url, "foo 1.2")
request.add_header("Authorization", "Basic %s" % base64.encodestring(api_key).strip())
result = urllib2.urlopen(request)
This works successfully, returning a HTTP 200.
So far I have ported this much to Python3, and while I was (finally) able to get it to make a valid HTTP request (i.e. no syntax errors), the request fails, returning HTTP 400
import urllib.request, base64
url = "https://hostedgraphite.com/api/v1/sink"
api_key = b'YOUR-API-KEY'
metric = "testing.python3.http 1".encode('utf-8')
request = urllib.request.Request(url, metric)
request.add_header("Authorization", "Basic %s" % base64.encodestring(api_key).strip())
result = urllib.request.urlopen(request)
The full result is:
>>> result = urllib.request.urlopen(request)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/Cellar/python3/3.3.1/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 160, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/Cellar/python3/3.3.1/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 479, in open
response = meth(req, response)
File "/usr/local/Cellar/python3/3.3.1/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 591, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/Cellar/python3/3.3.1/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 517, in error
return self._call_chain(*args)
File "/usr/local/Cellar/python3/3.3.1/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 451, in _call_chain
result = func(*args)
File "/usr/local/Cellar/python3/3.3.1/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 599, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
Is it obvious what I am doing wrong? Are there any suggestions on how I might capture and compare what the successful (python2) and failing (python3) requests are actually sending?
Don't mix Unicode strings and bytes:
>>> "abc %s" % b"def"
"abc b'def'"
You could construct the header as follows:
from base64 import b64encode
headers = {'Authorization': b'Basic ' + b64encode(api_key)}
A quick way to see the request is to change the host in the url to localhost:8888 and run before making the request:
$ nc -l 8888
You could also use wireshark to see the requests.

Resources