Ruby http Get returning truncated response - http

I tried sending a HTTP Get to a device's Restful API via Postman and it worked fine returning all text I was expecting. Postman suggested Ruby code for the request was the following:
url = URI('http://192.168.1.5/rest/op/BD1FD3D893613E79')
http = Net::HTTP.new(url.host, url.port)
request = Net::HTTP::Get.new(url)
request.basic_auth 'admin', 'admin'
request["accept"] = 'Application/json'
response = http.request(request)
puts response.read_body
but when I tried that in my code, it is returning a truncated response (missing lines) and I have to resend the same Get multiple times to get the entire text response response.
Is there anything missing in the Ruby code above that's causing this truncated response?
Update 1
I tried this
url = URI('http://192.168.1.5/rest/op/BD1FD3D893613E79')
http = Net::HTTP.new(url.host, url.port)
request = Net::HTTP::Get.new(url)
request.basic_auth 'admin', 'admin'
request["accept"] = 'Application/json'
response = http.request(request)
puts response.read_body
response.read_body do |segment|
puts segment.to_s
end
and that generated this error
IOError (Net::HTTPOK#read_body called twice):
Update 2
I tried this
1073 url = URI('http://192.168.1.5/rest/op/BD1FD3D893613E79')
1074 http = Net::HTTP.new(url.host, url.port)
1075 request = Net::HTTP::Get.new(url.to_s)
1076 request.basic_auth 'admin', 'admin'
1077 request["accept"] = 'Application/json'
1078 response = http.request(request)
1079 response.read_body do |segment|
1080 puts segment.to_s
1081 end
and got this error
IOError (Net::HTTPOK#read_body called twice):
app/controllers/Apps_controller.rb:1079:in `block in get_config'
app/controllers/Apps_controller.rb:1045:in `each'
app/controllers/Apps_controller.rb:1045:in `get_config'

Based on: http://ruby-doc.org/stdlib-1.9.3/libdoc/net/http/rdoc/Net/HTTPResponse.html#method-i-read_body
read_body returns the body as stream, so you should iterate it, something like:
response.read_body do |segment|
puts segment
end
If you want to get the full body just use: response.body

Looks very similar....
We are getting truncated responses when pulling a set of artifacts (zip files) from a Bamboo server and writing to local storage. It happens inconsistently, on only one of 8 servers, and not always the same file. Seems always to be the last portion of the file that is missing.
Using Ruby 2.0.0p598 and net/http.
The zip files range in size from 2 mb to 34 mb.
While we haven't figured out why the response body has been truncated, our workaround is to compare the expected content length (in the response header), to the response body size. If they don't match, try it again. Example code:
package = "packages/applicationname.zip"
def retrieve(package)
# generates the Bamboo url for the given package
sourceURL = package_url package
expectedLength = 0
bodyLength = 1
# Try to catch the bad download and re-issue the GET request,
while expectedLength != bodyLength do
# add a counter if worried about getting stuck
# issue the request to get the current package zip file
response = my_request_wrapper sourceURL
expectedLength = response['content-length'].to_i
theBody = response.body
bodyLength = theBody.size
if expectedLength != bodyLength then
puts "!! SIZE MISMATCH !!"
else
# the response body is good, process as needed
open package, 'wb' do |io|
io.write theBody
end
end
end
end
def my_request_wrapper(url)
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
req = Net::HTTP::Get.new(uri.request_uri)
req.basic_auth("user", "pwd")
return http.request req
end

I was using OSX Sierra, rvm, and ruby 2.3.0
Updating my rvm, ruby, and gems seems to have fixed the problem for me

I think you must call http.start before you call http.request.
Not sure why, but I see the same.

Related

Passing files in POST using Requests in Python

I'm getting below error when making a requests call post method
{'detail': [{'loc': ['body', 'files'], 'msg': 'field required', 'type': 'value_error.missing'}]}
I tried
response = requests.post("url",headers={mytoken},params=p,files=files)
files = { "file 1": open("sample.pdf",'rb'), "file 2":open("sample11.pdf",'rb')}
I want to get 200 status but I'm getting 422 validation error. Any Idea Why? Its for API Testing purpose, Im new to this I've been debugging this for whole day but still couldn't figure out.
It is not clear from the question what kind of a request the server is expecting. Also, its not clear the exact code snippet you are using too.
From the question, the snippet looks like as follows,
response = requests.post("url",headers={mytoken},params=p,files=files)
files = { "file 1": open("sample.pdf",'rb'), "file 2":open("sample11.pdf",'rb')}
if so, that means you are reading files after you send the request, may be thats why the server complained about missing files field.
See the below example on how you can send two files to an endpoint expecting files.
import requests
import logging
logger = logging.getLogger(__name__)
post_url = "https://exampledomain.local/upload"
file1 = open("sample1.pdf", "rb")
file2 = open("sample2.pdf", "rb")
files = {"file1": file1, "file2": file2}
headers = {"Authorization": "Bearer <your_token_here>"}
params = {"file_type": "pdf"}
response = requests.post(post_url, files=files, headers=headers, params=params)
file1.close()
file2.close()
logger.debug(response.text)

Python requests post: data and json

I used the following Python code to retrieve a web page behind a login page successfully for some years:
username = 'user'
password = 'pass'
login_url = 'https://company.com/login?url='
redirect_url = 'https://epaper.company.com/'
data = { 'email' : username, 'pass' : password }
initial_url = login_url + quote(redirect_url)
response = requests.post(initial_url, data=data)
Then something changed at company.com about 2 months ago, and the request returned status code 400. I tried changing the data parameter to json (response = requests.post(initial_url, json=data)) which gave me a 200 response telling me a wrong password was provided.
Any ideas what I could try to debug?
Thanks,
Jan
Update: I just tried using a requests session to retrieve the csrf_token from the login page (as suggested here), so now my code reads:
with requests.Session() as sess:
response = sess.get(login_url)
signin = BeautifulSoup(response._content, 'html.parser')
data['csrf_token'] = signin.find('input', {'name':'csrf_token'})['value']
response = sess.post(initial_url, data=data)
Unfortunately, the response is still 400 (and 200/wrong password with the json parameter).
First: When you send data=data, used {"Content-Type":"application/x-www-form-urlencoded"}; if you send json=data, in headers response should be used {"Content-Type":"application/json"}
Second: Perhaps redirects have been added. Try to add:
response = sess.post(url, data=data)
print("URL you expect", url)
print("Last request URL:", response.url)
Be sure to check:
print(sess.cookies.get_dict())
print(response.headers)
If you get an unexpected result when checking, change the code like this:
response = sess.post(url, data=data, allow_redirects=False)

Scrapy handle 301/302 response code as well as follow the target url

I am using scrapy version 1.0.5 for implementation of a crawler. Currently I have set REDIRECT_ENABLED = False and handle_httpstatus_list = [500, 301, 302] to scrape the pages with 301 and 302 responses. However, since REDIRECT_ENABLED is set to False, the spider doesn't goes to the target url in Location response header. How can I achieve this ?
It is a long tome since I did anything like this but you need to generate a request object with url, meta and callback parameters.
But I seem to recall you can do it along the lines of:
def parse(self,response):
# do whatever you need to do .... then
if response.status in [301, 302] and 'Location' in response.headers:
# test to see if it is an absolute or relative URL
newurl = urljoin(request.url, response.headers['location'])
# or
newurl = response.headers['location']
yield Request(url = newurl, meta = request.meta, callback=self.parse_whatever)

How do I inject new request header with json data in proxy request flow?

I am trying to inject a new request header in the proxy request flow using JS policy to be sent to the backend server. When I look at the debug trace, I see that the json data in the request header is distorted.
I am trying to inject some string like
{"scope":"","time_till":2264,"id_1":"hUXLXVqpA1J4vA9sayk2UttWNdM","custom_data":{"c_id":"test_data"}}
But when I look at the trace window I see this
{"scope":"","time_till":2264,id_1":"hUXLXVqpA1J4vA9sayk2UttWNdM,"custom_data":{"c_id":"test_data"}}
what am I doing wrong?
var obj = {"scope":"","time_till":2264,"id_1":"hUXLXVqpA1J4vA9sayk2UttWNdM","custom_data":{"c_id":"test_data"}};
var header_str = JSON.stringify(obj);
context.setVariable('json-header',header_str);
request.headers['x-json-hedar']= header_str;
I tested your code and it seems to work. Here's an example response where I set the header string as a response:
HTTP/1.1 200 OK
User-Agent: curl/7.30.0
Accept: */*
x-json-header: {"scope":"","time_till":2264,"id_1":"hUXLXVqpA1J4vA9sayk2UttWNdM","custom_data":{"c_id":"test_data"}}
Content-Length: 0
It appears this is only an issue with the Apigee debug session / trace tool as the header value was set correctly. Here was the JSON download of the debug session showing this header value:
{
"name": "x-json-header",
"value": "{\"scope\":\"\",\"time_till\":2264,id_1\":\"hUXLXVqpA1J4vA9sayk2UttWNdM,\"custom_data\":{\"c_id\":\"test_data\"}}"
}
You can see that the value passed to the UI for displaying the debug info has the malformed json:
id_1\":\"hUXLXVqpA1J4vA9sayk2UttWNdM,
This does not appear to be a problem with the Apigee debug/trace UI. I see the malformed JSON trickle down to my backend service.
Here is the header I'm trying to send -
{"timeStamp":"2349218349381274","latitude":"34.589","longitude":"-37.343","clientIp":"127.0.0.0","deviceId":"MOBILE_TEST_DEVICE_AGAIN","macAddress":"23:45:345:345","deviceType":"phone","deviceOS":"iOS","deviceModel":"iPhone 5S","connection":"5G","carrier":"Vodafone","refererURL":"http://www.google.com","xforwardedFor":"129.0.0.0","sessionId":"kfkls498327ksdjf","application":"mobile-app","appVersion":"7.6.5","serviceVersion":"1.0","userAgent":"Gecko"}
But Apigee reads the header as below. Note the missing start quotes from some fields.
{"timeStamp":"2349218349381274",latitude":"34.589,longitude":"-37.343,clientIp":"127.0.0.0,deviceId":"MOBILE_TEST_DEVICE_AGAIN,macAddress":"23:45:345:345,deviceType":"phone,deviceOS":"iOS,deviceModel":"iPhone 5S,connection":"5G,carrier":"Vodafone,refererURL":"http://www.google.com,xforwardedFor":"129.0.0.0,sessionId":"kfkls498327ksdjf,application":"mobile-app,appVersion":"7.6.5,serviceVersion":"1.0,"userAgent":"Gecko"}
The header is used in a service callout to a backend service which parses it. And rightly so, I get the below error -
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('l' (code 108)): was expecting double-quote to start field name
at [Source: java.io.StringReader#22549cdc; line: 1, column: 35]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1378)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:599)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:520)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleUnusualFieldName(ReaderBasedJsonParser.java:1275)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._parseFieldName(ReaderBasedJsonParser.java:1170)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:611)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:301)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:121)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:2796)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1942)
I encounter strange behaviour when adding JSON to a context variable for example like the following:
var header_str = JSON.stringify(obj);
context.setVariable('json-header',header_str);
I appreciate this is an example so you may not have included the full extent of the problem but this normally works (now it is not added to a variable first):
request.headers['x-json-header'] = JSON.stringify(obj);
Code like this also works if you can send the request from JavaScript
var headers = {"Accept": "application/json", "Accept-Language": "en"};
var sessionRequest = new Request(url, 'POST', headers, body);
var exchange = httpClient.send(sessionRequest);
exchange.waitForComplete()
if (exchange.isSuccess()){
var responseObj = exchange.getResponse().content.asJSON;
if (responseObj.error){
request.content += JSON.stringify(responseObj);
}
}
Also, I have had success with using an AssignMessage policy to build a request, followed by a Callout policy to read the stored request and then make that request and store the result in a response object which can then be read by an Extract Variables policy.

How to safely handle raw (file) data in Java?

An image gets corrupted while being retrieved (through HTTP) and then sent (through HTTP) to a database. Image's raw data is handled in String form.
The service sends a GET for an image file, receives response with the raw image data (response's body) and the Content-Type. Then, a PUT request is sent with the aforementioned request's body and Content-Type header. (The PUT request is constructed by providing the body in String) This PUT request is sent to a RESTful database (CouchDB), creating an attachment (for those unfamiliar with CouchDB an attachment acts like a static file).
Now I have the original image, which my service GETs and PUTs to a database, and this 'copy' of the original image, that I can now GET from the database. If I then `curl --head -v "[copy's url]" it has the Content-Type of the original image, but Content-Length has changed, went from 200kb to about 400kb. If I GET the 'copy' image with a browser, it is not rendered, whereas, the original renders fine. It is corrupted.
What might be the cause? My guess is that while handling the raw data as a string, my framework guesses the encoding wrong and corrupts it. I have not been able to confirm or deny this. How could I handle this raw data/request body in a safe manner, or how could I properly handle the encoding (if that proves to be the problem)?
Details: Play2 Framework's HTTP client, Scala. Below a test to reproduce:
"able to copy an image" in {
def waitFor[T](future:Future[T]):T = { // to bypass futures
Await.result(future, Duration(10000, "millis"))
}
val originalImageUrl = "http://laughingsquid.com/wp-content/uploads/grumpy-cat.jpg"
val couchdbUrl = "http://admin:admin#localhost:5984/testdb"
val getOriginal:ws.Response = waitFor(WS.url(originalImageUrl).get)
getOriginal.status mustEqual 200
val rawImage:String = getOriginal.body
val originalContentType = getOriginal.header("Content-Type").get
// need an empty doc to have something to attach the attachment to
val emptyDocUrl = couchdbUrl + "/empty_doc"
val putEmptyDoc:ws.Response = waitFor(WS.url(emptyDocUrl).put("{}"))
putEmptyDoc.status mustEqual 201
//uploading an attachment will require the doc's revision
val emptyDocRev = (putEmptyDoc.json \ "rev").as[String]
// create actual attachment/static file
val attachmentUrl = emptyDocUrl + "/0"
val putAttachment:ws.Response = waitFor(WS.url(attachmentUrl)
.withHeaders(("If-Match", emptyDocRev), ("Content-Type", originalContentType))
.put(rawImage))
putAttachment.status mustEqual 201
// retrieve attachment
val getAttachment:ws.Response = waitFor(WS.url(attachmentUrl).get)
getAttachment.status mustEqual 200
val attachmentContentType = getAttachment.header("Content-Type").get
originalContentType mustEqual attachmentContentType
val originalAndCopyMatch = getOriginal.body == getAttachment.body
originalAndCopyMatch aka "original matches copy" must beTrue // << false
}
Fails at the last 'must':
[error] x able to copy an image
[error] original matches copy is false (ApplicationSpec.scala:112)
The conversion to String is definitely going to cause problems. You need to work with the bytes as Daniel mentioned.
Looking at the source it looks like ws.Response is just a wrapper. If you get to the underlying class then there are some methods that may help you. On the Java side, someone made a commit on GitHub to expose more ways of getting the response data other than a String.
I'm not familiar with scala but something like this may work:
getOriginal.getAHCResponse.getResponseBodyAsBytes
// instead of getOriginal.body
WS.scala
https://github.com/playframework/playframework/blob/master/framework/src/play/src/main/scala/play/api/libs/ws/WS.scala
WS.java
Here you can see that Response has some new methods, getBodyAsStream() and asByteArray.
https://github.com/playframework/playframework/blob/master/framework/src/play-java/src/main/java/play/libs/WS.java

Resources