I notice that when I update a feed (set a new value) and then read the feed, the data returned does not always include my last update. It's as if Xively is returning a cached response. Eventually (after 5 minutes or so) a request will result in the new value will be returned.
This is a development-mode device; I can see the new value reflected in the Xively status page for the device immediately following the update. I've also noticed that the requests resulting in out-of-date data do not appear in the Xively Request Log. After 5 minutes or so of sending requests and getting out-of-date responses a request will appear in the Request Log and an up-to-date response will be returned.
I update my feed like this:
curl -X PUT -d#feedupdate.json -H X-ApiKey:my_key http://api.xively.com/v2/feeds/feed_id.json
where feedupdate.json contains:
{
"version": "1.0.0",
"datastreams": [
{
"id": "stream_id",
"current_value": "202"
}
]
}
And I request the feed status like this:
curl -o feedrange.json -H Cache-control:no-cache -H Pragma:no-cache -H X-ApiKey:my_key http://api.xively.com/v2/feeds/feed_id.xml?start=2014-01-13T10:00:00Z
TIA.
Nothing has changed on the Xively side as far as know.
That being said, I also noticed this problem recently. However, I was not able to reproduce it evenly. Try going to the feed in your browser, is it still cached? Try on a different internet connection, is it still cached?
My solution was to set the cache-control: no-cache header on the device, and this immediately took care of the problem for me. This is what I was going to recommend to you but it appears you are already passing that as a header.
I would recommend looking around at the other options you can set in the cache-control header and experiment with those. As well as try other devices and internet connections to see if you can figure out where it is being cached.
It would also be help me help you if you provided some info about what your setup is. Language, hardware, internet setup, etc.
Related
I'm trying to receive data from a website using Arduino and Sim5216 3g board.
The AT+CHTTPSSEND requires the following information (Taken from the example in the SIM5216 manual)
"AT+CMMSSEND=88"
">GET / HTTP/1.1 "
"Host:www.mywebsite.com"
"User-Agent: MY WEB AGENT "
"Content-Length: 0"
What is the "User-Agent"? From what I searched, it represents the browser you are using. I'm inclined to leave it blank, however when I do that, once I enter the command with the other info filled out, I get no response. The sim5216 board doesn't respond anymore and I have to press the reset button to get any further responses from it (even if I type AT, I don't get an ok unless I reset the board).
Any comments/help is much appreciated.
Thanks,
Check my tutorial HTTPS GET request with SIM5215E!
Just don't care about User-Agent for now. It represents browser you are using, but it doesn't matter for success of GET request.
I'm a student and I'm taking my first networking class. I'm working on an assignment designed to get me used to using Wireshark and understanding packet transfers. Part of the assignment is to collect some data about a certain GET request, but my Wireshark isn't showing anything related to GET or POST requests.
I've discussed this with my instructor and he can't figure it out, either. I've tried uninstalling/reinstalling Wireshark, but haven't gotten anything different.
Here's what I'm getting when I should be getting GET data:
26030 1157.859131000 128.119.245.12 10.0.0.7 HTTP 564 HTTP/1.1 404 Not Found (text/html)
This is the first packet I get after connecting to the server (this comes from right-click "copy"). From what I've gathered from the assignment instructions and the instructor, this should get a GET request. Anyone have any ideas?
You can use a browser plugin like firebug to examine actual request and response headers being exchanged. Sometimes due to page caching actual document may not be refetched after only headers like if modified since since being exchanged or the browsers cached version has not expired.
maybe you guys can help me with this. I am trying to implement
reCAPTCHA in my node.js application and no matter what I do, I keep
getting "invalid-site-private-key" as a response.
Here are the things I double and double checked and tried:
Correct Keys
Keys are not swapped
Keys are "global keys" as I am testing on localhost and thought it might be an issue with that
Tested in production environment on the server - same problem
The last thing I can think of is that my POST request to the reCAPTCHA
API itself is incorrect as the concrete format of the body is not
explicitly documented (the parameters are documented, I know). So this
is the request body I am currently sending (the key and IP is changed
but I checked them on my side):
privatekey=6LcHN8gSAABAAEt_gKsSwfuSfsam9ebhPJa8w_EV&remoteip=10.92.165.132& challenge=03AHJ_Vuu85MroKzagMlXq_trMemw4hKSP648MOf1JCua9W-5R968i2pPjE0jjDGX TYmWNjaqUXTGJOyMO3IKKOGtkeg_Xnn2UVAfoXHVQ-0VCHYPNwrj3PQgGj22EFv7RGSsuNfJCyn mwTO8TnwZZMRjHFrsglar2zQ&response=Coleshill areacce
Is there something wrong with this format? Do I have to send special
headers? Am I completely wrong? (I am working for 16 hours straight
now so this might be ..)
Thank you for your help!
As stated in the comments above, I was able to solve the problem myself with the help of broofa and the node-recaptcha module available at https://github.com/mirhampt/node-recaptcha.
But first, to complete the missing details from above:
I didn't use any module, my solution is completely self-written based on the documentation available at the reCAPTCHA website.
I didn't send any request headers as there was nothing stated in the documentation. Everything that is said concerning the request before they explain the necessary parameters is the following:
"After your page is successfully displaying reCAPTCHA, you need to configure your form to check whether the answers entered by the users are correct. This is achieved by doing a POST request to http://www.google.com/recaptcha/api/verify. Below are the relevant parameters."
-- "How to Check the User's Answer" at http://code.google.com/apis/recaptcha/docs/verify.html
So I built a querystring myself (which is a one-liner but there is a module for that as well as I learned now) containing all parameters and sent it to the reCAPTCHA API endpoint. All I received was the error code invalid-site-private-key, which actually (as we know by now) is a wrong way of really sending a 400 Bad Request. Maybe they should think about implementing this then people would not wonder what's wrong with their keys.
These are the header parameters which are obviously necessary (they imply you're sending a form):
Content-Length which has to be the length of the query string
Content-Type which has to be application/x-www-form-urlencoded
Another thing I learned from the node-recaptcha module is, that one should send the querystring utf8 encoded.
My solution now looks like this, you may use it or built up on it but error handling is not implemented yet. And it's written in CoffeeScript.
http = require 'http'
module.exports.check = (remoteip, challenge, response, callback) ->
privatekey = 'placeyourprivatekeyhere'
request_body = "privatekey=#{privatekey}&remoteip=#{remoteip}&challenge=#{challenge}&response=#{response}"
response_body = ''
options =
host: 'www.google.com'
port: 80
method: 'POST'
path: '/recaptcha/api/verify'
req = http.request options, (res) ->
res.setEncoding 'utf8'
res.on 'data', (chunk) ->
response_body += chunk
res.on 'end', () ->
callback response_body.substring(0,4) == 'true'
req.setHeader 'Content-Length', request_body.length
req.setHeader 'Content-Type', 'application/x-www-form-urlencoded'
req.write request_body, 'utf8'
req.end()
Thank you :)
+1 to #florian for the very helpful answer. For posterity, I thought I'd provide some information about how to verify what your captcha request looks like to help you make sure that the appropriate headers and parameters are being specified.
If you are on a Mac or a Linux machine or have access to one of these locally, you can use the netcat command to setup a quick server. I guess there are netcat windows ports but I have no experience with them.
nc -l 8100
This command creates a TCP socket listening on pot 8100 and will wait for a connection. You then can change the captcha verify URL from http://www.google.com/recaptcha/... in your server code to be http://localhost:8100/. When your code makes the POST to the verify URL you should see your request outputted to the scree by netcat:
POST / HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Content-Length: 277
Host: localhost:8100
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1 (java 1.5)
privatekey=XXX&remoteip=127.0.0.1&challenge=03AHJYYY...&response=some+words
Using this, I was able to see that my private-key was corrupted.
I need to test some client application code I've written to test its' handling of various status codes returned in an HTTP response from a web server.
I have Fiddler 2 (Web Debugging Proxy) installed and I believe there's a way to modify responses using this application, but I'm struggling to find out how. This would be the most convenient way, as it would allow me to leave both client and server code unmodified.
Can anyone assist as I'd like to intercept the HTTP response being sent from server to client and modify the status code before it reaches the client?
Any advice would be much appreciated.
Ok, so I assume that you're already able to monitor your client/server traffic. What you want to do is set a breakpoint on the response then fiddle with it before sending it on to the client.
Here are a couple of different ways to do that:
Rules > Automatic Breakpoints > After Responses
In the quickexec box (the black box at the bottom) type "bpafter yourpage.svc". Now Fiddler will stop at a breakpoint before all requests to any URL that contains "yourpage.svc". Type "bpafter" with no parameters to clear the breakpoint.
Programmatically tamper with the response using FiddlerScript. The best documentation for FiddlerScript is on the official site: http://www.fiddler2.com/Fiddler/dev/
Once you've got a response stopped at the breakpoint, just double click it to open it in the inspectors. You've got a couple of options now:
Right next to the green Run to Completion button (which you click to send the response) there's a dropdown that lets you choose some default response types.
Or, on the Headers inspector, change the response code & message in the textbox at the top.
Or, click the "Raw" inspector and mess with the raw response to do arbitrary things to it. Also a good way to see what your client does when it gets a malformed response, which you'll probably test accidentally :)
Another alternative is to use Fiddler's AutoResponder tab (on the right-hand panel). This allows you to catch a request to any URI that matches a string and serve a "canned" response from a file. The file can contain both headers and payload. The advantage of this approach is that you don't have to write FiddlerScript and you don't have to handle each request manually via a breakpoint.
You would set the rule up in Fiddler like shown below (ensure you enable unmatched requests passthrough otherwise all other http requests will fail).
In this example, any request whose URI includes "fooBar" will get the canned response. The format of the file will vary depending on your APIs (you can use your browser to intercept a "real" response and base it on that) but mine looked like the following:
HTTP/1.1 409 Conflict
Server: Apache-Coyote/1.1
X-Powered-By: Servlet 2.5; JBoss-5.0/JBossWeb-2.1
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, DELETE, PUT, PATCH, OPTIONS
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, Authorization
Access-Control-Max-Age: 86400
Content-Type: application/vnd.api+json
Content-Length: 149
Date: Tue, 28 Mar 2017 10:03:29 GMT
{"errors":[{"code":"OutOfStock","detail":"Item not in stock","source":{"lineId":{"type":"Order line Number","id":"1"}},"meta":{"availableStock":0}}]}
I found that it needed a carriage return at the end of the last line (i.e. after the json), and that the Content-Length header had to match the number of characters in the json, otherwise the webapp would hang. Your mileage may vary.
Create a FiddlerScript rule. Here's what I used in order to generate a local copy of a website that was intentionally using 403 on every page to thwart HTTrack/WGET.
https://gist.github.com/JamoCA/22db8d68a9a2fb20cb04a85360185333
/* 20180615 Fiddler rule to ignore all 403 HTTP Status errors so WGET or HTTrack can generate local copy of remote website */
SCENARIO: Changing the user agent or setting a delay isn't enough and the entire remote server is configured to respond w/403.
CONFIGURE: Add below rule to FiddlerScript OnBeforeReponse() section. Configure HTTrack/WGET/CRON to use proxy 127.0.0.01:8888 */
static function OnBeforeResponse(oSession: Session) {
if (oSession.HostnameIs("TARGETHOSTNAME_FILTER.com") && oSession.responseCode == 403) {
oSession.responseCode = 200;
oSession.oResponse.headers.HTTPResponseCode = 200;
oSession.oResponse.headers.HTTPResponseStatus = "200 OK";
}
}
I'm currently working on a project that needs to request a url multiple times. Having studied the the HTTP Proxy (Charles) it seems that AIR will cache the first response and then return the same response for each subsequent request.
Does anybody know how to know if the response has been cached other than setting the URLRequest to useCache, but this doesn't say if the response was a cached response or not. The digest isn't set on the URLRequest either, although it does mention this is for swz only, so how does it know if the content is the current content or not? Is the responseHeaders used to find out how long to hold the cache i.e.
Cache-Control: max-age=900
Also does anyone know how to flush/purge the cache or are we at the whim of the GC and in that case how does it know if to leave it in the cache or now?
This makes sense to me, but still I would like to know how to regulate this cache.
Further more: I've tested a set up where parallel URLLoaders (10) are made and created which open the same url to see what happens in that instance. It seems that each parallel request is made until a successful response is given, all subsequent calls are then cached. Calls which are sent out before the successful request is then completed. It looks like the items which are already in being processed do not use the cache and return with correct data.
Additional The AIR runtime doesn't even send a "If-Modified-Since" header, so the cache isn't even honoring HTTP protocol. So it seems as if Adobe has implemented it's own version of a cache which doesn't even use HTTP/1.1 Header Field Definitions. Perfect.
Thanks for any help.
Simon
From the documentation of URLRequest class, it seems that it uses Operating System's HTTP Cache. On a windows 7 OS, it seems that it is using IE's cache.
You can use a HTTP monitor tool like Fiddler, and verify this.
First request is 200, and subsequent requests are 304. After clearing the IE cache. and run the application again, You can see that it results in a HTTP 200 status.