I'm writing a flex application that polls an xml file on the server to check for updated data every few seconds, and I'm having trouble preventing it from caching the data and failing to respond to it being updated.
I've attempted to set headers using the IIS control panel to use the following, without any luck:
CacheControl: no-cache
Pragma: no-cache
I've also attempted adding a random HTTP GET parameter to the end of the request URL, but that seems like it's stripped off by the HttpService class before the request is made. Here's the code to implement it:
http.url = "test.xml?time=" + new Date().getMilliseconds();
And here's the debug log that makes me think it failed:
(mx.messaging.messages::HTTPRequestMessage)#0
body = (Object)#1
clientId = (null)
contentType = "application/x-www-form-urlencoded"
destination = "DefaultHTTP"
headers = (Object)#2
httpHeaders = (Object)#3
messageId = "AAB04A17-8CB3-4175-7976-36C347B558BE"
method = "GET"
recordHeaders = false
timestamp = 0
timeToLive = 0
url = "test.xml"
Has anyone dealt with this problem?
The cache control HTTP header is "Cache-Control" ... note the hyphen! It should do the trick. If you leave out the hyphen, it is not likely to work.
I used the getTime() to make the date into a numeric string that did the trick. I also changed GET to POST. There were some issues with different file extensions being cached differently. For instance, a standard dynamic extension like .php or .jsp might not be cached by the browser and
private var myDate:Date = new Date();
[Bindable]
private var fileURLString:String = "http://www.mysite.com/data.txt?" + myDate.getTime();
Hopefully this helps someone.
I also threw a ton of the header parameters at it but they never fully did the trick. Examples:
// HTTPService called service
service.headers["Pragma"] = "no-cache"; // no caching of the file
service.headers["Cache-Control"] = "no-cache";
Related
I'm creating client side Blazor app with Microsoft.OData.Client. When I create new entity like this:
var dataServiceContext = this.ClientFactory.CreateClient<Container>(new Uri("http://localhost:5000/odata"));
var newAsset = new CreateAssetDto()
{
TechnicalName = "from_client_4",
DisplayNameFormat = "format from client",
Icon = "client/icon",
InheritedFrom = Guid.NewGuid(),
IsActive = true,
Translation = new AssetTranslationDto
{
Title = "Client Asset",
Language = "en",
Description = "This is asset from client"
}
};
dataServiceContext.AddToAssets(newAsset);
await dataServiceContext.SaveChangesAsync();
I get an exception stating that response to this POST request is missing Location header. When I run fiddle to see what's going on I can see that it actually made 2 requests.
The first request is POST but doesn't include the body and recieves 204 response.
The second request is the one that actually contains the data creating new Asset and response contains Location header as it should.
I guess OData Client is complaining about Location header missing in the response for the first request (since response for second request does contain the header). But why is it even making the first request?
Any idea how to deal with this problem?
It's possible that the first request is a preflight request sent by the browser. But normally CORS preflight requests are sent using OPTIONS method, not POST. So this case is curious.
I am a contributor to the project but do not have enough reputation to add comments here to get clarifications. Could you create an issue on https://github.com/OData/odata.net ?
I've created a Xamarin.Forms application and added a connected Odata Service.
Now when i try sending a request to the backend it will fail with the exception System.ArgumentException: The 'User-Agent' header must be modified using the appropriate property or method.
Parameter name: name.
What i've tried:
1: Handling the DataServiceContext_BuildingRequest and setting the header there using e.Headers["User-Agent"] = "xyz";
2: Setting the Content-Type header because i read in another thread somewhere that it helped (e.Headers.Add("Content-Type", "application/json"))
3: Setting the delegate of the DataServiceContext.Configurations.RequestPipeline.OnMessageCreating. The problem is the exception will be thrown as soon as i try to create a new instance of HttpWebRequestMessage passing in the args that it receives.
Pipeline delegate:
Configurations.RequestPipeline.OnMessageCreating = (args) =>
{
args.Headers["User-Agent"] = "xyz";
args.Headers.Add("Content-Type", "application/json");
args.Headers["Accept"] = "application/json";
var req = new HttpWebRequestMessage(args);
req.HttpWebRequest.ContentType = "application/json";
return req;
};
My suspicion regarding this is, that there was something changed in the underlying code that is used by the Odata HttpWebRequestMessage class which doesnt allow direct setting of the User-Agent header anymore.
Did anyone find a way around this?
I am trying to inject a new request header in the proxy request flow using JS policy to be sent to the backend server. When I look at the debug trace, I see that the json data in the request header is distorted.
I am trying to inject some string like
{"scope":"","time_till":2264,"id_1":"hUXLXVqpA1J4vA9sayk2UttWNdM","custom_data":{"c_id":"test_data"}}
But when I look at the trace window I see this
{"scope":"","time_till":2264,id_1":"hUXLXVqpA1J4vA9sayk2UttWNdM,"custom_data":{"c_id":"test_data"}}
what am I doing wrong?
var obj = {"scope":"","time_till":2264,"id_1":"hUXLXVqpA1J4vA9sayk2UttWNdM","custom_data":{"c_id":"test_data"}};
var header_str = JSON.stringify(obj);
context.setVariable('json-header',header_str);
request.headers['x-json-hedar']= header_str;
I tested your code and it seems to work. Here's an example response where I set the header string as a response:
HTTP/1.1 200 OK
User-Agent: curl/7.30.0
Accept: */*
x-json-header: {"scope":"","time_till":2264,"id_1":"hUXLXVqpA1J4vA9sayk2UttWNdM","custom_data":{"c_id":"test_data"}}
Content-Length: 0
It appears this is only an issue with the Apigee debug session / trace tool as the header value was set correctly. Here was the JSON download of the debug session showing this header value:
{
"name": "x-json-header",
"value": "{\"scope\":\"\",\"time_till\":2264,id_1\":\"hUXLXVqpA1J4vA9sayk2UttWNdM,\"custom_data\":{\"c_id\":\"test_data\"}}"
}
You can see that the value passed to the UI for displaying the debug info has the malformed json:
id_1\":\"hUXLXVqpA1J4vA9sayk2UttWNdM,
This does not appear to be a problem with the Apigee debug/trace UI. I see the malformed JSON trickle down to my backend service.
Here is the header I'm trying to send -
{"timeStamp":"2349218349381274","latitude":"34.589","longitude":"-37.343","clientIp":"127.0.0.0","deviceId":"MOBILE_TEST_DEVICE_AGAIN","macAddress":"23:45:345:345","deviceType":"phone","deviceOS":"iOS","deviceModel":"iPhone 5S","connection":"5G","carrier":"Vodafone","refererURL":"http://www.google.com","xforwardedFor":"129.0.0.0","sessionId":"kfkls498327ksdjf","application":"mobile-app","appVersion":"7.6.5","serviceVersion":"1.0","userAgent":"Gecko"}
But Apigee reads the header as below. Note the missing start quotes from some fields.
{"timeStamp":"2349218349381274",latitude":"34.589,longitude":"-37.343,clientIp":"127.0.0.0,deviceId":"MOBILE_TEST_DEVICE_AGAIN,macAddress":"23:45:345:345,deviceType":"phone,deviceOS":"iOS,deviceModel":"iPhone 5S,connection":"5G,carrier":"Vodafone,refererURL":"http://www.google.com,xforwardedFor":"129.0.0.0,sessionId":"kfkls498327ksdjf,application":"mobile-app,appVersion":"7.6.5,serviceVersion":"1.0,"userAgent":"Gecko"}
The header is used in a service callout to a backend service which parses it. And rightly so, I get the below error -
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('l' (code 108)): was expecting double-quote to start field name
at [Source: java.io.StringReader#22549cdc; line: 1, column: 35]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1378)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:599)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:520)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleUnusualFieldName(ReaderBasedJsonParser.java:1275)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._parseFieldName(ReaderBasedJsonParser.java:1170)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:611)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:301)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:121)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:2796)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1942)
I encounter strange behaviour when adding JSON to a context variable for example like the following:
var header_str = JSON.stringify(obj);
context.setVariable('json-header',header_str);
I appreciate this is an example so you may not have included the full extent of the problem but this normally works (now it is not added to a variable first):
request.headers['x-json-header'] = JSON.stringify(obj);
Code like this also works if you can send the request from JavaScript
var headers = {"Accept": "application/json", "Accept-Language": "en"};
var sessionRequest = new Request(url, 'POST', headers, body);
var exchange = httpClient.send(sessionRequest);
exchange.waitForComplete()
if (exchange.isSuccess()){
var responseObj = exchange.getResponse().content.asJSON;
if (responseObj.error){
request.content += JSON.stringify(responseObj);
}
}
Also, I have had success with using an AssignMessage policy to build a request, followed by a Callout policy to read the stored request and then make that request and store the result in a response object which can then be read by an Extract Variables policy.
I'm attempting to set the request header 'Referer' to spoof a request coming from another site. We need the ability test that a specific referrer is used, which returns a specific form to the user, otherwise an alternative form is given.
I can do this within poltergeist by:
page.driver.headers = {"Referer" => referer_string}
but I can't find the equivalent functionality for the selemium driver.
How can I set request headers in the capybara selenium driver?
Webdriver doesn't contain an API to do it. See issue 141 from Selenium tracker for more info. The title of the issue says that it's about response headers but it was decided that Selenium won't contain API for request headers in scope of this issue. Several issues about adding API to set request headers have been marked as duplicates: first, second, third.
Here are a couple of possibilities that I can propose:
Use another driver/library instead of selenium
Write a browser-specific plugin (or find an existing one) that allows you to add header for request.
Use browsermob-proxy or some other proxy.
I'd go with option 3 in most of cases. It's not hard.
Note that Ghostdriver has an API for it but it's not supported by other drivers.
For those people using Python, you may consider using Selenium Wire which can set request headers as well as provide you with the ability to inspect requests and responses.
from seleniumwire import webdriver # Import from seleniumwire
# Create a new instance of the Chrome driver (or Firefox)
driver = webdriver.Chrome()
# Create a request interceptor
def interceptor(request):
del request.headers['Referer'] # Delete the header first
request.headers['Referer'] = 'some_referer'
# Set the interceptor on the driver
driver.request_interceptor = interceptor
# All requests will now use 'some_referer' for the referer
driver.get('https://mysite')
Install with:
pip install selenium-wire
I had the same issue. I solved it downloading modify-headers firefox add-on and activate it with selenium.
The code in python is the following
fp = webdriver.FirefoxProfile()
path_modify_header = 'C:/xxxxxxx/modify_headers-0.7.1.1-fx.xpi'
fp.add_extension(path_modify_header)
fp.set_preference("modifyheaders.headers.count", 1)
fp.set_preference("modifyheaders.headers.action0", "Add")
fp.set_preference("modifyheaders.headers.name0", "Name_of_header") # Set here the name of the header
fp.set_preference("modifyheaders.headers.value0", "value_of_header") # Set here the value of the header
fp.set_preference("modifyheaders.headers.enabled0", True)
fp.set_preference("modifyheaders.config.active", True)
fp.set_preference("modifyheaders.config.alwaysOn", True)
driver = webdriver.Firefox(firefox_profile=fp)
Had the same issue today, except that I needed to set different referer per test. I ended up using a middleware and a class to pass headers to it. Thought I'd share (or maybe there's a cleaner solution?):
lib/request_headers.rb:
class CustomHeadersHelper
cattr_accessor :headers
end
class RequestHeaders
def initialize(app, helper = nil)
#app, #helper = app, helper
end
def call(env)
if #helper
headers = #helper.headers
if headers.is_a?(Hash)
headers.each do |k,v|
env["HTTP_#{k.upcase.gsub("-", "_")}"] = v
end
end
end
#app.call(env)
end
end
config/initializers/middleware.rb
require 'request_headers'
if %w(test cucumber).include?(Rails.env)
Rails.application.config.middleware.insert_before Rack::Lock, "RequestHeaders", CustomHeadersHelper
end
spec/support/capybara_headers.rb
require 'request_headers'
module CapybaraHeaderHelpers
shared_context "navigating within the site" do
before(:each) { add_headers("Referer" => Capybara.app_host + "/") }
end
def add_headers(custom_headers)
if Capybara.current_driver == :rack_test
custom_headers.each do |name, value|
page.driver.browser.header(name, value)
end
else
CustomHeadersHelper.headers = custom_headers
end
end
end
spec/spec_helper.rb
...
config.include CapybaraHeaderHelpers
Then I can include the shared context wherever I need, or pass different headers in another before block. I haven't tested it with anything other than Selenium and RackTest, but it should be transparent, as header injection is done before the request actually hits the application.
I wanted something a bit slimmer for RSpec/Ruby so that the custom code only had to live in one place. Here's my solution:
/spec/support/selenium.rb
...
RSpec.configure do |config|
config.after(:suite) do
$custom_headers = nil
end
end
module RequestWithExtraHeaders
def headers
$custom_headers.each do |key, value|
self.set_header "HTTP_#{key}", value
end if $custom_headers
super
end
end
class ActionDispatch::Request
prepend RequestWithExtraHeaders
end
Then in my specs:
/specs/features/something_spec.rb
...
$custom_headers = {"Referer" => referer_string}
If you are using javacsript and only want to implement on chrome, Puppeteer is the best option as it has native support to modify headers.
Check this out: https://pptr.dev/#?product=Puppeteer&version=v10.1.0&show=api-pagesetextrahttpheadersheaders
Although for cross-browser usage you might check out #requestly/selenium npm package. It is a wrapper around requestly extension to enable easy integration in selenium-webdriver.The extension can modify headers.
Check out: https://www.npmjs.com/package/#requestly/selenium
Setting request headers in the web driver directly does not work. This is true.
However, you can work around this problem by using the browser devtools (I tested with edge & chrome) and this works perfectly.
According to the documentation, you have the possibility to add custom headers:
https://chromedevtools.github.io/devtools-protocol/tot/Network/
Please find below an example.
[Test]
public async Task AuthenticatedRequest()
{
await LogMessage("=== starting the test ===");
EdgeOptions options = new EdgeOptions {UseChromium = true};
options.AddArgument("no-sandbox");
var driver = new RemoteWebDriver(new Uri(_testsSettings.GridUrl), options.ToCapabilities(), TimeSpan.FromMinutes(3));
//Get DevTools
IDevTools devTools = driver;
//DevTools Session
var session = devTools.GetDevToolsSession();
var devToolsSession = session.GetVersionSpecificDomains<DevToolsSessionDomains>();
await devToolsSession.Network.Enable(new Network.EnableCommandSettings());
var extraHeader = new Network.Headers();
var data = await Base64KerberosTicket();
var headerValue = $"Negotiate {data}";
await LogMessage($"header values is {headerValue}");
extraHeader.Add("Authorization", headerValue);
await devToolsSession.Network.SetExtraHTTPHeaders(new Network.SetExtraHTTPHeadersCommandSettings
{
Headers = extraHeader
});
driver.Url = _testsSettings.TestUrl;
driver.Navigate();
driver.Quit();
await LogMessage("=== ending the test ===");
}
This is an example written in C# but the same shall probably work with java, python as well as the major platforms.
Hope it helps the community.
If you use the HtmlUnitDriver, you can set request headers by modifying the WebClient, like so:
final case class Header(name: String, value: String)
final class HtmlUnitDriverWithHeaders(headers: Seq[Header]) extends HtmlUnitDriver {
super.modifyWebClient {
val client = super.getWebClient
headers.foreach(h => client.addRequestHeader(h.name, h.value))
client
}
}
The headers will then be on all requests made by the web browser.
With the solutions already discussed above the most reliable one is using Browsermob-Proxy
But while working with the remote grid machine, Browsermob-proxy isn't really helpful.
This is how I fixed the problem in my case. Hopefully, might be helpful for anyone with a similar setup.
Add the ModHeader extension to the chrome browser
How to download the Modheader? Link
ChromeOptions options = new ChromeOptions();
options.addExtensions(new File(C://Downloads//modheader//modheader.crx));
// Set the Desired capabilities
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
// Instantiate the chrome driver with capabilities
WebDriver driver = new RemoteWebDriver(new URL(YOUR_HUB_URL), options);
Go to the browser extensions and capture the Local Storage context ID of the ModHeader
Navigate to the URL of the ModHeader to set the Local Storage Context
.
// set the context on the extension so the localStorage can be accessed
driver.get("chrome-extension://idgpnmonknjnojddfkpgkljpfnnfcklj/_generated_background_page.html");
Where `idgpnmonknjnojddfkpgkljpfnnfcklj` is the value captured from the Step# 2
Now add the headers to the request using Javascript
.
((Javascript)driver).executeScript(
"localStorage.setItem('profiles', JSON.stringify([{ title: 'Selenium', hideComment: true, appendMode: '',
headers: [
{enabled: true, name: 'token-1', value: 'value-1', comment: ''},
{enabled: true, name: 'token-2', value: 'value-2', comment: ''}
],
respHeaders: [],
filters: []
}]));");
Where token-1, value-1, token-2, value-2 are the request headers and values that are to be added.
Now navigate to the required web-application.
driver.get("your-desired-website");
You can do it with PhantomJSDriver.
PhantomJSDriver pd = ((PhantomJSDriver) ((WebDriverFacade) getDriver()).getProxiedDriver());
pd.executePhantomJS(
"this.onResourceRequested = function(request, net) {" +
" net.setHeader('header-name', 'header-value')" +
"};");
Using the request object, you can filter also so the header won't be set for every request.
If you just need to set the User-Agent header, there is an option for Chrome:
chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"')
Now the browser sends User-Agent.
I am new to Restlet development, trying to add headers to do a HTTP request. I tried the following code, but got "400 bad request, the header is not valid"
String url = "http://xxxxx";
Client c = new Client(Protocol.HTTP);
Request request = new Request(Method.GET, url);
HashMap attributes = new HashMap();
attributes.put = ("DeviceID", "myDeviceID");
attributes.put = ("Centent-Type", "myCT");
attributes.put = ("User-Agent", "my user agent");
attributes.put = ("ClientID", "myCid");
request.setAttributes(attributes);
Response r =c.handle(request);
I am using Restlet 2.0.
Please help. any sample code would be great help. thanks in advance.
KC
HTTP protocol has a list of allowed headers: http://en.wikipedia.org/wiki/List_of_HTTP_header_fields
ClientID and DeviceID don't seem to be allowed headers. If you want custom headers you should prefix them with "X-".
Try using
attributes.put = ("Content-Type", "myCT");
Altough there might be other problems as well (myCT content-type?). Never used ClientID and DeviceID header also... but I'm a PHP guy :)