Related
I'm restricted to only using Cowboy for a web server that handles a JSON REST API. I need to be able to use only Cowboy + whatever the language capabilities are to manage and process different and variable routes, as well as using the GET values.
I'm getting the path as explained in the following routine:
def handle(req, router) do
headers = [{"content-type", "application/json"}]
{path, req} = :cowboy_req.path(req)
{:ok, resp} = :cowboy_req.reply(200, headers, router.call(path), req)
{:ok, resp, router}
end
And ultimately route.call(path) calls the following:
defp serve("/call/[:thing]") do
list = [path: "oy"]
IO.puts :thing
{status, result} = JSON.encode(list)
result
end
By itself, serve("/call") returns the JSON without issues, but trying to request any other route under /call to the server, makes it answer with the 404 response (already handled by me).
What's the best approach when handling these dynamic routes? Bear in mind that I'm delimited to only using Cowboy and nothing else.
Your code is not very clear - how did you start the server? More specifically, how did you setup your router? This seems to be the problem here, I'm guessing you made a route only for /call.
You'd need something like this:
dispatch_config = :cowboy_router.compile([{:_, [{"/call/[:thing]", YourHandlerModule, []}]}])
{ :ok, _ } = :cowboy.start_http(:http,
100,
[{:port, 8080}],
[{ :env, [{:dispatch, dispatch_config}]}]
)
The path /call/[:thing] should be specified at the router, not inside your handler.
I found one simple solution by only using Cowboy and Elixir:
def call(conn) do
serve(conn.req_path, conn)
end
defp serve(<< "/call/", name::binary >>, conn) do
list = conn.req_qs
IO.puts name
{_, result} = JSON.encode(list)
put_resp_body(conn, result)
If you do it like this, all subsequent routes stay in rest. A simple split would do. conn carries the query string, so I can get the values from there.
When starting an http server using HTTP.serve there is apparently no way to reload the code that is actually handling the HTTP request.
In the example below I would like to have the modifications in my_httphandler taken into account without having to restart the server.
For the moment I need to stop the server from the REPL by pressing CTRL+C twice and then run the script again.
Is there a workaround ?
module MyModule
using HTTP
using Mux
using JSON
using Sockets
function my_httphandler(req::HTTP.Request)
return HTTP.Response(200, "Hello world")
end
const MY_ROUTER = HTTP.Router()
HTTP.#register(MY_ROUTER, "GET", "/*", my_httphandler)
HTTP.serve(MY_ROUTER, Sockets.localhost, 8081)
end
I'm not sure whether Mux caches handlers. As long as it does not, this should work:
module MyModule
using HTTP
using Mux
using JSON
using Sockets
function my_httphandler(req::HTTP.Request)
return HTTP.Response(200, "Hello world")
end
const functionref = Any[my_httphandler]
const MY_ROUTER = HTTP.Router()
HTTP.#register(MY_ROUTER, "GET", "/*", functionref[1])
HTTP.serve(MY_ROUTER, Sockets.localhost, 8081)
end
function newhandler(req::HTTP.Request)
return HTTP.Response(200, "Hello world 2")
end
MyModule.functionref[1] = newhandler
Revise.jl lets you automatically update code in a live Julia session. You may be especially interested in entr; see Revise's documentation for details.
When using HTTP.jl: just add #async before HTTP.serve
module MyModule
using HTTP
using Sockets
function my_httphandler(req::HTTP.Request)
return HTTP.Response(200, "Hello world")
end
const MY_ROUTER = HTTP.Router()
HTTP.#register(MY_ROUTER, "GET", "/*", my_httphandler)
#async HTTP.serve(MY_ROUTER, Sockets.localhost, 8081)
end # module
When using Mux.jl: nothing to do, the server is started in the background
using Mux
function sayhellotome(name)
return("hello " * name * "!!!")
end
#app test = (
Mux.defaults,
route("/sayhello/:user", req -> begin
sayhellotome(req[:params][:user])
end),
Mux.notfound())
Mux.serve(test, 8082)
I've added a ticket #587 to HTTP.jl project for developer workflow support. I'm not sure this is your use case or not.
# hello.jl -- an example showing how Revise.jl works with HTTP.jl
# julia> using Revise; includet("hello.jl"); serve();
using HTTP
using Sockets
homepage(req::HTTP.Request) =
HTTP.Response(200, "<html><body>Hello World!</body></html>")
const ROUTER = HTTP.Router()
HTTP.#register(ROUTER, "GET", "/", homepage)
serve() = HTTP.listen(request -> begin
Revise.revise()
Base.invokelatest(HTTP.handle, ROUTER, request)
end, Sockets.localhost, 8080, verbose=true)
Alternatively, you could have a test/serve.jl file, that assumes MyModule with a top-level HTTP.jl router is called ROUTER. You'll need to remove the call to serve in your main module.
#!/usr/bin/env julia
using HTTP
using Sockets
using Revise
using MyModule: ROUTER
HTTP.listen(request -> begin
Revise.revise()
Base.invokelatest(HTTP.handle, ROUTER, request)
end, Sockets.localhost, 8080, verbose=true)
A more robust solution would catch errors; however, I had challenges getting this to work and reported my experience at #541 in Revise.jl.
I have the following controller concern module:
#controllers/concerns/response.rb
module Response
extend ActiveSupport::Concern
def json_response(object, status = :ok, opts = {})
response = {json: object, status: status}.merge(opts)
render response
end
...
end
ApplicationController includes it as follows:
class ApplicationController < ActionController::API
include Response
...
end
How would it be possible to test the above concern methods? What kind of RSpec tests should it be (controller, request)?
I tried to define a shared_examplesas follows:
#spec/shared/json_response.rb
require 'rails_helper'
RSpec.shared_examples 'JSON Responsive controller' do |controller_class|
let(:controller_class) { controller_class }
it 'render JSON response' do
expect(controller_class).to respond_to(:json_response)
end
end
and to use it in a controller spec:
#spec/controllers/concerns/fake_controller.rb
require 'rails_helper'
class FakeController < ApplicationController
end
RSpec.describe FakeController, type: :controller do
it_behaves_like 'JSON Responsive controller', FakeController
end
but it fails with:
Failures:
1) FakeController behaves like JSON Responsive controller render JSON response
Failure/Error: expect(controller_class).to respond_to(:json_response)
expected FakeController to respond to :json_response
Shared Example Group: "JSON Responsive controller" called from ./spec/controllers/concerns/fake_controller_spec.rb:7
# ./spec/shared/json_response.rb:7:in `block (2 levels) in <main>'
Finished in 0.23535 seconds (files took 1.11 seconds to load)
1 example, 1 failure
What am I missing ?
Here is the solution I came to to make it work.
Create a controller spec in spec/controller/fake_controller_spec.rb as follows:
require 'rails_helper'
class FakeController < ApplicationController
def render(*args)
args.first
end
end
RSpec.describe FakeController, type: :controller do
it_should_behave_like "JSON Responsive controller", FakeController
end
I had to override render(*args) method to be able to call render from inside Response module concern.
Create a shared_examples spec in spec/shared/json_response.rb:
require 'rails_helper'
RSpec.shared_examples 'JSON Responsive controller' do |including_controller|
let(:instance) { including_controller.new }
it 'should respond to #json_response' do
expect(instance).to respond_to(:json_response)
end
it 'should respond #respond_with_errors' do
expect(instance).to respond_to(:respond_with_errors)
end
it 'should respond to #paginated_response_status' do
expect(instance).to respond_to(:paginated_response_status)
end
context '#paginated_response_status' do
it 'returns 200 if collection is not paginated' do
expect(instance.paginated_response_status([1])).to eq :ok
end
it 'returns 206 if collection is paginated' do
collection = (1..35).to_a
expect(instance.paginated_response_status(collection)).to eq :partial_content
end
end
context '#respond_with_errors' do
it 'returns :unprocessable_entity status' do
model = double(:model)
errors = double(:errors, messages: {})
allow(model).to receive(:errors).and_return(errors)
response = instance.respond_with_errors(model)
expect(response[:status]).to eq :unprocessable_entity
end
end
context '#json_response' do
it 'returns JSON with default :ok status' do
model = double(:model)
response = instance.json_response(model)
expect(response[:status]).to eq :ok
end
it 'returns JSON with the specified status' do
model = double(:model)
response = instance.json_response(model, :partial_content)
expect(response[:status]).to eq :partial_content
end
end
end
Note, to be able to use shared example deined in shared folder, you have to add the following to rails_helper.rb file:
Dir[Rails.root.join('spec/shared/**/*.rb')].each { |f| require f }
...
RSpec.configure do |config|
..
end
Finally, here is the code to test defined in controllers/concerns/response.rb:
module Response
extend ActiveSupport::Concern
def json_response(object, status = :ok, opts = {})
response = {json: object, status: status}.merge(opts)
render response
end
def respond_with_errors(object)
render json: { errors: ErrorSerializer.serialize(object) }, status: :unprocessable_entity
end
def paginated_response_status(collection)
collection.size > WillPaginate.per_page ? :partial_content : :ok
end
end
ErrorSerializer is just another module that creates a JSON to return in case of errors:
#controllers/concerns/error_serializer.rb
module ErrorSerializer
extend ActiveSupport::Concern
def self.serialize(object)
object.errors.messages.map do |field, errors|
errors.map do |error_message|
{
status: 422,
title: 'Invalid attribute',
source: { pointer: "/data/attributes/#{field}" },
detail: error_message
}
end
end.flatten
end
end
Hope this helps.
I've been trying to scrape some lists from this website http://www.golf.org.au its an ASP.NET based I did some research and it appears that I must pass some values in a POST request to make the website fetch the data into the tables I did that but still I'm failing any Idea what I'm missing?
Here is my code:
# -*- coding: utf-8 -*-
import scrapy
class GolfscraperSpider(scrapy.Spider):
name = "golfscraper"
allowed_domains = ["golf.org.au","www.golf.org.au"]
ids = ['3012801330', '3012801331', '3012801332', '3012801333']
start_urls = []
for id in ids:
start_urls.append('http://www.golf.org.au/handicap/%s' %id)
def parse(self, response):
scrapy.FormRequest('http://www.golf.org.au/default.aspx?
s=handicap',
formdata={
'__VIEWSTATE':
response.css('input#__VIEWSTATE::attr(value)').extract_first(),
'ctl11$ddlHistoryInMonths':'48',
'__EVENTTARGET':
'ctl11$ddlHistoryInMonths',
'__EVENTVALIDATION' :
response.css('input#__EVENTVALIDATION::attr(value)').extract_first(),
'gaHandicap' : '6.5',
'golflink_No' : '2012003003',
'__VIEWSTATEGENERATOR' : 'CA0B0334',
},
callback=self.parse_details)
def parse_details(self,response):
for name in response.css('div.rnd-course::text').extract():
yield {'name' : name}
Yes, ASP pages are tricky to scrape. Most probably some little parameter is missing.
Solution for this:
instead of creating the request through scrapy.FormRequest(...) use the scrapy.FormRequest.from_response() method (see code example below). This will capture most or even all of the hidden form data and use it to prepopulate the FormRequest's data.
it seems you forgot to return the request, maybe that's another potential problem too ...
as far as I recall the __VIEWSTATEGENERATOR also will change each time and has to be extracted from the page
If this doesn't work, fire up your Firefox browser with Firebug plugin or Chrome's developer tools, do the request in the browser and then check the full request header and body data against the same data in your request. There will be some difference.
Example code with all my suggestions:
def parse(self, response):
req = scrapy.FormRequest.from_response(response,
formdata={
'__VIEWSTATE': response.css('input#__VIEWSTATE::attr(value)').extract_first(),
'ctl11$ddlHistoryInMonths':'48',
'__EVENTTARGET': 'ctl11$ddlHistoryInMonths',
'__EVENTVALIDATION' : response.css('input#__EVENTVALIDATION::attr(value)').extract_first(),
'gaHandicap' : '6.5',
'golflink_No' : '2012003003',
'__VIEWSTATEGENERATOR' : 'CA0B0334',
},
callback=self.parse_details)
log.info(req.headers)
log.info(req.body)
return req
I'm attempting to set the request header 'Referer' to spoof a request coming from another site. We need the ability test that a specific referrer is used, which returns a specific form to the user, otherwise an alternative form is given.
I can do this within poltergeist by:
page.driver.headers = {"Referer" => referer_string}
but I can't find the equivalent functionality for the selemium driver.
How can I set request headers in the capybara selenium driver?
Webdriver doesn't contain an API to do it. See issue 141 from Selenium tracker for more info. The title of the issue says that it's about response headers but it was decided that Selenium won't contain API for request headers in scope of this issue. Several issues about adding API to set request headers have been marked as duplicates: first, second, third.
Here are a couple of possibilities that I can propose:
Use another driver/library instead of selenium
Write a browser-specific plugin (or find an existing one) that allows you to add header for request.
Use browsermob-proxy or some other proxy.
I'd go with option 3 in most of cases. It's not hard.
Note that Ghostdriver has an API for it but it's not supported by other drivers.
For those people using Python, you may consider using Selenium Wire which can set request headers as well as provide you with the ability to inspect requests and responses.
from seleniumwire import webdriver # Import from seleniumwire
# Create a new instance of the Chrome driver (or Firefox)
driver = webdriver.Chrome()
# Create a request interceptor
def interceptor(request):
del request.headers['Referer'] # Delete the header first
request.headers['Referer'] = 'some_referer'
# Set the interceptor on the driver
driver.request_interceptor = interceptor
# All requests will now use 'some_referer' for the referer
driver.get('https://mysite')
Install with:
pip install selenium-wire
I had the same issue. I solved it downloading modify-headers firefox add-on and activate it with selenium.
The code in python is the following
fp = webdriver.FirefoxProfile()
path_modify_header = 'C:/xxxxxxx/modify_headers-0.7.1.1-fx.xpi'
fp.add_extension(path_modify_header)
fp.set_preference("modifyheaders.headers.count", 1)
fp.set_preference("modifyheaders.headers.action0", "Add")
fp.set_preference("modifyheaders.headers.name0", "Name_of_header") # Set here the name of the header
fp.set_preference("modifyheaders.headers.value0", "value_of_header") # Set here the value of the header
fp.set_preference("modifyheaders.headers.enabled0", True)
fp.set_preference("modifyheaders.config.active", True)
fp.set_preference("modifyheaders.config.alwaysOn", True)
driver = webdriver.Firefox(firefox_profile=fp)
Had the same issue today, except that I needed to set different referer per test. I ended up using a middleware and a class to pass headers to it. Thought I'd share (or maybe there's a cleaner solution?):
lib/request_headers.rb:
class CustomHeadersHelper
cattr_accessor :headers
end
class RequestHeaders
def initialize(app, helper = nil)
#app, #helper = app, helper
end
def call(env)
if #helper
headers = #helper.headers
if headers.is_a?(Hash)
headers.each do |k,v|
env["HTTP_#{k.upcase.gsub("-", "_")}"] = v
end
end
end
#app.call(env)
end
end
config/initializers/middleware.rb
require 'request_headers'
if %w(test cucumber).include?(Rails.env)
Rails.application.config.middleware.insert_before Rack::Lock, "RequestHeaders", CustomHeadersHelper
end
spec/support/capybara_headers.rb
require 'request_headers'
module CapybaraHeaderHelpers
shared_context "navigating within the site" do
before(:each) { add_headers("Referer" => Capybara.app_host + "/") }
end
def add_headers(custom_headers)
if Capybara.current_driver == :rack_test
custom_headers.each do |name, value|
page.driver.browser.header(name, value)
end
else
CustomHeadersHelper.headers = custom_headers
end
end
end
spec/spec_helper.rb
...
config.include CapybaraHeaderHelpers
Then I can include the shared context wherever I need, or pass different headers in another before block. I haven't tested it with anything other than Selenium and RackTest, but it should be transparent, as header injection is done before the request actually hits the application.
I wanted something a bit slimmer for RSpec/Ruby so that the custom code only had to live in one place. Here's my solution:
/spec/support/selenium.rb
...
RSpec.configure do |config|
config.after(:suite) do
$custom_headers = nil
end
end
module RequestWithExtraHeaders
def headers
$custom_headers.each do |key, value|
self.set_header "HTTP_#{key}", value
end if $custom_headers
super
end
end
class ActionDispatch::Request
prepend RequestWithExtraHeaders
end
Then in my specs:
/specs/features/something_spec.rb
...
$custom_headers = {"Referer" => referer_string}
If you are using javacsript and only want to implement on chrome, Puppeteer is the best option as it has native support to modify headers.
Check this out: https://pptr.dev/#?product=Puppeteer&version=v10.1.0&show=api-pagesetextrahttpheadersheaders
Although for cross-browser usage you might check out #requestly/selenium npm package. It is a wrapper around requestly extension to enable easy integration in selenium-webdriver.The extension can modify headers.
Check out: https://www.npmjs.com/package/#requestly/selenium
Setting request headers in the web driver directly does not work. This is true.
However, you can work around this problem by using the browser devtools (I tested with edge & chrome) and this works perfectly.
According to the documentation, you have the possibility to add custom headers:
https://chromedevtools.github.io/devtools-protocol/tot/Network/
Please find below an example.
[Test]
public async Task AuthenticatedRequest()
{
await LogMessage("=== starting the test ===");
EdgeOptions options = new EdgeOptions {UseChromium = true};
options.AddArgument("no-sandbox");
var driver = new RemoteWebDriver(new Uri(_testsSettings.GridUrl), options.ToCapabilities(), TimeSpan.FromMinutes(3));
//Get DevTools
IDevTools devTools = driver;
//DevTools Session
var session = devTools.GetDevToolsSession();
var devToolsSession = session.GetVersionSpecificDomains<DevToolsSessionDomains>();
await devToolsSession.Network.Enable(new Network.EnableCommandSettings());
var extraHeader = new Network.Headers();
var data = await Base64KerberosTicket();
var headerValue = $"Negotiate {data}";
await LogMessage($"header values is {headerValue}");
extraHeader.Add("Authorization", headerValue);
await devToolsSession.Network.SetExtraHTTPHeaders(new Network.SetExtraHTTPHeadersCommandSettings
{
Headers = extraHeader
});
driver.Url = _testsSettings.TestUrl;
driver.Navigate();
driver.Quit();
await LogMessage("=== ending the test ===");
}
This is an example written in C# but the same shall probably work with java, python as well as the major platforms.
Hope it helps the community.
If you use the HtmlUnitDriver, you can set request headers by modifying the WebClient, like so:
final case class Header(name: String, value: String)
final class HtmlUnitDriverWithHeaders(headers: Seq[Header]) extends HtmlUnitDriver {
super.modifyWebClient {
val client = super.getWebClient
headers.foreach(h => client.addRequestHeader(h.name, h.value))
client
}
}
The headers will then be on all requests made by the web browser.
With the solutions already discussed above the most reliable one is using Browsermob-Proxy
But while working with the remote grid machine, Browsermob-proxy isn't really helpful.
This is how I fixed the problem in my case. Hopefully, might be helpful for anyone with a similar setup.
Add the ModHeader extension to the chrome browser
How to download the Modheader? Link
ChromeOptions options = new ChromeOptions();
options.addExtensions(new File(C://Downloads//modheader//modheader.crx));
// Set the Desired capabilities
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
// Instantiate the chrome driver with capabilities
WebDriver driver = new RemoteWebDriver(new URL(YOUR_HUB_URL), options);
Go to the browser extensions and capture the Local Storage context ID of the ModHeader
Navigate to the URL of the ModHeader to set the Local Storage Context
.
// set the context on the extension so the localStorage can be accessed
driver.get("chrome-extension://idgpnmonknjnojddfkpgkljpfnnfcklj/_generated_background_page.html");
Where `idgpnmonknjnojddfkpgkljpfnnfcklj` is the value captured from the Step# 2
Now add the headers to the request using Javascript
.
((Javascript)driver).executeScript(
"localStorage.setItem('profiles', JSON.stringify([{ title: 'Selenium', hideComment: true, appendMode: '',
headers: [
{enabled: true, name: 'token-1', value: 'value-1', comment: ''},
{enabled: true, name: 'token-2', value: 'value-2', comment: ''}
],
respHeaders: [],
filters: []
}]));");
Where token-1, value-1, token-2, value-2 are the request headers and values that are to be added.
Now navigate to the required web-application.
driver.get("your-desired-website");
You can do it with PhantomJSDriver.
PhantomJSDriver pd = ((PhantomJSDriver) ((WebDriverFacade) getDriver()).getProxiedDriver());
pd.executePhantomJS(
"this.onResourceRequested = function(request, net) {" +
" net.setHeader('header-name', 'header-value')" +
"};");
Using the request object, you can filter also so the header won't be set for every request.
If you just need to set the User-Agent header, there is an option for Chrome:
chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"')
Now the browser sends User-Agent.