I would like to be able to monitor all the HTTP requests being made by a web page in an automated testing scenario.
I know how to drive browsers with Selenium.
Is there some kind of proxy that can be interacted with programatically. What would help is something that can be flagged to start recording all the HTTP requests then flagged to stop.
I believe FireFox has some proxy settings that can be driven from Selenium but Chrome is the highest priority browser for testing.
I have heard of BetaMax but think this is more about simulating and replaying REST calls rather than monitoring traffic programatically.
Take a look at Hoverfly and it has a mode where it acts as a proxy. I haven't used it, but I believe you can replay whatever is recorded by the proxy when the requests are re-sent. And yes there is an API.
Related
With Tools like Charles or Telerik's Fiddler you can set up your own proxy server on your home PC, intercepting in- and outgoing traffic on a more High-Level (but easier to read) view than e.g. wireshark.
Is there a way to actually analyze and or intercept the real outgoing traffic without needing to use a proxy? Just by somehow catching all the Traffic before it gets sent out to the wire? It doesn't need to be on bit layer (the plain HTTP requests would be just fine ;) )
Regards
If you're mentioning about iOS app, you can do it by Atlantis Framework + Proxyman app.
Instead of proxying your traffic to a proxy server, like Charles, Fiddler does, Atlantis will capture your traffic on-the-wire (by using the Method Swizzling technique) and broadcast it to the Proxyman app for previewing.
I'm implementing an HTTP proxy. Sometimes when a browser makes a request via my proxy, I get an error such as ECONNRESET, Address not found, and the like. These indicate errors below the HTTP level. I'm not talking about bugs in my program -- but how other servers behave when I send them an HTTP request.
Some servers might simply not exist, others close the socket, and still others not answer at all.
What is the best way to report these errors to the caller? Is there a standard method that, if I use it, browsers will convert my HTTP message to an appropriate error message? (i.e. they get a reply from the proxy that tells them ECONNRESET, and they act as though they received the ECONNRESET themselves).
If not, how should it be handled?
Motivations
I really want my proxy to be totally transparent and for the browser or other client to work exactly as if it wasn't connected to it, so I want to replicate the organic behavior of errors such as ECONNRESET instead of sending an HTTP message with an error code, which would be totally different behavior.
I kind of thought that was the intention when writing an HTTP proxy.
There are several things to keep in mind.
Firstly, if the client is configured to use the proxy (which actually I'd recommend) then fundamentally it will behave differently than if it were directly connecting out over the Internet. This is mostly invisible to the user, but affects things like:
FTP URLs
some caching differences
authentication to the proxy if required
reporting of connection errors etc <= your question.
In the case of reporting errors, a browser will show a connectivity error if it can't connect to the proxy, or open a tunnel via the proxy, but for upstream errors, the proxy will be providing a page (depending on the error, e.g. if a response has already been sent the proxy can't do much but close the connection). This page won't look anything like your browser page would.
If the browser is NOT configured to use a proxy, then you would need to divert or intercept the connection to the proxy. This can cause problems if you decide you want to authenticate your users against the proxy (to identify them / implement user-specific rules etc).
Secondly HTTPS can be a real pain in the neck. This problem is growing as more and more sites move to HTTPS only. There are several issues:
browsers configured to use a proxy, for HTTPS URLS will firstly open a tunnel via the proxy using the CONNECT method. If your proxy wants to prevent this then any information it provides in the block response is ignored by the browser, and instead you get the generic browser connectivity error page.
if you want to provide any other benefits one normally wishes from a proxy (e.g. caching / scanning etc) you need to implement a MitM (Man-in-the-middle) and spoof server SSL certificates etc. In fact you need to do this if you just want to send back a block-page to deny things.
There is a way a browser can act a bit more like it was directly connected via a proxy, and that's using SOCKS. SOCKS has a way to return an error code if there's an upstream connection error. It's not the actual socket error code however.
These are all reasons why we wrote the WinGate Internet Client, which is a LSP-based product for our product WinGate. Client applications then learn the actual upstream error codes etc.
It's not a favoured approach nowadays though, as it requires installation of software on the client computer.
I wouldn't provide them too much info. Report what you need through internal logs in case you have to solve the problem. Return a 400, 403 or 418. Why? Perhaps the're just hacking.
I have a web application hosted in servlet container. It is making call to google sheet api to create google sheet. It is throwing GoogleJsonResponseException with message 401 unauthorized. I want to see HTTP traffic between client program and google server. How to do that? Any other way to debug?
I use two tools for this purpose, both free. The first is Fiddler from Telerik. This tool is an http proxy that intercepts your traffic and lets you dissect the underling http. I have found though that not all traffic is intercepted. I use it on Windows and I think it intercepts traffic that runs thru http.sys.
My second tool is Wireshark which is a network packet sniffer and intercepts http traffic that sometimes does not get picked up by Fiddler.
There is tons of articles and tutorials on how to use both tools. If your traffic is using SSL/TLS then it can be tricky to setup. I have had success with Fiddler, but not Wireshark, even where I had the private key.
Both tools are indispensable.
I am having an issue with Server Sent events.
My endpoint is not available on mobile 3G network.
One observation I have is that a https endpoint like the one below which is available on my mobile network.
https://s-dal5-nss-32.firebaseio.com/s1.json?ns=iot-switch&sse=true
But the same endpoint when proxy passed using an nginx and accessed over http (without ssl) is not available on my mobile network.
http://aws.arpit.me/live/s1.json?ns=iot-switch&sse=true
This is available on my home/office broadband network though. Only creates an issue over my mobile 3g network.
Any ideas what might be going on?
I read that mobile networks use broken transparent proxies that might be causing this. But this is over HTTP.
Any help would be appreciated.
I suspect the mobile network is forcing use of an HTTP proxy that tries to buffer files before forwarding them to the browser. Buffering will make SSE messages wait in the buffer.
With SSE there are a few tricks to work around such proxies:
Close the connection on the server after sending a message. Proxies will observe end of the "file" and forward all messages they've buffered.
This will be equivalent to long polling, so it's not optimal. To avoid reducing performance for all clients you could do it only if you detect it's necessary, e.g. when a client connects always send a welcome message. The client should expect that message and if the message doesn't arrive soon enough report the problem via an AJAX request to the server.
Send between 4 and 16KB of data in SSE comments before or after a message. Some proxies have limited-size buffers, and this will overflow the buffer forcing messages out.
Use HTTPS. This bypasses all 3rd party proxies. It's the best solution if you can use HTTPS.
I was playing around with fiddler (http proxy) and I noticed that some apps are making http get/post requests in the background and sending data and stats to and from the web. This got me interested and a little concerned to see what data various apps were sending but it seems that most of them are not doing it on port 80 via http but presumably on another port so you can't see the data in fiddler. Is there some way to view and/or potentially block the data being sent?
You're asking: "Using Fiddler, I saw that traffic was being sent by clients to servers. How can I see that traffic?"
Might I suggest you use Fiddler?
You can see the process sending the traffic in the Process column, and you can view the contents of the requests and responses using the Inspectors tab.
I would check out burp suite. It is a proxy you set up in your web browser and shows all of the data that passes through it. There's plenty of tutorials online. Check it out here