With Tools like Charles or Telerik's Fiddler you can set up your own proxy server on your home PC, intercepting in- and outgoing traffic on a more High-Level (but easier to read) view than e.g. wireshark.
Is there a way to actually analyze and or intercept the real outgoing traffic without needing to use a proxy? Just by somehow catching all the Traffic before it gets sent out to the wire? It doesn't need to be on bit layer (the plain HTTP requests would be just fine ;) )
Regards
If you're mentioning about iOS app, you can do it by Atlantis Framework + Proxyman app.
Instead of proxying your traffic to a proxy server, like Charles, Fiddler does, Atlantis will capture your traffic on-the-wire (by using the Method Swizzling technique) and broadcast it to the Proxyman app for previewing.
Related
Is there a way to completely encrypt the outgoing HTTPS requests from the software to the server ? i mean there are apps like Charles and Fiddler that can capture the HTTPS traffic and see everything like the Headers,URL,...
i don not want anyone to see or capture the traffic going from my app.
i'm using Delphi 10.1 VCL App
If you're using HTTPS and you are properly validating certificates as a browser would, there is no way for an intermediate to view URLs or headers, or content. All they can see is which server you're communicating with.
The way Charles gets around this is that it presents its own non-genuine certificate, which won't validate, and proxies the communication. If your app is validating certificates it would refuse to communicate with the Charles proxy. If you viewed the Charles proxy with a web browser it would present an SSL certificate error.
If you trust that particular Charles proxy and want to add an exception in your client or browser, you can. But it only allows that particular one - it doesn't mean anyone else can intercept your HTTPS, or read URLs, etc, using their own Charles proxy or similar.
I have a web application hosted in servlet container. It is making call to google sheet api to create google sheet. It is throwing GoogleJsonResponseException with message 401 unauthorized. I want to see HTTP traffic between client program and google server. How to do that? Any other way to debug?
I use two tools for this purpose, both free. The first is Fiddler from Telerik. This tool is an http proxy that intercepts your traffic and lets you dissect the underling http. I have found though that not all traffic is intercepted. I use it on Windows and I think it intercepts traffic that runs thru http.sys.
My second tool is Wireshark which is a network packet sniffer and intercepts http traffic that sometimes does not get picked up by Fiddler.
There is tons of articles and tutorials on how to use both tools. If your traffic is using SSL/TLS then it can be tricky to setup. I have had success with Fiddler, but not Wireshark, even where I had the private key.
Both tools are indispensable.
I am having an issue with Server Sent events.
My endpoint is not available on mobile 3G network.
One observation I have is that a https endpoint like the one below which is available on my mobile network.
https://s-dal5-nss-32.firebaseio.com/s1.json?ns=iot-switch&sse=true
But the same endpoint when proxy passed using an nginx and accessed over http (without ssl) is not available on my mobile network.
http://aws.arpit.me/live/s1.json?ns=iot-switch&sse=true
This is available on my home/office broadband network though. Only creates an issue over my mobile 3g network.
Any ideas what might be going on?
I read that mobile networks use broken transparent proxies that might be causing this. But this is over HTTP.
Any help would be appreciated.
I suspect the mobile network is forcing use of an HTTP proxy that tries to buffer files before forwarding them to the browser. Buffering will make SSE messages wait in the buffer.
With SSE there are a few tricks to work around such proxies:
Close the connection on the server after sending a message. Proxies will observe end of the "file" and forward all messages they've buffered.
This will be equivalent to long polling, so it's not optimal. To avoid reducing performance for all clients you could do it only if you detect it's necessary, e.g. when a client connects always send a welcome message. The client should expect that message and if the message doesn't arrive soon enough report the problem via an AJAX request to the server.
Send between 4 and 16KB of data in SSE comments before or after a message. Some proxies have limited-size buffers, and this will overflow the buffer forcing messages out.
Use HTTPS. This bypasses all 3rd party proxies. It's the best solution if you can use HTTPS.
I would like to know how "realistic" is to consider implementing an intercepting proxy(with cache support) for the purpose of web filtering. I would like to support also IPv6, authentication of clients and caching.
Reading to the list of disadvantages from squid wiki http://wiki.squid-cache.org/SquidFaq/InterceptionProxy that implements an intercepting proxy, it mentions some things to consider as disadvantages when using it(that I want to clarify):
Requires IPv4 with NAT - proxy intercepting does not support IPv6, why ?
it causes path-MTU (PMTUD) to possibly fail - why ?
Proxy authentication does not work - client thinks it's talking directly to the originating server, in there a way to do authentication in this case ?
Interception Caching only supports the HTTP protocol, not gopher, SSL, or FTP. You cannot setup a redirection-rule to the proxy server for other protocols other than HTTP since it will not know how to deal with it - This seems quite plausible as the way redirecting of traffic to proxy is done in this case is by a firewall changing the destination address of a packet from the originating server to the proxy's own address(Destination NAT). How would in this case, if i want to intercept other protocols besides http know where the connection was intended to go so I can relay it to that destination ?
Traffic may be intercepted in many ways. It does not necessarily need to use NAT (which is not supported in IPv6). A transparent interception will surely not use NAT for example (transparent in the sense that the Proxy will not generate requests with his own address but with the client address, spoofing the IP address).
PMTUD is used to detect the largest MTU size available in the path between the client and server and vise versa, it is useful for avoiding fragmentation of Ip packets on the path between the client and server. When you use a Proxy in the middle, even if the MTU is detected, it not necessarily the same as the one from the client to the proxy and from the proxy to the server. But this is not always relevant, it depends on what traffic is being served and how the proxy is behaving.
If the proxy is authenticating in the client behalf, it needs to be aware of the authentication method, and it will probably need some cookies that exist in the client. Think of it this way... If a proxy can authenticate an access to a restricted resource on your behalf, it means anyone can do it on your behalf, and the purpose of a good authentication is to protect you from such possibilities.
I guess this was a very old post from the Squid guys, but the technology exists to redirect anything you want to a specific server. One simple way to do it is by placing your server as a Default Gateway for the network, then all packets pass through it and you could redirect the packets you like to your application (or another server). And you are not limited to HTTP, BUT you are limited to the way the application protocol works.
I'm learning about network programming. Specifically proxy servers. I've created a very rudimentary proxy server on my mobile phone. However I think there's some proxy server basics that I don't know that will help me create a more robust proxy server.
What I've done so far: server on my mobile device listens for requests from laptop. When server receives a request like www.google.com the web page contents are fetched and returned to the client on the laptop. The client then opens the page contents in a desktop browser.
I think the sending/receiving of requests can happen on a lower OSI model layer (perhaps transport). How can I create a more robust proxy server? (one that just sends and receives bytes and doesn't care/know about HTTP)
A proxy server runs at the same layer as the protocol being proxied. It seems you are talking about an HTTP proxy. HTTP runs over TCP, and so does an HTTP proxy.
Define 'more robust'. What have you done so far?
An HTTP proxy server is a pretty simple thing, unless it has elaborate logging, caching, etc. The basis of it is (1) something to recognize and action the GET/POST/PUT/CONNECT etc. commands and (2) thereafter just copying bytes in both directions simultaneously.