I am using squid for the very first time , actually i am forwarding from privoxy to squid and then to server.
I could able to see privoxy logs when i hit a request not the Squid log.
I have checked in access.log but no luck.
Can anyone help me with this.
Unless you have configured Squid otherwise, all requests (whether successful or not) will be written to the access.log file after completion. In the case of a successful request, this happens almost immediately, but an unsuccessful request might take up to 30 seconds to appear in the log. Are you waiting long enough for this to happen before checking the file?
Assuming you're running on Linux, perhaps you could send some requests through the proxy using a tool like wget, then check the log. This will confirm that logging is actually working. Use this syntax:
http_proxy=http://localhost:3128/ wget www.google.com --debug
Prepending the "http_proxy=" at the start of the command tells wget to use a proxy listening at localhost on port 3128. What does wget show you? Does Squid write anything to the access.log file? If so, that would seem to indicate your proxy is logging fine, but privoxy isn't sending anything to it.
Another thing you could try is to run a packet capture on the TCP port you expect privoxy to send traffic to Squid over. I don't know how you configure privoxy, but I'm assuming the sequence goes something like this:
Client > Privoxy > Squid > Server
In that case, lets say privoxy connects to Squid on localhost over TCP port 3128. You could run a packet capture to see if privoxy is even connecting to Squid, like this:
tcpdump -i localhost port 3128 -vv
Or, if you see packets being transferred but want to know what's inside them, then the excellent tcpflow is your friend:
tcpflow -c -i localhost port 3128
If you edit your question to provide more context, it might be possible to provide a better answer.
The following configuration i have added and i could see the logs now.
access_log /var/log/squid/access.log squid
access_log syslog:daemon.debug squid
Related
I'm trying to set up a proxy_pass while also using a socks5 proxy. I can access my backing service with curl using the following:
curl -x socks5h://localhost:8001 -svo /dev/null -I http://[abcd:1234::]:8000
So what I've currently got in my nginx config which doesn't work is:
location / {
proxy_pass http://[abcd:1234::]:8000
proxy_redirect http://localhost:8001 /;
}
It also seems like nginx has no notion of ALL_PROXY|HTTP(S)_PROXY environment variables which other applications can use.
Any idea how I can get this to work?
I did find a related question - socks5 proxy/tunnel for nginx upstream? but it's now 6 years old and I'm not sure works still.
Why it does not work?
To my knowledge, proxy_pass, proxy_redirect and other functionality in the ngx_http_proxy_module is meant to act as a HTTP/HTTPS proxy only. This seems to be confirmed by the 'As a protocol, “http” or “https” can be specified.' note in proxy_pass documentation (no mention of SOCKS).
The proxy_pass directive allows you to tell NGINX to take whatever requests it receives at specific location and blindly send them to another HTTP server, wait for the response from said server and return the response to the client. Other directives from the module (for example proxy_redirect) allow slight modifications to requests/responses. What is important is that the entire process is very simple and there is no tunneling (aside from TLS when location is https) or wrapping in additional protocols.
In contrast, SOCKS proxies require implementation of the SOCKS proxy protocol and using it to wrap all the connections. This additional work cannot be performed using the ngx_http_proxy_module.
How to make it work?
Unfortunately, using SOCKS proxies in NGINX does not seem to be supported by any of the core modules (listed here below 'Module reference'). It also does not seem to be a popular use case, so i would not expect support for it in NGINX core anytime soon. In another question you linked one of the answers references a third party nginx module which is also listed on the nginx.com website (the list has no anchors, so CTRL + F for "SOCKS" and you will find it). Last commit is from 2016 but it is possible it will still work.
If you can't change the way you access your backend service, i would say your best bet is either using the module mentioned above and trying to fix it if it does not work or writing your own module. Alternatively, maybe you could establish port forwarding to the backend service over the SOCKS proxy, and just proxy_pass to your local port. If you have an ssh server running on your backend service host, you could set up a simple proof-of-concept like this:
ssh <YOUR-SSH-LOGIN>#<BACKEND-HOST> \
-L 8081:localhost:80 \
-o "ProxyCommand=nc -X 5 -x <YOUR-SOCKS-PROXY-IP>:<YOUR-SOCKS-PROXY-PORT> %h %p"
The -L argument creates port forwarding between your local 8081 port and port 80 (http) on backend host. The -o argument adds a ProxyCommand option which uses netcat to forward traffic over a SOCKS proxy (not all netcat versions support the -X and -x arguments, the one i am using is openbsd-netcat on Arch Linux). After using that you should be able to just proxy_pass to localhost:8081 in NGINX. This setup is not very performant and serves only as a proof-of-concept, if you decide to go this way you should find another method of forwarding ports over the proxy.
Finally, in my personal opinion, if you can you should change the way you access your backend service. If you were the one to set up the connection then SOCKS proxy is an overkill when all you want to do is to connect to few hosts. If it is a proxy put in place by your company or someone else above you then i would discuss it with the network administrators.
This is about a server A which I use to browse the website pixiv.net.
One day, all my http requests (pings or wget) from this server stopped working (they keep timing out). I concluded it was most likely an IP block from pixiv, blocking the IP of server A.
I luckily have access to another server B which I could use for testing, this one is able to issue requests to pixiv just fine (but I cant use it permanently it's not mine).
To bypass what I thought was an IP block, I tried to issue the HTTP requests through proxies. I've tried a few different ones, courtesy of https://gimmeproxy.com/, but the requests still time out. However, they still work fine from server B even with proxy, which leads me to believe there is nothing wrong with the proxy.
I've concluded that one of the following is true:
I'm misusing wget with proxy and I'm actually not doing anything at all. I'm doing
wget pixiv.net -e use_proxy=yes -e http_proxy=ip:port
Proxies don't help solving my IP block issue.
The original issue is not an IP block. In that case I have no idea what it could be
I need to send (duplicate) traffic from one machine (port) and to two different machines (ports). I need to take care of TCP session as well.
In the beginnig I used em-proxy, but it seems to me that the overhead is quite large (it goes over 50% of cpu).
Then I installed haproxy and I managed to redirect traffic (not to duplicate). The overhead is reasonable (less than 5%).
The problem is that I could not say in haproxy config file the following:
- listen on specific address:port and whatever you find send on the two different
machines:ports and discard the answers from one of them.
Em-proxy code for this is quite simple, but it seems to me that EventMachine generates
a lot of overhead.
Before I dig in haproxy code and try to change (duplicate traffic) I would like
to know is there something similar out there?
Thanks.
I have created a proxy just for this purpose.
https://github.com/chrislusf/teeproxy
Usage
./teeProxy -l :8888 -a localhost:9000 -b localhost:9001
tee-proxy is a reverse proxy. For each incoming request, it clones the request into 2 and then forwards them to 2 servers. The results from server a is returned as usual, but the results from server b is ignored.
tee-proxy handles both GET, POST, and other HTTP methods.
How about the iptables experimental ROUTE target? It has a "tee" option for mirroring traffic:
http://www.netfilter.org/projects/patch-o-matic/pom-external.html#pom-external-ROUTE
Which would let you mirror traffic with something like:
iptables -A PREROUTING -t mangle -p tcp --dport 80 -j ROUTE --gw 1.2.3.4 --tee
iptables -A POSTROUTING -t mangle -p tcp --sport 80 -j ROUTE --gw 1.2.3.4 --tee
The second machine would need to be on the same subnet and would either need to listen on the target IP address (and not reply to arps) or listen promiscuously.
Try https://github.com/agnoster/duplicator.
I tried teeproxy but got strange results with some requests other than GET's.
I have also written a reverse proxy / load balancer for a similar purpose with Node.js (it is just for fun, not production ready at the moment).
https://github.com/losnir/ampel
It is very opinionated, and currently supports:
GET Using round-robin selection (1:1)
POST Using request splitting. There is no concept of "master" and "shadow" -- the first backend that responds is the one that will serve the client request, and then all of the other responses will be discarded.
If someone finds it useful then I can improve it to be more flexible.
I needed something that could tee the TCP traffic as well, but being not intrusive, thus not being able to put something in-between as a reverse proxy for example.
What I did is basically did is use the tcpdump/wireshark logic (packet sniffing) wrap it in a Go process that you can configure to do some things.
For whom it may be helpful the code can be found here: https://github.com/RobinUS2/teecp
I need to monitor my application from incoming http POST and GET requests originating from outside and sometimes inside the machine.
Is this possible?
Been using fiddler but this only does outgoing not incoming (from outside the machine) or have I configured it incorrectly?
This is for my web app that is meant to be receiving a POST from an external server.
What you need to do is configure Fiddler to work as a "reverse proxy"
There are instructions on 2 different ways you can do this on Fiddler's website. Here is a copy of the steps:
Step #0
Before either of the following options will work, you must enable other computers to connect to Fiddler. To do so, click Tools > Fiddler Options > Connections and tick the "Allow remote computers to connect" checkbox. Then close Fiddler.
Option #1: Configure Fiddler as a Reverse-Proxy
Fiddler can be configured so that any traffic sent to http://127.0.0.1:8888 is automatically sent to a different port on the same machine. To set this configuration:
Start REGEDIT
Create a new DWORD named ReverseProxyForPort inside HKCU\SOFTWARE\Microsoft\Fiddler2.
Set the DWORD to the local port you'd like to re-route inbound traffic to (generally port 80 for a standard HTTP server)
Restart Fiddler
Navigate your browser to http://127.0.0.1:8888
Option #2: Write a FiddlerScript rule
Alternatively, you can write a rule that does the same thing.
Say you're running a website on port 80 of a machine named WEBSERVER. You're connecting to the website using Internet Explorer Mobile Edition on a Windows SmartPhone device for which you cannot configure the web proxy. You want to capture the traffic from the phone and the server's response.
Start Fiddler on the WEBSERVER machine, running on the default port of 8888.
Click Tools | Fiddler Options, and ensure the "Allow remote clients to connect" checkbox is checked. Restart if needed.
Choose Rules | Customize Rules.
Inside the OnBeforeRequest handler, add a new line of code:
if (oSession.host.toLowerCase() == "webserver:8888") oSession.host = "webserver:80";
On the SmartPhone, navigate to http://webserver:8888
Requests from the SmartPhone will appear in Fiddler. The requests are forwarded from port 8888 to port 80 where the webserver is running. The responses are sent back through Fiddler to the SmartPhone, which has no idea that the content originally came from port 80.
I would install Microsoft Network Monitor, configure the tool so it would only see HTTP packets (filter the port) and start capturing packets.
You could download it here
Configure Fiddler as a 'reverse proxy' on Windows
(for Mac, see the link in Partizano's comment below)
I know there's already an answer suggesting this, however I want to provide the explanation and instructions for this that Telerik should have provided, and also cover some of the 'gotchas', so here goes:
What does it mean to configure Fiddler as a 'reverse proxy'?
By default, Fiddler only monitors outgoing requests from the machine on which you're running Fiddler
To monitor incoming requests, you need to configure Fiddler to work as a 'reverse proxy'
What this means is that you need to set Fiddler up as a 'proxy' that will intercept incoming http requests that are sent to a specific port (8888) on the machine where you want to listen to the incoming requests. Fiddler will then forward those requests to the web server on the same machine by sending them to the usual port for http requests (usually port 80 or 443 for https). It's actually very quick and easy to do!
The standard way to set this up with Fiddler is to get Fiddler to intercept all request sent to Port '8888' (since this won't normally be used by anything else, although you could just as easily use another port)
You then need to use the registry editor to get Fiddler to forward any http requests that Fiddler receives on port 8888, so that they're forwarded to the standard http port (port 80, port 443 for an https request, or another custom port that your web server is set to listen on)
NOTE: For this to work, any request you want to intercept must be sent to port 8888
You do this by appending :8888 to your hostname, for example like this for an MVC route:
http://myhostname:8888/controller/action
Walkthrough
Ensure Fiddler can accept remote http requests on port 8888:
Run Fiddler as administrator Go to Tools > Fiddler Options > Connections, and ensure that 'Allow remote computers to connect' is checked, and 'Fiddler listens on port' is set to 8888:
Configure Fiddler to forward requests received on port 8888 to port 80
Close Fiddler
Start REGEDIT
Create a new DWORD named ReverseProxyForPort inside HKEY_CURRENT_USER\SOFTWARE\Microsoft\Fiddler2.
Now set the DWORD value to the local port you'd like to re-route inbound traffic to (generally port 80 for a standard HTTP server)
To do this, right-click the DWORD you created and select 'Modify'. Ensure 'Base' is set to 'Decimal' and enter '80' (or another port) as the 'Value data':
Ensure that port 8888 is opened on the firewall
You must ensure that port 8888 is open to external requests (it won't be by default if your server is firewall-protected)
That's it! Fiddler should now be set up as a reverse proxy, to intercept all requests from port 8888 (so that you can view them in Fiddler), and it will then forward them to your web server to actually be handled.
Test a request
Restart Fiddler
To test that Fiddler is intercepting external requests, open a browser on the same machine where you've set up Fiddler as a reverse proxy. Navigate your browser to http://127.0.0.1:8888
This tests making a basic request to to port 8888
You should see the request intercepted by Fiddler
Now you can test a request from another machine, for example by making a request from a browser on another machine like this (where 'remoteHostname' is a hostname on the machine where you've set up Fiddler as a reverse proxy) :
http://remoteHostname:8888/controller/action
Alternatively, you can compose a request by using another instance of Fiddler on a remote machine, using a URL similar to the one above. This will allow you to make either a GET or a POST request.
IMPORTANT: Once you've finished viewing your request(s), go back to Tools > Fiddler Options > Connections and remove the 'Allow remote computers to connect' option, otherwise 3rd parties will be able to bounce traffic through your server
Guys found the perfect way to monitor ALL traffic that is flowing locally between requests from my machine to my machine:
Install Wireshark
When you need to capture traffic that is flowing from a localhost to a localhost then you will struggle to use wireshark as this only monitors incoming traffic on the network card. The way to do this is to add a route to windows that will force all traffic through a gateway and this be captured on the network interface.
To do this, add a route with <ip address> <gateway>:
cmd> route add 192.168.20.30 192.168.20.1
Then run a capture on wireshark (make sure you select the interface that has bytes flowing through it) Then filter.
The newly added routes will come up in black. (as they are local addresses)
Microsoft Message Analyzer is the successor of the Microsoft Network Monitor 3.4
If your http incoming traffic is going to your web server at 58000 port, start the Analyzer in Administrator mode and click new session:
use filter: tcp.Port = 58000 and HTTP
trace scenario: "Local Network Interfaces (Win 8 and earlier)" or "Local Network Interfaces (Win 8.1 and later)" depends on your OS
Parsing Level: Full
You might consider running Fiddler as a reverse proxy, you should be able to get clients to connect to Fiddler's address and then forward the requests from Fiddler to your application.
This will require either a bit of port manipulation or client config, depending on what's easier based on your requirements.
Details of how to do it are here: http://www.fiddler2.com/Fiddler/Help/ReverseProxy.asp
Use TcpView to see ports listening and connections. This will not give you the requests though.
In order to see requests, you need reverse of a proxy which I do not know of any such tools.
Use tracing to give you parts of the requests (first 1KB of the request).
Using Wireshark..
I have not tried this:
http://wiki.wireshark.org/CaptureSetup/Loopback
If that works, you could then filter for http/http contains GET/http contains POST traffic.
You might have to run two Wireshark instances, one capturing local, and one capturing remote. I'm not sure.
You can also try the HTTP Debugger, it has the built-in ability to display incoming HTTP requests and does not require any changes to the system configuration.
So I'm not a very good network person so I was hoping someone could point me in the right direction to figuring out what I am doing wrong.
I am trying to use curl to post a SOAP message. I am running the following:
curl -d "string of xml message" -H "Content-Type:text/xml; charset=utf-8" "[ip]:[port]/[service]"
This results in a 'Connection refused' message.
So I try pinging ip by itself...no problems.
Then I think maybe I need http://[ip]:[port]/[service] so I tried pinging http://[ip] and I get:
unknown host http://[ip] yet if I ping the IP by itself I get no issues.
Any thoughts on where to start debugging this issue?
First of all, ping can't use the HTTP-protocol, you can only ping domain names. Have a look at ping at wikipedia to learn more.
Curl normally doesn't need anything fancy, just begin by typing curl [protocol]:[host]:[port]/[service] and see if you get a response at all. I think that's what you're looking for when you tried to ping the remote address.
Judging by the response of the cURL attempt, you'll know if your attempt was successfull. It probably won't be since it is indeed the connection that was refused, you didn't include bad parameters.
Now, assuming it's a connection problem, try curling something else (a regular domain, like Google.com) to make sure you don't have a connection problem. Then, to learn whether the remote server has a problem, perform the same Curl attempt from another server somewhere (or ask someone else to do it) and see if they, too, are refused to connect. This is a good attempt to circle in around the problem and gain more clarity.
Ping (or ICMP) traffic runs (usually) on a different port than HTTP traffic. HTTP typically runs on port 80.
Try to telnet to the service (the ip and the port) using the following command:
telnet (ip) 80 (without the parens).
If you are able to connect to the service then you have some other issues, however, if the service doesn't let you connect, then you know the service is blocking you from the port (usually 80 for http) on which the service is running.