Is it possible to tell the client to use a different port to request source if not defined?
Example
I serve the client a web page on port 8080. The content of this website contains some links to other sources (e.g. img or style with src). The URI to this source is http://localhost/requested_file. By default the client will request this source on http://localhost:80/requested_file and I want the client to instead request http://localhost:8080/requested_file.
Note: I can manipulate the web server (and client) but NOT the source (e.g. served HTML, CSS and JS)
If not possible with HTTP response headers, maybe there is some other way (e.g. "injection" of JS in the response)?
Assuming this is a linux server you could create a routing rule with iptables from one port to another such as:
-A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8080
COMMIT
Related
I have an NGINX Server set up, I'd like to take a request and forward it to another application on a TCP port.
Let's say I have the following JSON payload
{
"someKey1": 1234,
"someKey2": "a string"
}
This is sent inside query parameters like the following
https://mywebsite.com?payload=%7B%0A%20%22someKey1%22%3A%201234%2C%0A%20%22someKey2%22%3A%20%22a%20string%22%0A%7D
Is there a way to forward that JSON payload to TCP port 1234 natively with NGINX?
Additionally, can I do any pre-processing of the above payload prior to it being forwarded to TCP port 1234. For example, I'd like to covert the above JSON to
someKey1=1234,someKey2="a string"
And then forward this data to TCP port 1234
I understang I'd have to create some sort of REST endpoint using something like springboot to do this, but I'd really like to try and accomplish the above natively with NGINX if possible.
Nginx's primary purpose is HTTP server/proxy.
It can be scripted via ngx_http_lua_module, but for your task it is much simpler to make an app/microservice that will listen HTTP and forward your custom protocol, or modify your app that listens mentioned port to understand HTTP.
When your endpoint talks HTTP - nginx can then be used for routing:
location /some_path/ {
proxy_pass http://localhost:1234/;
}
location /some_other_path/ {
proxy_pass http://localhost:1235/;
}
NGINX is simple web-server, which accepts HTTP requests and forwards them to configured location (may be application server, or any other web-server), and responds back on HTTP to the requester. Data can't be processed inside NGINX.
You can configure forwarding rules in default file under sites-available directory in NGINX installation directory.
Here is the nice tutorial of NGINX configuration which might help you.
Let's say I have this DNS entry: mysite.sample. I am developing, and have a copy of my website running locally in http://localhost:8080. I want this website to be reachable using the (fake) DNS: http://mysite.sample, without being forced to remember in what port this site is running. I can setup /etc/hosts and nginx to do proxing for that, but ... Is there an easier way?
Can I somehow setup a simple DNS entry using /etc/hosts and/or dnsmasq where also a non-standard port (something different than :80/:443) is specified? Without the need to provide extra configuration for nginx?
Or phrased in a simpler way: Is it possible to provide port mappings for dns entries in /etc/hosts or dnsmasq?
DNS has nothing to do with the TCP port. DNS is there to resolv names (e.g. mysite.sample) into IP addresses - kind of like a phone book.
So it's a clear "NO". However, there's another solution and I try to explain it.
When you enter http://mysite.sample:8080 in your browser URL bar, your client (e.g. browser) will first try to resolve mysite.sample (via OS calls) to an IP address. This is where DNS kicks in, as DNS is your name resolver. If that happened, the job of DNS is finished and the browser continues.
This is where the "magic" in HTTP happens. The browser is connecting to the resolved IP address and the desired port (by default 80 for http and 443 for https), is waiting for the connection to be accepted and is then sending the following headers:
GET <resource> HTTP/1.1
Host: mysite.sample:8080
Now the server reads those headers and acts accordingly. Most modern web servers have something called "virtual hosts" (i.e. Apache) or "sites" (i.e. nginx). You can configure multiple vhosts/sites - one for each domain. The web server will then provide the site matching the requested host (which is retreived by the browser from the URL bar and passed to the server via Host HTTP header). This is pure HTTP and has nothing to do with TCP.
If you can't change the port of your origin service (in your case 8080), you might want to setup a new web server in front of your service. This is also called reverse proxy. I recommend reading the NGINX Reverse Proxy docs, but you can also use Apache or any other modern web server.
For nginx, just setup a new site and redirect it to your service:
location mysite.example {
proxy_pass http://127.0.0.1:8080;
}
There is a mechanism in DNS for discovering the ports that a service uses, it is called the Service Record (SRV) which has the form
_service._proto.name. TTL class SRV priority weight port target.
However, to make use of this record you would need to have an application that referenced that record prior to making the call. As Dominique has said, this is not the way HTTP works.
I have written a previous answer that explains some of the background to this, and why HTTP isn't in the standard. (the article discusses WS, but the underlying discussion suggested adding this to the HTTP protocol directly)
Edited to add -
There was actually a draft IETF document exploring an official way to do this, but it never made it past draft stage.
This document specifies a new URI scheme called http+srv which uses a DNS SRV lookup to locate a HTTP server.
There is an specific SO answer here which points to an interesting post here
I need to send (duplicate) traffic from one machine (port) and to two different machines (ports). I need to take care of TCP session as well.
In the beginnig I used em-proxy, but it seems to me that the overhead is quite large (it goes over 50% of cpu).
Then I installed haproxy and I managed to redirect traffic (not to duplicate). The overhead is reasonable (less than 5%).
The problem is that I could not say in haproxy config file the following:
- listen on specific address:port and whatever you find send on the two different
machines:ports and discard the answers from one of them.
Em-proxy code for this is quite simple, but it seems to me that EventMachine generates
a lot of overhead.
Before I dig in haproxy code and try to change (duplicate traffic) I would like
to know is there something similar out there?
Thanks.
I have created a proxy just for this purpose.
https://github.com/chrislusf/teeproxy
Usage
./teeProxy -l :8888 -a localhost:9000 -b localhost:9001
tee-proxy is a reverse proxy. For each incoming request, it clones the request into 2 and then forwards them to 2 servers. The results from server a is returned as usual, but the results from server b is ignored.
tee-proxy handles both GET, POST, and other HTTP methods.
How about the iptables experimental ROUTE target? It has a "tee" option for mirroring traffic:
http://www.netfilter.org/projects/patch-o-matic/pom-external.html#pom-external-ROUTE
Which would let you mirror traffic with something like:
iptables -A PREROUTING -t mangle -p tcp --dport 80 -j ROUTE --gw 1.2.3.4 --tee
iptables -A POSTROUTING -t mangle -p tcp --sport 80 -j ROUTE --gw 1.2.3.4 --tee
The second machine would need to be on the same subnet and would either need to listen on the target IP address (and not reply to arps) or listen promiscuously.
Try https://github.com/agnoster/duplicator.
I tried teeproxy but got strange results with some requests other than GET's.
I have also written a reverse proxy / load balancer for a similar purpose with Node.js (it is just for fun, not production ready at the moment).
https://github.com/losnir/ampel
It is very opinionated, and currently supports:
GET Using round-robin selection (1:1)
POST Using request splitting. There is no concept of "master" and "shadow" -- the first backend that responds is the one that will serve the client request, and then all of the other responses will be discarded.
If someone finds it useful then I can improve it to be more flexible.
I needed something that could tee the TCP traffic as well, but being not intrusive, thus not being able to put something in-between as a reverse proxy for example.
What I did is basically did is use the tcpdump/wireshark logic (packet sniffing) wrap it in a Go process that you can configure to do some things.
For whom it may be helpful the code can be found here: https://github.com/RobinUS2/teecp
I need to monitor my application from incoming http POST and GET requests originating from outside and sometimes inside the machine.
Is this possible?
Been using fiddler but this only does outgoing not incoming (from outside the machine) or have I configured it incorrectly?
This is for my web app that is meant to be receiving a POST from an external server.
What you need to do is configure Fiddler to work as a "reverse proxy"
There are instructions on 2 different ways you can do this on Fiddler's website. Here is a copy of the steps:
Step #0
Before either of the following options will work, you must enable other computers to connect to Fiddler. To do so, click Tools > Fiddler Options > Connections and tick the "Allow remote computers to connect" checkbox. Then close Fiddler.
Option #1: Configure Fiddler as a Reverse-Proxy
Fiddler can be configured so that any traffic sent to http://127.0.0.1:8888 is automatically sent to a different port on the same machine. To set this configuration:
Start REGEDIT
Create a new DWORD named ReverseProxyForPort inside HKCU\SOFTWARE\Microsoft\Fiddler2.
Set the DWORD to the local port you'd like to re-route inbound traffic to (generally port 80 for a standard HTTP server)
Restart Fiddler
Navigate your browser to http://127.0.0.1:8888
Option #2: Write a FiddlerScript rule
Alternatively, you can write a rule that does the same thing.
Say you're running a website on port 80 of a machine named WEBSERVER. You're connecting to the website using Internet Explorer Mobile Edition on a Windows SmartPhone device for which you cannot configure the web proxy. You want to capture the traffic from the phone and the server's response.
Start Fiddler on the WEBSERVER machine, running on the default port of 8888.
Click Tools | Fiddler Options, and ensure the "Allow remote clients to connect" checkbox is checked. Restart if needed.
Choose Rules | Customize Rules.
Inside the OnBeforeRequest handler, add a new line of code:
if (oSession.host.toLowerCase() == "webserver:8888") oSession.host = "webserver:80";
On the SmartPhone, navigate to http://webserver:8888
Requests from the SmartPhone will appear in Fiddler. The requests are forwarded from port 8888 to port 80 where the webserver is running. The responses are sent back through Fiddler to the SmartPhone, which has no idea that the content originally came from port 80.
I would install Microsoft Network Monitor, configure the tool so it would only see HTTP packets (filter the port) and start capturing packets.
You could download it here
Configure Fiddler as a 'reverse proxy' on Windows
(for Mac, see the link in Partizano's comment below)
I know there's already an answer suggesting this, however I want to provide the explanation and instructions for this that Telerik should have provided, and also cover some of the 'gotchas', so here goes:
What does it mean to configure Fiddler as a 'reverse proxy'?
By default, Fiddler only monitors outgoing requests from the machine on which you're running Fiddler
To monitor incoming requests, you need to configure Fiddler to work as a 'reverse proxy'
What this means is that you need to set Fiddler up as a 'proxy' that will intercept incoming http requests that are sent to a specific port (8888) on the machine where you want to listen to the incoming requests. Fiddler will then forward those requests to the web server on the same machine by sending them to the usual port for http requests (usually port 80 or 443 for https). It's actually very quick and easy to do!
The standard way to set this up with Fiddler is to get Fiddler to intercept all request sent to Port '8888' (since this won't normally be used by anything else, although you could just as easily use another port)
You then need to use the registry editor to get Fiddler to forward any http requests that Fiddler receives on port 8888, so that they're forwarded to the standard http port (port 80, port 443 for an https request, or another custom port that your web server is set to listen on)
NOTE: For this to work, any request you want to intercept must be sent to port 8888
You do this by appending :8888 to your hostname, for example like this for an MVC route:
http://myhostname:8888/controller/action
Walkthrough
Ensure Fiddler can accept remote http requests on port 8888:
Run Fiddler as administrator Go to Tools > Fiddler Options > Connections, and ensure that 'Allow remote computers to connect' is checked, and 'Fiddler listens on port' is set to 8888:
Configure Fiddler to forward requests received on port 8888 to port 80
Close Fiddler
Start REGEDIT
Create a new DWORD named ReverseProxyForPort inside HKEY_CURRENT_USER\SOFTWARE\Microsoft\Fiddler2.
Now set the DWORD value to the local port you'd like to re-route inbound traffic to (generally port 80 for a standard HTTP server)
To do this, right-click the DWORD you created and select 'Modify'. Ensure 'Base' is set to 'Decimal' and enter '80' (or another port) as the 'Value data':
Ensure that port 8888 is opened on the firewall
You must ensure that port 8888 is open to external requests (it won't be by default if your server is firewall-protected)
That's it! Fiddler should now be set up as a reverse proxy, to intercept all requests from port 8888 (so that you can view them in Fiddler), and it will then forward them to your web server to actually be handled.
Test a request
Restart Fiddler
To test that Fiddler is intercepting external requests, open a browser on the same machine where you've set up Fiddler as a reverse proxy. Navigate your browser to http://127.0.0.1:8888
This tests making a basic request to to port 8888
You should see the request intercepted by Fiddler
Now you can test a request from another machine, for example by making a request from a browser on another machine like this (where 'remoteHostname' is a hostname on the machine where you've set up Fiddler as a reverse proxy) :
http://remoteHostname:8888/controller/action
Alternatively, you can compose a request by using another instance of Fiddler on a remote machine, using a URL similar to the one above. This will allow you to make either a GET or a POST request.
IMPORTANT: Once you've finished viewing your request(s), go back to Tools > Fiddler Options > Connections and remove the 'Allow remote computers to connect' option, otherwise 3rd parties will be able to bounce traffic through your server
Guys found the perfect way to monitor ALL traffic that is flowing locally between requests from my machine to my machine:
Install Wireshark
When you need to capture traffic that is flowing from a localhost to a localhost then you will struggle to use wireshark as this only monitors incoming traffic on the network card. The way to do this is to add a route to windows that will force all traffic through a gateway and this be captured on the network interface.
To do this, add a route with <ip address> <gateway>:
cmd> route add 192.168.20.30 192.168.20.1
Then run a capture on wireshark (make sure you select the interface that has bytes flowing through it) Then filter.
The newly added routes will come up in black. (as they are local addresses)
Microsoft Message Analyzer is the successor of the Microsoft Network Monitor 3.4
If your http incoming traffic is going to your web server at 58000 port, start the Analyzer in Administrator mode and click new session:
use filter: tcp.Port = 58000 and HTTP
trace scenario: "Local Network Interfaces (Win 8 and earlier)" or "Local Network Interfaces (Win 8.1 and later)" depends on your OS
Parsing Level: Full
You might consider running Fiddler as a reverse proxy, you should be able to get clients to connect to Fiddler's address and then forward the requests from Fiddler to your application.
This will require either a bit of port manipulation or client config, depending on what's easier based on your requirements.
Details of how to do it are here: http://www.fiddler2.com/Fiddler/Help/ReverseProxy.asp
Use TcpView to see ports listening and connections. This will not give you the requests though.
In order to see requests, you need reverse of a proxy which I do not know of any such tools.
Use tracing to give you parts of the requests (first 1KB of the request).
Using Wireshark..
I have not tried this:
http://wiki.wireshark.org/CaptureSetup/Loopback
If that works, you could then filter for http/http contains GET/http contains POST traffic.
You might have to run two Wireshark instances, one capturing local, and one capturing remote. I'm not sure.
You can also try the HTTP Debugger, it has the built-in ability to display incoming HTTP requests and does not require any changes to the system configuration.
I have web application that runs on Tomcat (and gets HTTP requests) and some other backend standalone application that gets only TCP. For some reasons, I can use outside only port 8080. So, I need to get all TCP requests (from outside) to port 8080 and forward HTTP ones to web application on Tomcat and all TCP pure requests (that are not HTTP) - to standalone application. Internal forwarding could be done to any port, e.g. 8181 on Tomcat and 8282 on standalone application. Is it possible to setup such configuration? How it could be done?
Thanks in advance.
TCP and HTTP are protocols in different networking stack layer. If you want to use some application to filter HTTP requests, your application should deal with Application-Layer information, not Network-Layer(like TCP/UDP).
I don't see how this can be possible generally. You could look packet-by-packet, but the middle of an http body can be arbitary so you can't just look at the data of each packet
If any particular client will send you either http or general TCP but not both, can you do this by source-IP address? Do you know the addresses of either the servers that will send you http requests or the ones that will send you TCP requests?
If you don't know the source IPs, you could heuristically look at the first packet from some previously unknown IP and see if it looks like http, then tag that address as containing http traffic.
What is the content/format ot the TCP communication? Is there any pattern you can detect in that?
Y
Perhaps you could do something like this using iptables + L7 filter. Of course this will only work if you run Linux on your box. Also I don't know how recently l7 filter project has been updated.
Java servlet technology is not limited to Http. The servlet interface lets you read in the incoming input stream via ServletRequest.getInputStream(). So you can create an implementation of Servlet interface and map it in web.xml and you are all set to receive any TCP traffic.
Once you have the read the input stream to sniff the content you will want to forward HTTP requests to an HttpServlet. To do this you will need to make sure that the input stream you pass on is positioned at the very beginning of the input.
EDIT: On reading your question once again, I noticed that you don't plan to expose the Tomcat directly on external port as I originally thought. If you are willing to make the tomcat listen on the external port, you can try the approach described above