Requesting the local machine fails right away:
telnet localhost 65535
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
while it runs forever when requesting google or any other remote machine:
telnet www.google.com 65535
Trying 2a00:1450:4007:812::2004...
I suppose non standard ports should be closed on web servers. If yes, telnet should end up with a "Connection refused" right away as well. right?
Actually, this can be caused by multiple sources. A not so known, but common one:
There are some firewalls allowing you to delay requests to specific ports.
Think about this: Scanning one IP address for all ports only takes a few seconds.
If you delay the response (in case the port is not opended for instance) it will take a potential attacker much longer to scan all ports.
You could argue that the attacker could count anything taking let´s say 5 seconds or longer as timeout, but, there are applications, SMTP servers for instance, that often actually respond only after 20 seconds or so because of this.
May protocols are trying to be attacked and if you generally define that your mailserver will only respond after 20 seconds, that does not really matter to mails in most cases, most attacks will already count this as timeout and won´t even notice that a mailserver is running there whereas the clients get configured for timeouts of 30 seconds or so and they can connect.
Another common thing is that the windows telnet client won´t really post the output until you press any key.
Related
localhost Tcp connections randomly disconnecting. Running tcp client/server apps with simple echoing the server sees the local client connection breaking but any client connection over the network stays up. All local host connections get severed. One can immediately reconnect with no problem. This happens randomly and relatively rarely but is still problematic in my environment. Has anyone observed this pattern?
Instead of "Localhost" use 127.0.0.1
Localhost is used in Linux. Windows works better with 127.0.0.1.
Beats me, Google for explanation or just use the knowledge.
Windows Server 2008 R2, fully patched and updated.
I have 4 static IPs on a dedicated server. I will refer to them as follows:
x.x.x.x
y.y.y.y
z.z.z.z
a.a.a.a
x.x.x.x is the default external and internal IP address of the server.
All external IPs are the same as the internal IPs, all running on the same NIC.
x.x.x.x and y.y.y.y were running on port 80 for HTTP through IIS, with different host headers handling the destinations. That worked perfectly.
I recently added two new IP addresses, z.z.z.z and a.a.a.a for a different application that uses two ports, but we want external port 80 traffic to translate to the internal ports it is using.
We want incoming traffic to work as follows:
Incoming traffic on x.x.x.x:80 map to x.x.x.x:8080
Incoming traffic on y.y.y.y:80 map to y.y.y.y:8080
Incoming traffic on z.z.z.z:80 map to y.y.y.y:8088
Incoming traffic on a.a.a.a:80 map to y.y.y.y:8089
We changed the binding in IIS to only listen on the specific IPs and internal ports so that port 80 was only being listened to by netsh portproxy.
We have been able to accomplish this with 4 separate netsh portproxy rules and everything works great. All traffic to the two HTTP IPs works fine, and the traffic for the other two IPs to the other two internal ports get routed properly as well.
The problem is that everything works as expected, but occasionally something hangs, usually around 4 PM EST, and the websites are no longer available. There are no application pool or website crashes. Just ports no longer routing.
When it hangs the easiest fix is to run "portproxy reset" and re-create the portproxy rules through a batch file and everything works again.
I guess my question is if there is some kind of idle timeout built into netsh portproxy or possibly some type of buffer overrun protection.
Not a single log shows any faults.
The application pools in IIS were adjusted to recycle at shorter intervals (30 minutes) to prevent any long running worker processes in case that was the issue. Same result.
I can easily create a Windows service to check the port status at a very short interval and reset the specific portproxy rule, but this is not ideal, as there is still the potential for packet loss and unavailability of the services (even if for only a few seconds) requiring the HTTP requests to be re-sent.
Again, I should reiterate that everything works great until a certain point in the day, and that this has absolutely nothing to do with the Windows Firewall, as we get the exact same results with it on or off. There are no apparent DDoS or other types of attacks either. All separate websites and other applications still run on their internal ports when the portproxy hangs (i.e. accessing http://example.com:8080 still works without issue).
The point of failure is the netsh portproxy.
Has anyone experienced similar issues? I am considering adding a Fortinet hardware firewall that has this functionality built in, but I am wondering if that will handle it any better than what is already in place.
I am pinging a working site from my cmd and I get this:
ping site.com
Pinging site.com [x.x.x.x] with 32 bytes of data:
Reply from x.x.x.x: Destination host unreachable.
Reply from x.x.x.x: Destination host unreachable.
Reply from x.x.x.x: Destination host unreachable.
Reply from x.x.x.x: Destination host unreachable.
Ping statistics for x.x.x.x:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
I have censored the IP and domain for obvious reasons. The site is accessible from my browser in this machine.
My question is, how/why would they do this?
ping is an ICMP command request which companies/organisations can block using their firewall for security reasons/purposes. It has nothing to do with website's accessibility using browser.
They are receiving your request and not taking any action further; so it says that destination host is unreachable!
So even though you can visit the organisation's website on the web-browser, your ping request can be blocked for the afore-mentioned reason.
The next diagnostic step is traceroute. To get even more hard-core, use a more complicated tool like nmap or hping2 that lets you send a variety of packets.
Assuming nothing out-of-the-box strange is going on, some intermediate host (or even the destination host) is dropping your ICMP echo packet and responding with ICMP destination unreachable.
This is confusing, because that's not really the problem. However, one big reason this happens is that some routers, even high-end ones, have limits on the kinds of ICMP errors that they can generate, and sometimes an administrator will configure it in an annoying way like this. Another common manifestation of the same issue is intermediate routers dropping TCP packets with RST, making ports look closed instead of just blocked by a firewall.
If it's on your network, consider asking the administrator to switch it to a less confusing message, such as administratively prohibited.
If you still want to ping, consider using a ping utility that uses an alternate packet type, such as hping2, as mentioned above.
Given the foregoing, blocking ping is pretty much a TSA-style security measure. It's tremendously annoying in practice for anyone trying to troubleshoot the site, but it doesn't actually prevent pinging using some other packet type. Hypothetically, it might defeat the dumbest of host up/down scanners, but if port 80 is open, any real scan is still going to find it.
As an alternative explanation, it is possible that the host actually is unreachable using your current routing table, and the browser is finding it some other way. For instance, your network might have no default route to the Internet, but the browser is using a proxy server, that does have access. This kind of configuration is used by some sites, generally to prevent rogue malware or hackers on your network from being able to trivially call out.
I have two applications that talk via TCP, both of which run on Windows XP machines. The client is a third-party application for which I have only the executable, no source. The IP address of the server it connects to is set in a text configuration file. The server is an application I am writing.
All netmasks are 255.255.255.0.
In all cases, the client runs on 192.168.142.202.
I am seeing a case where if I run my server on 192.168.142.207, everything works, but if I move my server over to another machine on the same subnet (192.168.142.105), everything does not work fine. Specifically, the connection does not seem to get properly established. I have looked at what's going on in Wireshark and would like to request assistance interpeting what I see.
On the server side, I see the 3-way handshake: SYN, SYN/ACK, ACK. I get no error codes on the return of accept(), and netstat shows the connection as established.
On the client side, the connection does not seem to be established properly. This causes the client to reconnect periodically, and it will also occasionally close all of the not-correctly-connected sockets that get created as a result. When I look at the client side in Wireshark, I most often see a SYN, SYN, SYN pattern, rather that the expected 3-way handshake. Occassionally, the 3-way handshake does appear, but even then, the client doesn't seem to be happy with the connection because it closes it.
I will note that there are actually two TCP connections between the client and server. The other connection (i.e. not the problematic connection I described above) works just fine. The problematic connection has listening port 5004; the good connection has listening port 1234.
I have placed both .txt and .pcap versions of the client and server Wireshark captures at this link: https://skydrive.live.com/redir.aspx?cid=c5beaf58ac752bb0&resid=C5BEAF58AC752BB0!105&parid=root
As far as the physical network setup goes, there is one switch in between the client and server in the case that works, and there are two switches in between the client and server in the case that doesn't work. All ping tests are successful. There are no wireless connections involved; everything is wired.
All firewalls are off.
Does anybody have any thoughts on either what the problem is or what further data I could gather to solve the problem?
Well, it appears this is not a network or network programming problem at all. I've figured out by trial and error that the third-party software that connects to me wants the machine it runs on to have a smaller IP address than the machine my software runs on. This seems completely arbitrary to me, but empirically, this very strongly appears to be the case. Arghhhh............
Thanks to any and all who may have spent time poring over the Wiresharks dumps I provided...
Based on my understanding, port numbers are just like telephone extensions. Just as a business telephone switchboard can use a main phone number and assign each employee an extension number (like x100, x101, etc.), so a computer has a main address and a set of port numbers to handle incoming and outgoing connections.
But the question is:
On what basis is a port number assigned? A process or an application?
Based on my experience with firewall, I usually open a port for a specific application. So port number should be assigned on an application's basis. But what if there're multiple instances of the same application running on a single machine. Each of the instances uses the same port number. So if a message is arrived at that port number, how could the system tell which instance should the message go?
And another question also related to port.
If a web server is setup to listen on port 80, client browser should always contact the 80 port. I am not sure if the following illustration of the communication between a web browser and the web server is correct.
Client Browser sent request to Server, the message should contain info like this:
To: < ServerAddress:80 >
From: < ClientAddress:XXX >
Server sent reponse to Client Browser like this:
To: < ClientAddress:XXX >
From: < ServerAddress:80 >
So the question is, will the server pick other port numbers for sending messages to client? Because I think a single 80 port doesn't look enough.
Add - 1 - 21:16 2010/12/19
In my above post, the word "application" represents a static program file that the system knows. Multiple instnaces of this application could be launched, which are multiple "processes"
Each client connection will be represented by a socket on the server. Sockets are uniquely represented by the combination of the following 4 pieces of information:
Peer IP address
Peer port
Local IP address
Local port
The client chooses a random port, so if there are multiple connections from one client to the same server/port, the connections will still differ by the client's port.
If there are multiple web server applications running on the same server, they will have to listen on different ports or the server will need to have multiple IP addresses.
On a computer, only one process can be listening on a specific port number. For example, if an Apache process is listening on port 80, no other application can also listen on port 80.
Apache usually pre-forks several processes, only one of those is listening on port 80. The job of that process is to hand over the processing for any connection to one of the pool of other Apache processes as quickly and efficiently as it can.
Each of many concurrent connections to port 80 is distinguished by it's source IP-address and by the source TCP port number (which the client computer chooses randomly from the set not in use).
(Edit)
I was pretty sure that webservers have one process (or thread) listening which accepts incoming connections and passes corresponding filehandles to the worker processes (or threads). EJP advises that this is not so.
Apache seems to have several different multi-processing modules that affect how it spreads the load of responding to multiple concurrent requests. For example: MPM Prefork and MPM Worker
Jeff Pozkaner wrote an overview of HTTP server design that I found interesting:
The basic operation of a web server is to accept a request and send back a response. The first web servers were probably written to do exactly that. Their users no doubt noticed very quickly that while the server was sending a response to someone else, they couldn't get their own requests serviced. There would have been long annoying pauses.
The second generation of web servers addressed this problem by forking off a child process for each request. …
A slight variant of this type of server uses "lightweight processes" or "threads" instead of full-blown Unix processes. …
The third generation of servers is called "pre-forking". Instead of starting a new subprocess for each request, they have a pool of subprocesses that they keep around and re-use. …
The fourth generation. One process only. No non-portable threads/LWPs. Sends multiple files concurrently using non-blocking I/O, calling select()/poll()/kqueue() to tell which ones are ready for more data. …
Network stack distinguishes TCP connections by triple <source IP,source port,destination port>, so knowing client address and port is enough to work correctly.
What is the application, if it is not a process? In firewalls you open ports for executables. It may be considered as an application, and it is a process when it is running.
Multiple listeners cannot listen to the same port. The same process can listen to multiple ports.
Ports are assigned to the listeners. Depending on the firewall (and its configuration) you can allow the process (executable) to listen several ports, or to create several exceptions for the same process listening to multiple ports.
I'm not sure what you mean by the difference between a "process" and an "application". Everything is just code executing on your box.
Anyway, a process/application will listen/bind to whatever port number the authors of the application have configured. By convention, many port numbers are reserved for particular types of application - that is applications which communicate using a particular protocol. So for example web servers which use HTTP typically run on port 80. SMTP servers run on port 22. HTTPS is 443 and so on.
Of course you can configure your web server (e.g apache httpd) to run on whatever port you like - but your client needs to know else it will assume port 80.
Two processes/applications may not bind to the same port. If you try to start another process/application on a port already in use you'll get an error: cannot bind to port or something to that effect.
will the server pick other port
numbers for sending messages to
client?
No. All the accepted sockets use the same server-side port number as the original listening socket. The identifying tuple mentioned above disambiguates this so as to make each connection unique.