This could not be the right place, as it's not about pure programming;
nevertheless, as a simple web developer I find myself quite
ignorant on the subject of networking(Wikipedia usually mix
different subjects on the matter), and I feel as it is a "must" to know.
I sort of have an image of what happens when you write google.com
on your browser, and I don't know the whole process(I have a modem,
a router and a few computers connected to it. let's use my case for an example):
You write characters into chrome ->
there is some character encoding done to translate the address(ASCII or else) ->
DNS does something, not sure ->
your router receives a digital request from a computer's internet cable/WIFI, it saves the internal IPV4 address of
the sender in order to know to which computer to respond back. it sends the digital data to the modem ->
your modem receives digital data, and translates it from digital to analog ->
now your network provider does some work - >
the google server receives a request from an IP address - >
not sure how the google server handles the data, nevertheless it sends back data ->
service provider - > router gets translated digital data from the modem and remembers who sent the request, and sends it to the right person.
in order to optimize a web server or maybe to write a better code which involves networking, perhaps each beginner(such as myself) needs to understand this first? Thank you for your time.
EDIT: I did read wikipedia's OSI model, though it's not quite as helpful as I thought it would.
i will try to explain the idea, although its may be much ,more complicate - it depends on how deep you want to go ...
you write "www.stackoverflow.com"
your OS will try to resolve the www.stackoverflow.com to an IP address
since your OS probably cant, it needs to ask a DNS server
assuming you use an external DNS ( say IP=5.5.5.5 and your IP=10.10.10.10 which is on a different networks ), your OS will check if it knows how to reach 5.5.5.5
a default route 0.0.0.0/0 exists on your PC (this is also known as 'default-gw' which includes ALL internet, it points to your local router
an IP packet will be sent to the router MAC address with the DNS IP address in the destination
your router will probably change your private IP address to its own public IP address and will sends it to the ISP
ISP will route it to the internet until it reaches 5.5.5.5 which is the DNS
DNS will reply back resolving stackoverflow.com to an IP address
your PC now knows how to send packets to stackoverflow.com
packet will be sent to stackoverflow ip address (104.16.36.249) to port 80 (http)
stackoverflow web server listen to requests on port 80
once a packet arrives it will generate a response packet
it will send it back to you exactly in the same way
all that traffic can be seen with a network capture utility like wireshark, u can use those commands (windows) to verify...
ping stackoverflow.com
netstat -rn
ipconfig
nslookup
tracert -d
Related
I'm new to the networking world and I'm using Wireshark to learn stuffs about the network.
I was hanging around in Wireshark while I was using my VPN for circumventing the internet filter (living in a dictatorship country).
While I was using the VPN I see packets that their source or destination IP was neither my IP nor VPN server IP.
I considered two things about my problem:
1- I know my VPN uses its VPN server to send my whole packets to that server then forward them to my deserved destination, then take the response and send it to my client (is that correct?).
2- In Wireshark, I just can see the packets that their either source or destination address is my IP address.
My packets before using VPN:
As you can see, their either source or destination IP is my IP (192.168.1.101).
After using VPN, VPN:
Wireshark:
I tried filtering packets with my IP like this ip.addr == 192.168.1.101 to see only the packets that their either source or destination IP was my private IP, and guess what? nothing was changed. Why did this happen? Wasn't that supposed to filter my packets?
I also checked those packets' MAC and compared them with my MAC, and they were identical.
So my main question is why these kinds of wandering packets were showed by Wireshark? Is my VPN client insecure and trying to sending my data to other places? Or I'm wrong about how my VPN works?
Also, I appreciate it If somebody tells me more explanation about how my VPN or Wireshark works so that I have these types of packets in my Wireshark.
First of all your considerations, point no.1 is right, that's how a typical VPN works and point no.2 is wrong.
Wireshark can capture any packet (any source IP or destination IP) flowing in and out of a network adapter technically known as NIC card. NIC cards are the way through which you can connect to a network (Internet). A quick brief of NIC over here
A typical VPN client software would form a virtual NIC to encrypt and send your traffic through it. A quick brief of VPNs over here
So, to see the required traffic in Wireshark, you must capture traffic from an appropriate interface (NIC card). The answer over here might be helpful.
I'm trying to learn the basics of ARP/TCP/HTTP (in sort of a scatter-shot way).
As an example, what happens when I go to google.com and do a search?
My understanding so far:
For my machine to communicate with others (the gateway in this case),
it may need to do an ARP Broadcast (if it doesn't already have the
MAC address in the ARP cache)
It then needs to resolve google.com's IP address. It does this by
contacting the DNS server. (I'm not completely sure how it knows
where the DNS server is? Or is it the gateway that knows?)
This involves communication through the TCP protocol since HTTP is
built on it (TCP handshake: SYN, SYN/ACK, ACK, then requests for
content, then RST, RST/ACK, ACK)
To actually load a webpage, the browser gets the index.html, parses
it, then sends more requests based on what it needs? (images,etc)
And finally, to do the actual google search, I don't understand how
the browser knows to communicate "I typed something in the search box
and hit Enter".
Does this seem about right? / Did I get anything wrong or leave out anything crucial?
Firstly try to understand that your home router is two devices: a switch and a router.
Focus on these facts:
The switch connects all the devices in your LAN together(including the router).
The router merely connects your switch(LAN) with the ISP(WAN).
Your LAN is essentially an Ethernet network which works with MAC addresses.
For my machine to communicate with others (the gateway in this case),
it may need to do an ARP Broadcast (if it doesn't already have the MAC
address in the ARP cache)
Correct.
When you want to send a file from your dekstop to your laptop, you do not want to go through the router. You want to go through the switch, as that is faster(lower layer). However you only know the IP of the laptop in your network. For that reason you need to get its MAC address. That's where ARP kicks in.
In this case you would broadcast the ARP request in the LAN until someone responds to you. This could be the router or any other device connected to the switch.
It then needs to resolve google.com's IP address. It does this by
contacting the DNS server. (I'm not completely sure how it knows where
the DNS server is? Or is it the gateway that knows?)
If you use DHCP, then that has already provided you with the IP of the DNS server. If not, then it means that you manually provided the IP of the DNS. So the IP of the DNS server is stored locally on your computer.
Making a DNS request is just about putting its IP in the packet with the request and forwarding the packet to the network.
Sidenote: DHCP also provides the IP address of the router.
This involves communication through the TCP protocol since HTTP is
built on it (TCP handshake: SYN, SYN/ACK, ACK, then requests for
content, then RST, RST/ACK, ACK)
Yes. To clarify things: When your computer sends the request
FRAME[IP[TCP[GET www.google.com]]]
The frame is being sent to your LAN's switch which forwards it to the MAC of the router. Your router will open the frame to check the destination IP and route it accordingly(in this case to the WAN). Finally when the frame arrives at the server, the server will open the TCP segment and read the payload, which is the HTTP message. The ACK/SYN etc. messages are being processed just by your computer and the server and not any router or switch.
To actually load a webpage, the browser gets the index.html, parses
it, then sends more requests based on what it needs? (images,etc)
Yes. An HTML file is essentially a tree structure which can have embedded resources like images, javafiles, CSS etc. For each such resource a new request has to be sent.
Once your browser gets all these recourses, it will render the webpage.
And finally, to do the actual google search, I don't understand how
the browser knows to communicate "I typed something in the search box
and hit Enter".
When you type a single character, it is being sent to the server. The server then responds with its suggestions. Easy as that.
References(good reads):
http://www.tcpipguide.com/free/t_TheNeedForAddressResolution.htm
http://www.howtogeek.com/99001/htg-explains-routers-and-switches/
http://www.eventhelix.com/realtimemantra/networking/ip_routing.htm#.UsrYAvim3yO
http://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol
I thought I understood the whole thing about NAT etc but now I came to a problem.
First what I assumed:
Because there are not enough IPv4 addresses available we need another system.
The devices of today at home for connecting to the internet are a combination of:
1) A modem at the physical-level to change the type of signals on the wire.
2) A switch at link-level so you can connect multiple computers to the device
3) A router to connect all the computers to the internet and go beyond your home-subnet etc.
4) A NAT to allow all the internal computers to connect to the outside
5) A portforwarder to let connections from the outside to the internal network
What I call a NAT:
When making a request to the outside: the NAT-part of the device changes the source-port and the source-ip of the request coming from an internal computer. The new source-ip will be your public-ip. The NAT-part will hold a record in a table with this mapping: "original-ip, original-port, new-port".
When a reponse comes back, the NAT will check the destination-port and compare this with the new-ports is in his table. If it finds a match the NAT will replace the destination-ip with original-ip and new-port with original-port. As a consequence the response will be forwarded to the internal computer that made the request.
So, the NAT-part is for when a connection is initialized from the inside. When this request traverses the NAT, 2 things are changed: source-ip and source-port.
Then the portforwarder:
This part of the device will accept connections initialized in the outside-world to your network. It will look at the destination-port of the incoming request and by making a rule for that port-number it may change the destination-port and the destination-ip of the request to an internal ip. With these rules a request from the outside can connect to a computer on your internal network and thus the portforwarder changes 2 things: the destination-ip and the destination-port.
A: Before I ask my question, how is this explanation?
Now my problem is with the response after a request came from the outside through the portforwarder. Assume the right rules are made and a request came through portforwarding on an internal computer. So in the portforwarder the destination-ip was changed to the internal-ip of the computer and the destination-port was changed to the port where the service is running on. If this internal-computer is a webserver it will generate a response. So the destination-ip will be the request's source-ip and the destination-port will be the request's source-port. The source-ip will be the internal-ip of the computer and the source-port will be the port of the service.
Now that response has to go to the outside. So I assume it goes through the NAT to the outside?
So after passing the NAT, the source-ip will be the public-ip and the source-port will be random. Now I tested this with wireshark. I contacted a webserver behind a NAT and I saw the reponse was coming from port 80 ?! How is this possible? This indicates that the response of the forwarded request did not pass the NAT?
I rethought the concept and my new hypothesis is that when a connection is initialized from the outside, it will pass the portforwarder and reach the right computer. This will create a response and when this response reaches our "all-in-one"device, this device can recognize it forwarded the request of the response and will not change the source-port.
B: Is this indeed the case or is it done in another way?
Wikipedia says about portforwarding: "The source address and port are, in this case, left unchanged. When used on machines that are not the default gateway of the network, the source address must be changed to be the address of the translating machine, or packets will bypass the translator and the connection will fail." (http://en.wikipedia.org/wiki/Port_forwarding)
This confirms that the response of a forwarded request MUST go through the portforwarder again and not through the NAT so the source-port wont be changed. The portforwarder will change the source-ip to the public-ip.
Can someone verify this or give me another explanation than mine?
Now I tested this with wireshark. I contacted a webserver behind a NAT
and I saw the reponse was coming from port 80 ?! How is this possible?
This indicates that the response of the forwarded request did not pass
the NAT?
The webserver inside the NAT does not have to be running on port 80. It certainly is set up at the NAT to port forward and respond as if it were at port 80, but that doesn't mean much about the port the web server is actually running on.
Here is some ASCII "art" that may help.
**Internal Network** **NAT Router** **External Computer**
Web Server running at IP 9.9.9.9 port 80 IP 20.20.20.20
IP 192.168.1.7 port 4567
Request web page at 9.9.9.9:80
Forwards port 80 traffic
to 192.168.1.7:4567
Replies with the web page
Puts 9.9.9.9:80 in the
source field and sends
the page on
Gets the page from "9.9.9.9:80"
even though it actually came
from 192.168.1.7:4567
Im not too sure why this wont work. My application works fine if the client and server are ran on the same PC hence the 127.0.0.1 but it wont connect to my other laptop using IP 82.41.108.125 which is the IP of that device.
Any reason why this is happening?
Your firewall could quite possibly be blocking the port assuming all of your coding is working fine - try opening up the port number you're running the application on (in the client/server comptuers' firewall options).
Some additional information that should shed some light on things
Networked devices use ARP (Address Resolution Protocol) and RARP (Reverse Address Resolution Protocol) to map IPs to link layer (MAC) addresses and back. Your network interface card (the thing that plugs into your ethernet cable) will have a unique MAC address on both computers. Each PC has an IP configured for it which is used by higher level protocols (those in the Network Layer).
Computer A will know computer B's IP address (and it will be different from computer A's) if your program works correctly. When computer A goes to send to comptuer B, assuming they're directly connected and not going through a router or something in between, computer A's link layer will need to translate that IP for comptuer B into a MAC address it can use. It does this by sending out a broadcast to all network PCs on the same sub network asking "Hey, is this your IP!?" essentially. The one that has an IP matching the broadcasted one yes, "Yes, it's mine - and here's my MAC address so you can talk directly to me."
So, if two computers have the same IP this all goes to hell :) don't do it - give them unique IPs and make sure that comptuer A transmits to the same UDP port and IP that computer B is listening on as well. communications work in {IP, Port} pairs - its like a telephone number and area code.
Also...
As said by EJP - UDP is a connectionless protocol - computer A just sends to computer B and hopes that computer B is listening correctly. If computer B wasn't listening or was but wasn't in the right state to process the data, the data will simply be dropped and lost. Computer A will not know that this happened. If you want reliable communications where it will keep trying and you will have assurance that compuer B received computer A's data then use TCP instead - it does a 3 way handshake to establish a connection and uses acknowledgements to ensure data is retransmitted when it doesn't reach the other end.
It could be that you're listening on localhost (127.0.0.1) and don't accept other traffic. Use a wildcard address.
Check on the server with netstat -npu.
If that's not the case then it's probably firewall issue, verify with tcpdump or wireshark.
Check firewall on your laptop... Most of the time that acts as spoilsport.
I've noticed recently that I don't have to forward ports for mmorpg's that I play.
I'm thinking about working on a game that people can play online and had a question.
Why is this the case given its a two way socket connection that is constantly sending data back and forth? Doesn't their server need to get through my firewall in order to connect to me?
TCP crash lesson: TCP is a two-way protocol. The challenge is that at least one host needs to initiate. Since within an MMORPG, your own computer is never acting as a server, nobody has to connect to it. All the information about game state is passed through the company's public facing servers that have public facing IP addresses (and hey, maybe they actually use port forwarding there, just to confuse my explanation... but you never have to see their pretty network internals, proxies, and other wizardry.).
Anyway, when you connect to Stackoverflow, you're making one outbound connection that requests data from the server, and then over that same connection you're receiving it back. Same exact scenario, only with a webserver instead of a game.
UPnP allows you to tackle many routers. There's also NAT Punch-Through if you have access to a third party that isn't behind a router.
Either way, port forwarding is only necessary if you wish to act as a server (or the sender in a P2P relationship). A client does not need to forward ports.
You don't need to forward ports to access the web either, despite data coming in as well.
When you make an outgoing TCP connection, your NAT router puts the connection in a table, so that when data comes in, it knows what machine in your LAN to send the packet to.
Everyone mentions TCP, but NAT works for UDP as well: The first outgoing UDP packet associates that source port with the internal IP address, and your NAT device will forward incoming traffic to that port to the correct host on the internal network.
In other words, if your computer requests the connection (outbound) first, the router opens up the port automatically, on the assumption that you're going to want data back. But if you want remote users to connect to your computer without your computer requesting it, the router would normally drop the packets since it wouldn't know where to send them (they were unsolicited). So instead, you need to tell the router to deliver any unsolicited packets at port N to your computer.
Sorry to add another answer so late, and I know one was already accepted, but I personally found the other answers to be more confusing than this simple explanation.