I work in web development, but I don't have a great understanding of network protocols. I recall hearing an analogy that TCP, HTTP, and SSL can be thought of as a series of nested envelopes around the actual request content.
I also have a fuzzy idea TCP consists of packets, which are verified on the other end. But I'm sort of picturing the HTTP request being chopped into packets, too...
So basically, I don't understand this stuff well at all. Can anybody give a good overview of this? Also, is there a beginner-friendly book or other resource that you'd recommend?
Since I asked this question, I've learned more about this topic, so I'll take a crack at answering it myself.
The easiest way to picture the protocol stack is as a letter, wrapped in a series of envelopes. Each envelope has a different role in getting the letter to its recipient, and envelopes are added and removed as needed along the journey.
The Application Layer
The letter itself is an application-layer request. For example, you've typed "StackOverflow.com" in your browser and pressed enter. Your browser needs to ask the StackOverflow server for its home page. So it writes a letter saying, "Dear StackOverflow, would you please send me your home page?"
If the writer of the letter is your browser, the recipient of the letter is the web server program running on StackOverflow. The browser wants the web server to "write back" with a response in the form of a web page. Both the browser and server are applications - programs running on specific computers.
Because browsers speak HTTP, that's what it uses to make the request: the letter says something like "GET http://stackoverflow.com". The browser also writes down any cookie information it got from StackOverflow last time ("remember me? You told me my login ID was X") and adds some miscellaneous labeled information called "headers" (things like "I'm Firefox" and "I can accept HTML or text" and "it's OK with me if you compress the content with gzip"). All that information will help the server know how to personalize or customize its response.
At that point, the browser is basically done. It hands this letter to the operating system and says, "would you please send this for me?" The OS says, "Sure." It then does some work to connect to StackOverflow (more on that in a minute), then tells the browser, "I'm working on it. By the way, here's a little mail bin I made for you, called a socket. When I hear back from StackOverflow, I'll put its letter in there and you can read it just like a file." The browser then happily awaits the response.
The IP layer
To send the request from the browser to StackOverflow, the operating system has to do several things.
First, it has to look up the address for StackOverflow.com - specifically, the IP address. It does this using DNS (which I won't go into here). Once it knows the IP address, it will know how to wrap the request in one of the "envelopes" called the IP layer.
Why do we need the IP layer? Well, once upon a time, we didn't.
Why we need IP
Have you ever seen an old movie where someone makes a phone call by asking the operator to connect them? The operator would physically connect the wire from Person #1's house to the wire for Person #2's house. Before the protocol stack was invented, connecting computers was a lot like that phone call: you needed a dedicated wire from point to point.
So, for example, if the computer scientists at Stanford wanted to exchange data with the ones at Harvard, they'd pay a bunch of money to rent a dedicated wire between the two places (a "leased line"). Any data that went into one end came out reliably at the other end. However, this was very expensive: imagine paying for a separate line for every place you want to connect to!
People realized that this wouldn't scale up. We needed a way to have a network that was shared by all users, like a giant spiderweb of wires spread out all over the map. That way, each user would only need one connection to the network and could reach any other user through it.
But that presented a problem. If everyone's communications went on the same lines, how would the data get to the right place? Imagine a bunch of letters dumped on a conveyor belt. Obviously, every letter needs to be addressed to someone, or else they can't be delivered.
That's the basic idea of IP: every machine needs to have an IP address that uniquely identifies it. Messages are placed in IP packets, which are like envelopes with addresses and return addresses.
So, once the OS has looked up the IP address for Stackoverflow.com, it puts the HTTP request in an IP envelope. If it's a "long letter", too big for one envelope, the OS cuts it into pieces and puts it in several IP envelopes. Each envelope says something like "FROM: (your IP address); TO: (The Server's IP address." Like the HTTP request, the IP packet has some other miscellaneous header information, which we won't go into here, but the basic idea is just "to" and "from."
So, at this point, the letter is ready to go, right?
The messiness of IP
Not quite. This letter could easily get lost! See, with IP, we no longer have a dedicated line from place to place. If we did, we'd be sure that our letters were getting delivered: as long as the line wasn't broken, everything would go through.
But with IP, everyone's packets get dumped onto conveyor belts and carried along. The belts lead to little sorting stations, called "routers". If you imagine the routers like physical mail centers, you could picture one in, say, New York City.
"Here's a letter headed for Mexico City. I don't know exactly how to get there, but the station in Houston should be able to get it closer, so I'll send it there. Ah, here's a letter that's going to Atlanta. I'll send it to Charlotte; they should be able to forward it a step closer."
Generally this system works OK, but it's not as reliable as having your own dedicated line. Nearly anything could happen en route: a conveyor belt could break or catch fire, and everything on it could be lost. Or one could get bogged down for a while, so that its packets are delivered very late.
Besides that, because these conveyor belts and stations are used by everyone, nobody's letters get treated specially. So what happens if a router gets more letters than it can possibly handle? For a while, it can stack them in a corner (maybe in RAM), but eventually, it runs out of space.
What it does then may seem shocking: it starts throwing them away.
Yep. That's it. You might think that it would at least be kind enough to send back a note to you, saying, "sorry, we couldn't deliver your letter." But it doesn't. If you think about it, if the router is overwhelmed, it's probably because there's too much traffic on the lines already. Adding apology notes would only make the problem worse. So it throws away your packet and doesn't bother telling anyone.
Obviously, this is a problem for our HTTP request. We need it to get there, and we need the response to get back reliably, too.
To make sure it gets there, we want some kind of "delivery confirmation" service. For that, we'll wrap another envelope around our HTTP request before putting into IP packets. That layer is called TCP.
TCP
TCP stands for "transfer control protocol." It exists to control what would otherwise be a messy, error-prone delivery process.
As implied before, TCP lets us add some "delivery confirmation" to this messy delivery system. Before we wrap our HTTP request in IP packets, we first put it into TCP packets. Each one gets a number: packet 1 of 5, 2 of 5, etc. (The numbering scheme is actually more complicated and counts bytes rather than packets, but let's ignore that for now.)
The basic idea of TCP is this:
First, the client and server - in this case, your operating system and the StackOverflow server's operating system - do a "handshake" to establish a "connection". Both words needs quotes because the "handshake" is actually a few messages back and forth, proving that packets can get successfully there and back, and the "connection" is really nothing more than each side deciding that they'll keep track of the packets flowing between them.
Next, they send packets back and forth; the client maybe requesting a web page, and the server maybe sending it back (in as many packets as that takes).
As one side receives packets, it sends back confirmation messages, saying "so far I've received your packets up to packet 100" and so forth. If one party sends packets and doesn't hear a confirmation for a while, it will assume they were lost and re-send them.
(Getting confirmations when things arrive at the other end is better than getting error reports when a router drops things along the way for a couple of reasons. One is that confirmations go back over a working connection, whereas errors would further clog a non-working connection. Another is that we don't have to trust the intermediary routers to do the right thing; the client and server are the ones who care most about this particular conversation, so they're the ones who take charge of being sure that it works.)
Besides making sure that all the data gets to the other end, TCP also makes sure that the received data gets put back into the right order before handing it up the stack, in case earlier packets got resent and arrived later, or packets in the middle took a longer route, or whatever.
That's basically it - having this kind of delivery confirmation makes the unreliable IP network reliable.
Why wasn't it built straight into IP?
UDP
Well, confirmation has a drawback: it makes things slower. If something is missed, it must be repeated. In some cases, that would be a waste of time, because what you really want is a real-time connection. For example, if you're having a phone conversation over IP, or you're playing a real-time game over the internet, you want to know what's happening right now, even if it means you miss a bit of what happened a second ago. If you stop to repeat things, you'll fall out of sync with everyone else. In cases like that, you can use a cousin of TCP called UDP, which doesn't re-send lost packets. UDP stands for "user datagram protocol", but many people think of it as "unreliable data protocol". That's not an insult; sometimes reliability is less important than staying current.
Since both of these are valid use cases, it makes sense that the IP protocol stayed neutral on the issue of reliability; those who use it can choose whether to add reliability or not.
Both TCP and UDP add one other important piece of information to the request: a port number.
Port numbers
Remember, our original request is comes from a browser and is going to a web server program. But the IP protocol only has addresses that specify computers, not the applications running on them. The machine with StackOverflow's web server may also have other server programs that are listening for requests: a database server, an FTP server, etc. When that machine gets the request, how will it know which program should handle it?
It will know because the TCP request has a port number on it. This is just a number, nothing fancy, but by convention, certain numbers are interpreted to mean certain things. For example, using a port number of 80 is a conventional way of saying "this is a request for a web server." Then the server machine's operating system will know to hand that request to the web server program and not, say, the FTP server program.
When the TCP packets start streaming back to your computer, they will also have a port number, to let your machine know which program to give the response to. That number will vary based on the socket that your machine created initially.
Wait, what's a socket?
Sockets
Remember earlier when the browser asked the OS to send the request? The OS said it would set up a "mail bin" for any response it got back. That bin is called a socket.
You can think of a socket sort of like a file. A file is an interface that the OS provides. It says, "you can read and write data here, and I will take care of figuring out how to actually store it on the hard drive or USB key or whatever." The thing that uniquely identifies a file is the combination of path and filename. In other words, you can only have one file located in the same folder with the same name.
Similarly, a socket is an interface the OS provides. It says, "you can write requests here and read responses." The thing that uniquely identifies a socket is the combination of four things:
Destination IP
Destination Port
Source IP
Source Port
So, you can only have one socket on a system with the same combination of all of those. Notice that you could easily have several sockets open to the same destination IP and port - say, StackOverflow's web server - as long as they all have different source ports. The OS will guarantee that they do by choosing an arbitrary source port for each request, which is why you can have several tabs or several browsers all requesting the same web site simultaneously without anything getting confused; the packets coming back all say which port on your computer they're headed for, which lets the OS know "ah, this packet is for tab 3 in Firefox" or whatever.
Summing up so far
We've been thinking of the protocols as a series of envelops wrapped around the letter. In our example, the letter was an HTTP request, which got wrapped in TCP, then in IP. The IP packets get sent to the right destination computer. That computer removes the IP "envelope" and finds a TCP packet inside. The TCP packet has a port number, which lets the operating system know which port to collect its information in. It replies saying that it got that packet, and it puts its contents (the HTTP request) into the correct socket for the appropriate program to read from. When that program writes a reponse to the socket, the OS sends it back to the requester.
So our "stack" is:
An HTTP request (a "letter"). This is the application layer.
Wrapped in TCP packets ("envelopes"). This is the transport layer.
Wrapped in IP packets ("envelopes"). This is the IP layer.
It's important to understand that this stack is totally customizable. All of these "protocols" are just standard ways of doing things. You can put anything you want inside of an IP packet if you think the receiving computer will know what to do with it, and you can put anything you want inside a TCP or UDP packet if you think the receiving application will know what to do with it.
You could even put something else inside your HTTP request. You could say that some JSON data in there is the "phone number exchange protocol," and as long as both ends know what to do with it, that's fine, and you've just added a higher-level protocol.
Of course, there's a limit to how "high" you can go in the stack - that is, you can put a smaller envelope inside HTTP, and a smaller one inside that, etc, but eventually you won't have any room to go smaller; you won't have any bits for actual content.
But you can easily go "lower" in the stack; you can wrap more "envelopes" around the existing ones.
Other protocol layers
Once common "envelope" to wrap around IP is Ethernet. For example, when your computer decides to send IP packets to Google, it wraps them up as we've described so far, but to send them, it gives them to your network card. The network card may then wrap the IP packets in Ethernet packets (or token ring packets, if you've got an antique setup), addressing them to your router and sending them there. Your router removes those Ethernet "envelopes", checks the IP address, decides who the next closest router is, wraps another Ethernet envelope addressed to that router, and sends the packet along.
Other protocols could be wrapped as well. Maybe two devices are only connected wirelessly, so they wrap their Ethernet packets in a Wi-Fi or Bluetooth or 4G protocol. Maybe your packets need to cross a village with no electricity, so someone physically prints the packets on paper with numbered pages, rides them across town on a bicycle, and scans them into another computer in the order of the page numbers. Voila! A print-to-OCR protocol. Or maybe, I don't know, TCP over carrier pigeon would be better.
Conclusion
The protocol stack is a beautiful invention, and it works so well that we generally take it for granted.
It is a great example of abstracting functionality: each layer has its own work to do and can rely on others to deal with the rest.
The application layer is only concerned with applications talking to each other: "Firefox wants to talk to the web server at StackOverflow.com."
The transport layer is only concerned with getting a stream of packets delivered correctly from one app to another: "all the packets from port 123 on machine 1 need to get to port 80 on machine 2".
The IP layer is only concerned with routing individual packets: "this packet needs to get to the following IP address."
The link layer is only concerned with getting packets from one waypoint to the next: "this ethernet packet needs to get from the network card to the router."
The physical layer is only concerned with signal transmission: "these pulses need to be sent over this wire."
(Although these layer terms are borrowed from OSI, OSI was actually a competing standard to TCP/IP, and included things like the "session layer" and "presentation layer" that TCP/IP doesn't use. OSI was intended to be a more sane and standardized alternative to the scrappy hacked-together TCP/IP stack, but while it was still being discussed, TCP/IP was already working and was widely adopted.)
Because the layers can be mixed and matched as needed, the stack is flexible enough to accommodate nearly any use we can think of, so it's probably going to be around for a long time. And hopefully now you can appreciate it a bit more.
For the throughout description of TCP/IP networking (without physical layer, e.g., Ethernet), pick TCP/IP Illustrated by Stevens. If you going to do some low-level network programming, Unix network programming by the same author is the best.
There's a reason you'll often hear of TCP/IP implementations called a "stack". Part of the concept is that you have a low-level protocol (Ethernet, PPP, what-have-you), slightly higher-level protocols built on top of it (IP), and so on. It's quite similar to the OSI model, and can be described in terms of that model, though TCP/IP breaks up the layers just a bit differently. Anyway, programs generally send data using one of the upper-level protocols, and let the TCP/IP stack handle the details of getting the data from point A to point B.
TCP sits on top of IP and lets you think of the data flowing in and out as a pair of streams (one in, one out) rather than getting raw IP packets and having to figure out what to do with them. (Big BIG benefit: it simplifies multiplexing. Without TCP or UDP or the like, IP would be near useless -- only one program could normally communicate with the network at a given time.)
SSL sits on top of TCP, and lets you send data over the stream that TCP provides without having to get involved in the ugly details of encrypting and decrypting data, verifying certificates, etc.
HTTP sits on top of TCP (or SSL, in the case of HTTPS), and provides a way for a client and server to pass entire requests and responses, along with metadata describing them.
Network protocols are formal standards and policies comprised of rules, procedures and formats that define communication between two or more devices over a network. Network protocols govern the end-to-end processes of timely, secure and managed data or network communication.
There are several broad types of networking protocols, including:
• Network communication protocols: Basic data communication protocols, such as TCP/IP and HTTP.
• Network security protocols: Implement security over network communications and include HTTPS, SSL and SFTP.
• Network management protocols: Provide network governance and maintenance and include SNMP and ICMP.
The different layers of the Open Systems Interconnection (OSI) reference model are:
Application layer: This is the upper most layer in the OSI reference model. The application layer provides the means by which application processes can access network services, and is therefore associated with services that include direct support for applications.
Presentation layer: This layer in the OSI reference model deals with specifying the format which should be utilized to enable network data to be communicated between computers in the network. The presentation layer adds formatting, encryption, and data compression to the packet.
Session layer: This layer enables applications that reside on different computers to create and close network sessions. It also manages open network connections, or sessions that are open.
Transport layer: The transport layer is responsible for ensuring that data is delivered in sequence, error-free, and efficiently over the network. The transport layer also identifies duplicated packets, and drops them. Transport layer protocols include Transmission Control Protocol (TCP) and Sequenced Packet Exchange (SPX). These protocols open packets at the receiving computer, and reassemble the original messages as well.
Network layer: This layer of the OSI reference model provides addressing for messages for all networks. It translates your logical addresses and names to physical addresses, and then identifies the preferred route from the source computer to the destination computer.
Data Link layer: The Data Link layer prepares data for the physical connection by defining the means by which software drivers can access the physical medium. The Data Link layer transmits frames from the Network layer to the Physical layer.
Physical layer: This layer places the data on the physical medium which is carrying the data. It is responsible for the actual physical connection between two computers on the network that are exchanging data.
The function of protocols at the sending computer is summarized below:
• Segment data into smaller more manageable chunks or packets.
• Append addressing to the packets.
• Ensure that data is ready for sending via the network interface card (NIC) to the network cable
The function of protocols at the receiving computer is summarized below:
• Remove packets from the network cable, and move the packets through the NIC to the computer.
• Remove all information that relate to the sending of the packet. This is information added to the packet by the sending computer.
• Move the packets to the buffer for the reassembly process.
• Convey the data to the particular application.
Internet Protocol :
Internet protocol suite is the set of communication protocols that implement the protocol stack on which the internet runs. The Internet protocol suite is sometimes called the TCP/IP protocol suite, after TCP\IP, which refers to the important protocols in it, the Transmission Control Protocol(TCP) and the Internet Protocol(IP). The Internet protocol suite can be described by the analogy with the OSI model, but there are some differences. Also not all of the layers correspond well.
Protocol Stack:
A protocol stack is the complete set of protocol layers that work together to provide networking capabilities.
Transmission Control Protocol (TCP):
The Transmission Control Protocol is the core protocol of the internet protocol suite. It originated in the network implementation in which it complemented the Internet Protocol. Therefore the entire suite is commonly referred to as TCP/IP. TCP provides reliable delivery of a stream of octets over an IP network. Ordering and error-checking are main characteristics of the TCP. All major Internet applications such as World Wide Web, email and file transfer rely on TCP.
Internet Protocol(IP):
The Internet Protocol is the principal protocol in the Internet protocol suite for relaying data across networks. Its routing function essentially establishes the internet. Historically it was the connectionless datagram service in the original Transmission Control Program; the other being the connection oriented protocol(TCP). Therefore, the Internet protocol suite is referred as TCP/IP.
Is there a beginner-friendly book or other resource that you'd
recommend?
Data Communications and Networking by Behrouz Forouzan:
This contains introductory material and the explanation is beginner friendly. At the same time, it is not dumbed down and the material gets a bit more challenging as you go on. There are very good diagrams explaining concepts too. The typesetting is awesome and you'll have lots of interesting tips surrounding the content. The chapters are ordered according to the OSI stack as mentioned in other answers here. But a lot of the math and derivations for formulas for protocol efficiencies aren't explained.
Computer Networks by Andrew S. Tanenbaum
Everything found in Behrouz Forouzan + lots of equations.
My recommendation is to read the first book first and if you are particularly curious about the math, go to the second one.
We had computer networking on school and we had to buy this book it really helps. It explains every layer of the OSI model. (From the internetcabel and routers up to the tcp udp protecol layers up to the application layer). If you want to have more basic knowledge of how it all works this is a must read.
I'm writing a Comet-like app using Flex on the client and my own hand-written server.
I need to be able to send short bursts of data from the client at quite a high frequency (e.g. of the order of 10ms between sends).
I also need the server to push short bursts of data at a similarly high frequency.
I'm using NetConnection.call() to send the data to the server, and URLStream (with chunked encoding) to push the data from the server to the client.
What I've found is that the data isn't being sent/received as soon as it's available. For example, in IE, it seems the data is sent every 200ms rather than as soon as NetConnection.call() is called. Similarly, URLStream isn't making the data available as soon as the server is sending it.
Judging by the difference in behaviour between the browsers, it seems as though the Flash Player (version 10) is relying on the host browser to do all the comms. Can anyone confirm this? Update: This is very likely as only the host browser would know about the proxy settings that might be set.
I've tried using the Socket class and there's no problem with speed there: it works perfectly. However, I'd like to be able to use HTTP-based (port 80) connections so that my app can run in heavily fire-walled environments (I tried using a Socket over port 80, but that has its problems).
Incidentally, all development/testing has been done on an internal LAN, so bandwidth/latency is not an issue.
Update: The data being sent/received is in small packets and doesn't need to be in any particular format. For example, I might need to send a short array of Numbers, and this could either be encoded in AMF (e.g. via NetConnection.call()) or could be put into GET parameters (e.g. using sendToURL()). The main point of my question is really to see whether anyone else has experienced the same problem in calling NetConnection/URLStream frequently, and whether there is a workaround (it's also possible that the fault lies with my server code of course, rather than Flash).
Thanks.
Turns out the problem had nothing to do with Flash/Flex or any of the host browsers. The problem was in my server code (written in C++ on Linux), and without access to my source code the cause is hard to find (so I couldn't have hoped for an answer from this forum).
Still - thank you everyone who chipped in.
It was only after looking carefully at the output shown in Wireshark that I noticed the problem, which was twofold:
Nagle's algorithm
I was sending replies in multiple packets by calling write() multiple times (e.g. once for the HTTP response header, and again for the HTTP response body). The server's TCP/IP stack was waiting for an ACK for the first packet before sending the second, but because of Nagle's algorithm the client was waiting 200ms before sending back the ACK to the first packet, so the server took at least 200ms to send the full HTTP response.
The solution is to use send() with the flag MSG_MORE until all the logically connected blocks are written. I could also have used writev() or setsockopt() with TCP_CORK, but it suited my existing code better to use send().
Chunk-encoded streams
I'm using a never-ending HTTP response with chunk encoding to push data back to the client. Naggle's algorithm needs to be turned off here because even if each chunk is written as one packet (using MSG_MORE), the client OS TCP/IP stack will still wait up to 200ms before sending back an ACK, and the server can't push a subsequent chunk until it gets that ACK.
The solution here is to ask the server not to wait for an ACK for each sent packet before sending the next packet, and this is done by calling setsockopt() with the TCP_NODELAY flag.
The above solutions only work on Linux and aren't POSIX-compliant (I think), but that isn't a problem for me.
I'm almost 100% sure the player relies on the browser for such communications. Can't find an official page stating so atm, but check this out for example:
Applications hosting the Flash Player
ActiveX control or Flash Player
plug-in can use the
EnforceLocalSecurity and
DisableLocalSecurity API calls to
control security settings.
Which I think somehow implies the idea. Also, I've suffered some network related bugs on FF/IE only which again points out to the player using each browser for networking (otherwise there wouldn't be such differences).
And regarding your latency problem, I think that if speed is critical, your best bet is sockets. You have some work to do, but seems possible, check out the docs again:
This error occurs in SWF content.
Dispatched if a call to
Socket.connect() attempts to connect
either to a server outside the
caller's security sandbox or to a port
lower than 1024. You can work around
either problem by using a cross-domain
policy file on the server.
HTH,
Juan