I am learning Computer Networking this semester, on which I find it quite interesting in learning why Internet is designed like today. And I also enjoy reading paper referred to in the teaching slides, like End-to-End Argument in System Design and The Design Philosophy of the DARPA Internet Protocols.
Could you recommend some other interesting paper/articles to me, especially those related to higher layers like TCP/IP protocols?
Many thanks for your reply.
Roy Thomas Fielding wrote his dissertation at UCI about Architectural Styles and
the Design of Network-based Software Architectures (2000), which has more focus on applications in a networked environment; but I would recommend it, since it's related to your request about learning why Internet is designed like today.
Link: Architectural Styles and the Design of Network-based Software Architectures
Generally, the books by Andrew Tanenbaum are worth reading. They are normally used as standard lecture in networking courses.
On the other hand, they are rather basics.
One classic book is "Interconnections 2nd edition" by Radia Perlman.
Another very good one is "Routing in the Internet" by Christian Huitema.
If you are serious about learning how the Internet works, you also need to know about BGP. A decent introduction is "BGP4 Inter-Domain Routing in the Internet" by John Stewart.
A more advanced topic is MPLS. A very good book on this is "MPLS-Enabled Applications" by Ina Minei and Julian Lucek.
Another more advanced topic is multicast. I could recommend "Interdomain Multicast Routing" by Brian Edwards and others.
The books published by Cisco press tend to be good (although obviously vendor-specific) if you are interested in more practical details of how to configure network equipment.
Finally, "Unix Network Programming, Volume 1" by Richard Stevens is a must read if you want to do network programming.
http://www.cs.ucsb.edu/~almeroth/classes/F05.276/papers/vegas.pdf
I'm not sure if this is exactly what you were looking for, but there are some really interesting papers out there on peer-to-peer networking.
Some examples:
Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications
Incentives Build Robustness in BitTorrent
Protecting Free Expression Online with Freenet
If you want an introduction to what (not why) the TCP/IP protocols are, a classic book is TCP/IP Illustrated, Volume 1: The Protocols.
Related
I'm interested in understanding the networking technology behind video games -- specifically I'm interested in large multiplayer games and how companies are able to simulate real-time connection in these games.
What technology is generally used? Is it just WebSockets or is it something more sophisticated?
So an up and coming technology being used to implement real-time communications in video games is Web Real-time Communications (WebRTC). It's just beginning to be used in the entertainment space for video games, live streaming, and more.
BlogGeek.Me goes into a few specific instances where WebRTC is valuable for gaming. Games like Agar.io, Deip.io and others were built with WebRTC way back before 2016. Definitely something to look into when building a video game with peer-to-peer communication.
As a note - if I had to guess, this question was most likely downvoted because it is too vague, could elicit spam and personal opinions, and lacked research by the poster beforehand. There are a lot of articles on the underlying technology behind video games available with a quick Google search, which is outside of the scope of questions/answers on StackOverflow.
This may not be the typical stackoverflow question.
A colleague of mine has been speculating that flow-based routing is going to be the next big thing in networking. Openflow provides the technology to use low cost switches in large application, IT data-centers, etc; replacing Cisco, HP, etc switch and routers. The theory is that you can create a hierarchy these openflow switches with simple configuration, eg. no spanning tree. Open flow will route each flow to the appropriate switch/switch-port, using only the knowledge of the hierarchy of switches (no routers). The solution is suppose to save enterprises money and simplify networking.
Q. He is speculating that this may dramatically change enterprise networking. For many reasons, I am skeptical. I would like to hear your thoughts.
OpenFlow is a research project from Stanford University led by professor Nick McKeown. In the original OpenFlow research paper, the goal of OpenFlow was to give researchers a way "to run experimental protocols in the networks they use every day." For years networking researchers have had an almost impossible task deploying and evaluating their ideas on real networks with real Ethernet switches and IP routers. The difficultly is that real switches and routers from companies like Cisco, HP, and others, are all closed, proprietary boxes that implement standard "protocols", like Ethernet spanning tree, and OSPF. There are business reasons why Cisco and HP won't let you run software on their switches and routers; there is no technical reason. OpenFlow was invented to solve a people problem: if Cisco is not willing to let you run code on their switch, maybe they can at least provide a very narrow interface to let you remotely configure their switch, and that narrow interface is called OpenFlow.
To my knowledge more than a dozen companies are currently implementing OpenFlow support for their switches. Some like HP are only providing the OpenFlow software for research purposes. Others like NEC are actually offering commercial support.
For academic researchers that want to evaluate new routing protocols in real networks, OpenFlow is a huge win. For switch vendors, it is less clear if OpenFlow support will help, hurt, or have no effect in the long run. After all, the academic research market is very small.
The reason why OpenFlow is most often discussed in the context of enterprise networks is that OpenFlow grew out of a previous research project called Ethane that used OpenFlow's mechanism of remotely programming switches in an enterprise network in order to centralize a security policy. Ethane, and by extension OpenFlow, has led directly to two startup companies: Nicira, founded by Martin Casado, and Big Switch Networks, founded by Guido Appenzeller. It would be easier to implement an Ethane-like system if all of the switches in the network supported OpenFlow.
Closely related to enterprise networks are data center networks, the networks that interconnect thousands to tens of thousands of servers in companies such as Google, Facebook, Microsoft, Amazon.com, and Yahoo!. One problem with Ethernet is that it does not scale to this many servers on the same Layer 2 network. We attempted to solve this problem in a research project called PortLand. We used OpenFlow to facilitate programming the switches from a central controller, which we called a Fabric Manager. We released the PortLand source code as open source.
However, we also found a limitation to OpenFlow's functionality. In another data center networking research project called Helios, we were not able to use OpenFlow because it did not provide a mechanism for bonding multiple switch ports into a Link Aggregation Group (LAG). Presumably one could extend the OpenFlow specification indefinitely until it all possible switch features become exposed.
There are other networks as well such as the Internet access networks, Internet backbones, home networks, wireless networks, cellular networks, etc. Researchers are trying to see where OpenFlow fits into all of these markets. What it really comes down to is the question, "what problem does OpenFlow solve?" Ethane makes a case for enterprise networks but I have not yet seen a compelling case for any other type of network. OpenFlow might be the next big thing, or it might end up being a case of "don't solve a people problem with a technical solution."
In order to assess the future of flow-based networking and OpenFlow, here’s the way to think about it.
It starts with the silicon trends: Moore’s Law (2X transistors per 18-24 months), and a correlated but slower improvement in the I/O bandwidth available on a single chip (roughly 2X every 30-36 months). You can now buy full-featured 10GbE single chip switches with 64 ports, and chips which have a mix of 40GbE and 10GbE ports with comparable total I/O bandwidth.
There are a variety of ways physically connect these in a mesh (ignoring the loop-free constraints of spanning tree and the way Ethernet learns MAC addresses). In the high performance computing (HPC) world, a lot of work has been done building clusters with InfiniBand and other protocols using meshes of small switches to network the compute servers. This is now being applied to Ethernet meshes. The geometry of a CLOS or fat-tree topology enables a two stage mesh with a large number of ports. The math is thus: Where n is the # of ports per chip, the number of devices you can connect in a two-stage mesh is (n*2)/2, and the number you can connect in a three-stage mesh is (n*3)/4. While with standard spanning tree and learning, the spanning tree protocol will disable the multi-path links to the second stage, most of the Ethernet switch vendors have some sort of multi-chassis Link Aggregation protocol which gets around the multi-pathing limitation. There is also standards work in this area. Although it might not be obvious, the vast majority of Link Aggregation schemes allocate traffic so all the frames of any given flow take the same path. This is done in order to minimize out-of-order frames so they don’t get dropped by some higher level protocol. They could have chosen to call this “flow based multiplexing” but instead they call it “link aggregation”.
Although the devil is in the details, there are a variety of data center operators and vendors that have concluded they don’t need to have large multi-slot chassis switches in the aggregation/core layer for server connect, instead using meshes of inexpensive 1U or 2U switches.
People have also concluded that eventually you need some kind of management station to set up the configuration of all the switches. Again, drawing from the experience with HPC and InfiniBand, they use what is called an InfiniBand Controller. In the telecom world, most telecom networks have evolved to separate the management and part of the control plane from the boxes that carry the data traffic.
Summarizing the points above, meshes of Ethernet switches with an external management plane with multipath traffic where flows are kept in order is evolutionary, not revolutionary, and is likely to become mainstream. At least one major company, Juniper, has made a big public statement about their endorsement of this approach. I'd call all of these "flow-based routing".
Juniper and other vendors’ proprietary approaches notwithstanding, this is an area that cries out for standards. The Open Networking Foundation (ONF), was founded to promote standards in this area, starting with OpenFlow. Within a couple of months, the sixty+ members of ONF will be celebrating their first year anniversary. Each member has, I am led to believe, paid tens of thousands of dollars to join. While the OpenFlow protocol has a ways to go before it is widely adopted, it has real momentum.
#Nathan: OpenFlow 1.1 actually adds some primitives that enable the use of multiple links via the Multipath Proposal.
An excellent view of OpenFlow by Simon Crosby
http://community.citrix.com/display/ocb/2011/03/21/The+Rise+of+the+Software+Defined+Network
More context on SDN which discusses IETF's SDN initiative and ONF's Openflow. Working in conjuction is a powerful combination http://bit.ly/A8xYso
Nathan, Excellent historical account and overview of openflow. Thanks!
You've hit on the points that I've been wrapping my head around as to why Openflow might not be widely adopted. Since it was designed to be open to allow researcher the ability to run experimental protocols and not necessarily be "compatible with" the big players Cisco/HP/etc. it puts itself into niche (although potentially big) market, more on this later. And as you've stated it's recieved some adoption in the "cloud data centers (CDC)" e.g. google, facebook, etc because they need to exploit experimental protocols to gain a competitive advantage or optimize for their application.
As you've stated some switch vendors have added openflow capability to capitalize on the niche need in academia and potentially sell into the CDC; google, facebook. This is potentially a big market (or bubble if you're pessimistic).
The problem that I see is that the majority of the market (80% or more) is enterprise IT data centers. The requirements here is for stable, compatible networking. Open and less expensive would be nice, but not at the cost of the former.
One could think of a day where corporate IT is partially or completely cloud-sourced where QoS is maintained by the cloud provider. In this case, experimental protocols could be leveraged to provide a competitive advantaged for speed or QoS. In which case; openflow could play a more dominant roll. I personally think this scenario is many years off.
So, the conclusion I come to is that other than in research and perhaps CDCs (google, facebook), the market is pretty small. I suppose that if researchers use openflow to come up with a better protocol for say link aggregation, or congestion management, then eventually Cisco and HP will provide those in their standard offering because their customers will demand it. So openflow could be a market influencer (via the research community), but it would not be a market disruptor.
Do you agree with my conclusions? Thanks for your input.
I want to experiment with network programming in Haskell. The problem I have is that the documentation for the network package is pretty scarce, especially the one for Network.Socket which I want to use.
Do you know of some other references or clearly written projects where I can see how to use it? Are there any good alternatives to network?
Network.Socket is just bindings to the Berkeley socket API. You should read Beej's Guide to network programming.
EDIT: If you're on *nix then see the man pages for socket, bind, listen, accept, connect, recv, send and family. No matter your OS, there are also some higher level packages on Hackage (ex: network-fancy, network-server) you should look at if all you want to do is communicate (and not get involved in the gritty details).
Chapter 27 of ``Real World Haskell'' introduces networking in Haskell.
So I want to learn all about networks. Well below the socket, down to raw sockets and stuff. And I want to understand hubs, routers, access points, etc. For example, I'd like to be able to write my own software to do this kind of stuff.* Is there a great source for this kind of information?
I know that I'm asking a LOT here, and that to fully explain it all requires from high level down to low level. I guess I'm looking for a source similar in scope and depth to Applied Cryptography, but about networks.
Thanks to anyone who can help to point me (and others like me?) in the right direction.
* Yes, I realize using any of my hand-crafted network stack code would be a huge security issue, and am only looking to do it to learn :)
Similar Question: here. However I'm looking for more than just 'what's below TCP/UDP sockets?'.
Edited for Clarification: The depth I'm talking about is above the driver level. So assuming that the bits can make it to and from the other end of the wire, what next?
I learned IP networking from TCP/IP Illustrated. Highly recommended.
This may not help you learn it, but a packet sniffer like Wireshark will give you some insight into what the data looks like at a pretty low-level protocol (TCP/IP).
As you have obviously recognised, the universe does not start and end with the IP Protocol. Take a look at the OSI 7 Layer Model where IP is a Layer 3 (Network) protocol. Common IP Routers will operate at this level, but there is more complexity you probably should understand in the Data Link and Physical layers before you start coding your own network stacks.
Start with the fundamentals of data communications in all its myriad forms and work your way up the stack until you get to where you need to stop. Data Communications, Computer Networking and Open Systems is a good foundation text, and then look for more detail on each area you need to focus on. Previous answers include good links for IP and TCP/IP, and as mentioned Wireshark will let you look down through some of the layers
CISCO CCNA materials contain a great network fundamentals, but does not affect programming aspect. I'm not sure that there is an official free link, but you can try to find them.
You should equip yourself with a c compiler and the necessary libs and headers for your OS and play around. You may want to read for example:
http://snap.nlc.dcccd.edu/learn/fuller3/chap13/chap13.html
I had some more links in my delicious account, but they all went down the digital drain ;-)
Have you any embedded programming experience ? If so I recommend you buy one of these development boards. They are cheap and allow you work on every part of the networking stack plus all the software tools required are free.
Note that getting going on it isn't easy and I ended up reading the CS8900 IC datasheet to learn how to make it communicate with the ARM7 based processor. But if you enjoy that sort of thing (as I do) then they are great fun.
Hmmm ... have you looked into Computer Networks by Tanenbaum ?
The TCP/IP Guide
I have found the networking chapter in "understanding the linux kernel" and "understanding linux network internals" from oreilly to be very helpful.
The TCP/IP stack is a very good start but there is a lot more and a good understanding of how ethernet works and how ethernet != IP != the-interweb will go a long way.
books on network security often do a decent if not goos job explaining how networks work in a concise context.
what really did the trick for me was taking a job implementing NAT :)
This course worked for me: COS 461 at Princeton. Note that it assumes system-level programming experience with C.
Pretty much all the readings and lectures are available online under "Syllabus". And you can try the assignments too (unfortunately, you won't have access to the Virtual Network System).
Check this.. it is a good collection of information:
http://www.tcpipguide.com/free/t_toc.htm
What do you think of Quagga compared to XORP as a dynamic software routing engine? What are the technical merits of each engine comparatively? Additionally, what do most people think of them from a programming view. Who has manipulated networks using these enginers? I was wondering from an OSPF, routing, BGP protocol user's perpspective.
The following does not answer your question completely, but the Vyatta open source routers and the OpenSolaris customer gateway software for Amazon VPC both use quagga to implement BGP support.
From the wikipedia entry for XORP,
"The software suite was selected
commercially as the routing platform
for the Vyatta line of products in its
early releases, but later has been
replaced with quagga.
What do you think of Quagga compared to XORP as a dynamic software routing engine?
It is one of many options, but not particularly of very much use to you based upon your questions/information that you posted here. Have you tried looking into some of the alternatives such as (nothing comes to mind)?
What are the technical merits of each engine comparatively?
Small, fast, oddly placed, optimized, super-heroic and more filler for a resume.
Additionally, what do most people think of them from a programming view.
I can't speak for most people, but I for myself do not give it much credit or merit, or well... you know what I mean.
Who has manipulated networks using these enginers?
I could not find specific references, but I do remember reading that both Disney and the 'famous' YUV corporation of South Africa both played with this notion before. I believe Disney abandoned it with the fall of Michael Eisner.
I was wondering from an OSPF, routing, BGP protocol user's perpspective.
I am a BGP protocol user's prospective. Hopefully we hear from OSPF and routing user's perspectives shortly.
Good question.