I know that a protocol is a set of rules that governs communication between two computers on a network, but how are thoses rules implemented for the computer? Is a protocol basically a piece of code or, in other words, software?
Protocols are generally built upon each other. At the risk of sounding pedantic, here's an example of a protocol and where/how it's implemented:
Application Protocol - the way a particular application talks to another instance of itself or a corresponding server; this is implemented in the application code or a shared library
TCP (or UDP, or another layer) - the way that information is sent at the binary level and split up into usable chunks, then reassembled at the destination; this is usually implemented as part of the operating system, but it is still software code
IP - the way that information (having already been split or truncated by something like TCP or UDP) makes its way from one place to another by routing over one or more "hops"; this is always software code, but is sometimes implemented in the OS and sometimes implemented in the network device (your LAN card, for example)
base-T (ethernet), token ring, etc - Here we are physically getting into how the hardware talks to one another; ie, which wire corresponds to a particular type of signal; this is always implemented in hardware
electricity /photons - the laws that govern (or at least define) how electrons (or photons) flow over a conductive material or over the air; this is usually implemented in hardware ;)
In a sense, these are all "protocols" (a set of rules or expected behaviors that allow communication to take place), and they're built on one another.
Bear in mind that (aside from electricity) this is not an exhaustive list of the sort of protocols that exist at any of these layers!
Edit Thanks to dmckee for pointing out that electricity isn't the only physical process used in networking ;)
Networking protocols are not pieces of code or software, they are only a set of rules. When software uses a specific networking protocol, then the software is known as an implementation. There can be many different software implementations of the same protocol (i.e. Windows and UNIX have different TCP/IP implementations). It is possible to understand networking protocols without any knowledge of programming.
EDIT: How are they implemented? Here's a paper on taking an abstract specification of a protocol and implementing it into C. You'll see that less-strict protocols leave out certain details that programmers have to guess on, which makes some implementations incompatible with others.
A network protocol is basically like a spoken language. It is implemented by code that sends and receives specially prepared messages over the network/internet, much like the vocal chords you need to speak (the network and hardware) and a brain to actually understand what someone said (the protocol stack/software).
Sometimes protocols are implemented directly on the hardware [for speed reasons] (like the Ethernet protocol for LANs) - but it is always software/code required to do something useful with a protocol.
This might be interesting for you:
The OSI Model
Protocol (Computing)
Software implements the rules defined in the protocol, some protocols are formal defined and some informal.
a protocol is a set of rules governing the communication between two entities.
in the computer/programming context, a protocol is a set of rules governing the communication between two programs.
in the computer network context, a protocol is a set of rules governing the communication between two programs, well, over network.
in computers, in the end everything is embodied in code...
Protocols are basically set of rules. The way to implement them is to first of all make a state machine diagram as it completely tells that what is going to be the current state and how the state is going to change on the basis of input and what output actions are going to be performed.
Your answer is a very short one:
BY READING THE RFC.
The main networking problem is to share data between computers. All the networking protocols try to solve is a little part of that major problem. Some of them (the protocols) are implemented as software, some others as hardware. In short, protocols like algorithms, can be implemented it in many programming languages.
Back to the TCP, it is implemented by the operating system.
Related
i have a question about network layer, that is:
as we all know, in layer architecture, the N+2 layer should only depends on the N+1 layer, while knows nothing about N layer. for example, in a typical application, the web layer should only depends on the business logic layer, but not the data access layer
when it comes to computer network, things seem to be different. In application layer, program has to know not only transmition layer(TCP port), but also network layer(IP address)
this confuse me, what do you think about this?
thanks for your help.
generally you are right. Unfortunately borders between layers in networks are kinda blurry, not just because we have a standard which is not used (OSI) and de facto standard which does not enforce the idea you mentioned, but also because the protocols are often not strictly bound to one layer but can do stuff on more then one of them. Good amount of protocols is developed before the OSI model and before they were standardized and then it was already too late to make some radical changes. So there are protocols that are considered to be between two layers (or on both layers) like MPLS, ARP etc. And protocols that are based on another protocol which is on the same layer, like OSPF that runs on top of IP even if they are considered to be on L3.
What you mentioned is another example. The reason for that is that addressing is not done on the most-upper layer (application layer) but on network layer (for host/network adapter) and transport layer (for process/application). So you need to know the IP address and port number (and actually a protocol) to be able to address the remote application. That's where the network sockets come in as an gateway (or API) between application and the network. So, even if you are technically correct about defying the principle of layered model, you are not really doing anything on L3 or L4 (but you can;) ). You don't need to fragment packets, handle retransmissions or worry about error corrections etc., you are just passing down the required addressing information when creating a socket.
TCP/IP is more oriented towards the feasibility of implementation, where OSI is more concerned about the standard then the implementation of that standard. This has it's bad and good sides. The ability to freely implement the protocol can be an advantage if you use that power well and since you are not strictly bound to some specification you can do some things more efficiently... or fail epically. The drawbacks of mixing 'responsibilities' are obvious and great example are protocols like H.323 which embed the IP addresses inside user's payload so if you want to do NAT for example you need to inspect the payload, change IP addresses, recalculate checksums, and stuff like that instead of just handling the translation on network layer.
Why are stuff still like this? Probably because there is no easy way to change any of that because of sheer number of devices and protocols, applications, etc that needs to be updated and this takes a lot of time. Just look at the speed of adopting IPv6 which has been around for more then 15 years.
I want to build a decentralized, reddit-like system using P2P. Basically, I want to retain the basic capabilities of reddit, but make it decentralized, to make it more robust and immune to censorship. This will also allow people to develop different clients to match the way they want to browse it.
Could you recommend good p2p libraries to base my work on? They should be open-source, cross-platform, robust and easy to use. I don't care much about the language, I can adapt.
Disclaimer: warning, self-promotion here !!!
Have you considered JXTA's latest release? It is probably sufficient for what you want to do. Else, we are working on a new P2P framework called Chaupal, but it is not operational yet.
EDIT
There is also what I call the quick-and-dirty UDP solution (which is not so dirty after all, I should call it minimal).
Just implement one server with a public address and start listening for UPD.
Peers located behind NATs contact the server which can read how their private IP address has been translated into a public IP address from the received datagrams.
You send that information back to the peer who can forward it to other peers. The server can also help exchanging this information between peers.
Then peers can communicate directly (one-to-one) by sending datagrams to these translated addresses.
Simple, easy to implement, but does not cover for lost datagrams, replays, out-of-order etc... (i.e., the typical stuff that TCP solves for you at the IP stack level).
I haven't had a chance to use it, but Telehash seems to have been made for this kind of application. Peer2Peer apps have a particular challenge dealing with the restrictions of firewalls... since Telehash is based on UDP, it's well suited for hole-punching through firewalls.
EDIT for static_rtti's comment:
If code velocity is a requirement libjingle has a lot of effort going into it, but is primarily geared towards XMPP. You can port off parts of the ICE code and at least get hole-punching. See the libjingle architecture overview for details about their implementation.
Check out CouchDB. It's a decentralized web app platform that uses an HTTP API. People have used it to create "CouchApps" which are decentralized CouchDB-based applications that can spread in a viral nature to other CouchDB servers. All you need to know to write CouchApps is Javascript and learn the CouchDB API. You can read this free online book to learn more: http://guide.couchdb.org
The secret sauce to CouchDB is a Master-to-Master replication protocol that lets information spread like a virus. When I attended the first CouchConf, they demonstrated how efficient this is by throwing a "Couch Party" (which is where you have a room full of people replicating to the person next to them simulating an ad hoc network).
Also, all the code that makes a CouchApp work is public by default in special entities known as Design Documents.
P.S. I've been thinking of doing a similar project, but I don't have a lot of time to devote to it at the moment. GOD SPEED MY BOY!
Are there any libraries which put a reliability layer on top of UDP broadcast?
I need to broadcast large amounts of data to a large number of machines as quickly as possible, and generally it seems like such a problem must have already been solved many times over, but I wasn't able to find anything except for the Spread toolkit, which has a somewhat viral license (you have to mention it in all materials advertising the end product, which I'm not sure our customer will be willing to do).
I was already going to write such a thing myself (because it would be extremely fun to do!) but decided to ask first.
I looked also at UDT (http://udt.sourceforge.net) but it does not seem to provide a broadcast operation.
PS I'm looking at something as lightweight as a library - no infrastructure changes.
How about UDP multicast? Have a look at the PGM protocol for which there are several commercial and open source implementations.
Disclaimer: I'm the author of OpenPGM, an open source implementation of said protocol.
Though some research has been done on reliable UDP multicasting, I haven't yet used anything like that. You should take into consideration that this might not be as trivial as it first sounds.
If you don't have a list of nodes in the target network you have no idea when and to whom to resend, even if active nodes receiving your messages can acknowledge them. Sending to a large number of nodes, expecting acks from all of them might also cause congestion problems in the network.
I'd suggest to rethink the network architecture of your application, e.g. using some kind of centralized solution, where you submit updates to a server, and it sends this message to all connected clients. Or, if the original sender node's address is known a priori, then just let clients connect to it, and let the sender push updates via these connections.
Have a look around the IETF site for RFCs on Reliable Multicast. There is an entire working group on this. Several protocols have been developed for different purposes. Also have a look around Oracle/Sun for the Java Reliable Multicast Service project (JRMS). It was a research project of Sun, never supported, but it did contain Java bindings for the TRAM and LRMS protocols.
How do I go about executing a fuzzing strategy to stress a network stack, specifically at the third and fourth layers (network and transport)? I've looked at frameworks to generate fuzzers, like SPIKE, but it seems to me that they are mostly focused on the application layer and above? Is there any well known techniques out there to fuzz well-known protocols in these layers, say, TCP?
Thanks.
Look at Scapy. It allows you to fuzz at the network and transport layers. The fuzz function will fuzz anything you didn't explicitly specify in the IP or TCP layers (you can apply it separately to each). This gives you a range of abilities from just randomly generating ip addresses and port pairs to making and sending nonsense packets.
You may also want to look at Fragroute. This will twist TCP/IP into using all sorts of evasions techniques, but could potentially unveil otherwise hidden bugs/vulnerabilities in your network stack.
Furthermore, if your organization doesn't object, you could set up a Tor exit node and capture traffic from it. I've found it useful for testing correct TCP connection state tracking. Though your end of the connections is well-known and unchanging, there's a huge variety of servers as well as fun network congestion issues. It's basically an endless source of traffic. Be sure to check with your higher ups as your org may object to being a potential source of malicious traffic (even though there is a strong precedent of non-liability). I've gotten around that issue by running it/capturing at home, then bringing in the pcaps.
If you want to fuzz the IP, UDP, or TCP route your packets from your high level services via loopback to a process that reads them, fuzzes them, and forwards them. You need a driver that lets you talk to raw sockets and you need to read/learn what the applicable RFCs say for those protocols.
There is an easy way to do this. Just as Justdelegard recommends, Scapy is probably the best thing to use, in general.
Take a look at Releasing ICMPv4/IP fuzzer prototype by Laurent Gaffié. His Python code, which incidentally he has reposted in more readable fashion at pastebin.com, imports from scapy and uses some methods he defines to do a couple of types of fuzzing. IP and ICMP packets are handled in his sample code. So, this sounds exactly like what you are seeking.
Right now, there seems to be a lot of companies using Tcl/Expect to do custom automated testing of networks. SIP, H.323, layer 2 & 3 protocols, etc.
So if Scapy does not meet your needs, you might be able to make or find something written in Tcl using Expect to do the job. Or, you may wish to do some things in Python, using Scapy - and other things in Tcl, using Expect.
Tcl has long been used for network test and management applications. There was a book on how to use Tcl to do SNMP-based network management way back in the 1990's.
Syntax of Tcl is decidedly odd but the libraries are very powerful. It comes with a framework-like ability to define behavior of custom network behavior atop sockets, similar to what you can do with the standard libraries for the Python programming language.
Unlike Python and other scripting languages, there is an extremely powerful tool for Tcl programs named Expect (see expect man page).
Expect has a handy capability. It can auto generate a Tcl test script. The generated script makes calls to Expect functions. When doing this recording, it functions as a passive man-in-the-middle, recording both sides of the conversation. Kind of the way that you record Macros while you do some editing in MS Word or in Emacs.
Then afterward, you can edit the automatically-generated Expect script to fine tune it, make it behave differently, or creation multiple variations of it. It is very handy for creating regression tests. You should be able to use this to kickstart writing higher layer protocol tests, should you need some. Beats starting from scratch.
I think you can use Tcl/Expect to test standard TCP applications (FTP, HTTP, SMTP, etc.) that use string based commands. It works well for testing character based applications like TELNET that read input from stdin and generate output to stdout too.
I'm supposed to be setting up a 'geolocation based', ipv6, wireless mesh network to run on google android.
I found what seems to be a good app to support the meshing:
http://www.open-mesh.net/wiki/batman-adv
"Batman-advanced is a new approach to
wireless networking which does no
longer operate on the IP basis. Unlike
B.A.T.M.A.N, which exchanges
information using UDP packets and sets
routing tables, batman-advanced
operates on ISO/OSI Layer 2 only and
uses and routes (or better: bridges)
Ethernet Frames. It emulates a virtual
network switch of all nodes
participating. Therefore all nodes
appear to be link local, thus all
higher operating protocols won't be
affected by any changes within the
network. You can run almost any
protocol above B.A.T.M.A.N. Advanced,
prominent examples are: IPv4, IPv6,
DHCP, IPX."
But other members in my team has said it's a no-go because it operates on OSI, rather than TCP/IP. This was the first I'd heard of OSI, and I'm wondering how much of a problem this is? What are the implications for mesh network apps that can be developed on top of it? Considering the android is relatively new, we don't need to worry too much about compatibility with existing apps, so does it matter?
I haven't spent a lot of time working with networks, so please put in noobmans terms.
"You can run almost any protocol above B.A.T.M.A.N. Advanced, prominent examples are: IPv4, IPv6, DHCP, IPX."
"But other members in my team has said it's a no-go because it operates on OSI, rather than TCP/IP. "
The other members in your team are confused by the buzzword-fest in BATMAN.
The "IP" of TCP/IP is IPv4 (or IPv6). So BATMAN supports TCP/IP directly and completely.
There's no conflict of any kind. Just confusion.
They're probably referring to the OSI model, which is a commonly-used way of distinguishing between network layers. I'm not sure it's a useful way of looking at things, but it's taught in every networking course on the planet.
OSI level 2 is the data link layer, which operates immediately above the actual physical level. Basically, it's in charge of flow control, error detection, and possibly error correction. The data link layer is strictly "single hop". It's only concerned about about point-to-point data transfers, not about multi-hop transfers or routing.
If they're actually referring the OSI networking protocal itself, run screaming as fast as you can. OSI was notoriously hard to implement, and I've never heard of an actual working installation. See the Wikipedia article for the gory details.
The OSI model and the OSI protocols are different.
The OSI model is a way of breaking things down: physical, link, network, transport, session, presentation, application. OSI protocols are protocol implementations that map directly to those layers in the model.
The model is a way of looking at things. It mostly makes sense, but it breaks down at the higher levels. For example: what does a presentation layer really do?
During the '90s, OSI was (in some circles) thought to be the future, but was actually the downfall of some companies, and wasted the resources of many others. For example, DECnet Phase V was Digital's insanely complex implementation of an OSI stack that met government OSI requirements, but was run over by the TCP/IP steamroller.
The test is: What are the bytes on the wire? In this case it is UDP over IP, not the OSI equivalent, which was CLNP.
Having said all that, if it is a layer two protocol, it will probably have scalability problems because it is a layer two protocol. Fine for a small number of nodes, but if you're trying to get scale, you need a better solution.
"ISO/OSI Layer 2" does not mean the OSI protocols. It refers to the "Seven Layer" model of network stacks. It means the Data Link layer.
The layers are: Physical, Data Link, Network, Transport, Session, Presentation, Application.
OSI is a model not a protocol like IP and TCP. What your team seem to be saying is that the mesh won't be using IP. I suspect they are wrong as the text you have quoted states the BATMAN protocol is capable of supporting IP & IPv6 and if that is the case you'd need a very strong reason to use anything else.