I want to experiment with network programming in Haskell. The problem I have is that the documentation for the network package is pretty scarce, especially the one for Network.Socket which I want to use.
Do you know of some other references or clearly written projects where I can see how to use it? Are there any good alternatives to network?
Network.Socket is just bindings to the Berkeley socket API. You should read Beej's Guide to network programming.
EDIT: If you're on *nix then see the man pages for socket, bind, listen, accept, connect, recv, send and family. No matter your OS, there are also some higher level packages on Hackage (ex: network-fancy, network-server) you should look at if all you want to do is communicate (and not get involved in the gritty details).
Chapter 27 of ``Real World Haskell'' introduces networking in Haskell.
Related
I'm trying to learn network protocol stack(ie. Transport, IP, datalink layer library code implementation) along with linux. I'm confused where to start.
First question is whether these codes come as in-built features of linux kernel/above library layers.
If so why I can see 3rd party protocol stack in some applications (by blunk micro system - developer of protocol stack)
If Linux doesn't have it as core feature, is linux give only placeholders for network part(like just Macros to enable the 3rd party stack ). But an article says it has Net4 networking codebase.
If linux has in-built network features what are the linux modules I need to go through or where to start? Not only in the network perspective, if i'm guided to explore in linux in all aspects (process, memory, drivers) in the "code level", it would be helpful please.
Note: I'm greedy to write my own OS and protocol stack hence trying to understand an existing system.
Thanks in advance!
First question is whether these codes come as in-built features of linux kernel/above library layers.
Linux kernel has network stack up to including layer 4, i.e., TCP and UDP (well, kernel + a set of utilities needed to configure it). I think DNS is in kernel too, but I am not so sure. TLS used to be implemended as a library (OpenSSL and GnuTLS are I think the most common ones), but there seems to be kernel part too now (link.
Note, that some of the TCP functionality is offloaded to the network card (hardware). At high speeds (1Gb+) you won't get full performance without these features.
I am not familiar with all VoIP related protocols, but I think they are libraries, not kernel.
If so why I can see 3rd party protocol stack in some applications (by blunk micro system - developer of protocol stack)
I believe the reason is performance. If you implement a custom stack with a subset of features, it might work better for your applications. Also there are advanced features and protocols that might not be available in the kernel itself.
If Linux doesn't have it as core feature, is linux give only placeholders for network part(like just Macros to enable the 3rd party stack ). But an article says it has Net4 networking codebase.
there is a very large codebase
If linux has in-built network features what are the linux modules I need to go through or where to start? Not only in the network perspective, if i'm guided to explore in linux in all aspects (process, memory, drivers) in the "code level", it would be helpful please.
hmmm, this is a very good question, and I don't think there is an easy answer. In my experience reading the code is the only way to figure this out. However some people tried to fish LWN.net for information.
you could probably start somewhere here: include/net/
First question is whether these codes come as in-built features of linux kernel/above library layers.
If linux has in-built network features what are the linux modules I need to go through or where to start?
You can think of a protocol stack as of a library. Linux kernel has one which runs inside the kernel address space and uses kernel APIs unavailable in user-space: https://github.com/torvalds/linux/tree/master/net/ipv4
There are multiple in-depth books about Linux kernel networking. Reading one is required for good understanding.
If so why I can see 3rd party protocol stack in some applications (by blunk micro system - developer of protocol stack)
Zero-copy, low-latency and streaming (processing an Ethernet packet in CPU-L1-cache-line-sized chunks while it hasn't been read off the wire in full) networking have been problematic with Linux kernel network stack. For these reasons makers of networking hardware offered their own user-space network stacks, aka kernel bypass.
Linux kernel network stack is getting better these days with MSG_ZEROCOPY and io_uring.
I want to build a decentralized, reddit-like system using P2P. Basically, I want to retain the basic capabilities of reddit, but make it decentralized, to make it more robust and immune to censorship. This will also allow people to develop different clients to match the way they want to browse it.
Could you recommend good p2p libraries to base my work on? They should be open-source, cross-platform, robust and easy to use. I don't care much about the language, I can adapt.
Disclaimer: warning, self-promotion here !!!
Have you considered JXTA's latest release? It is probably sufficient for what you want to do. Else, we are working on a new P2P framework called Chaupal, but it is not operational yet.
EDIT
There is also what I call the quick-and-dirty UDP solution (which is not so dirty after all, I should call it minimal).
Just implement one server with a public address and start listening for UPD.
Peers located behind NATs contact the server which can read how their private IP address has been translated into a public IP address from the received datagrams.
You send that information back to the peer who can forward it to other peers. The server can also help exchanging this information between peers.
Then peers can communicate directly (one-to-one) by sending datagrams to these translated addresses.
Simple, easy to implement, but does not cover for lost datagrams, replays, out-of-order etc... (i.e., the typical stuff that TCP solves for you at the IP stack level).
I haven't had a chance to use it, but Telehash seems to have been made for this kind of application. Peer2Peer apps have a particular challenge dealing with the restrictions of firewalls... since Telehash is based on UDP, it's well suited for hole-punching through firewalls.
EDIT for static_rtti's comment:
If code velocity is a requirement libjingle has a lot of effort going into it, but is primarily geared towards XMPP. You can port off parts of the ICE code and at least get hole-punching. See the libjingle architecture overview for details about their implementation.
Check out CouchDB. It's a decentralized web app platform that uses an HTTP API. People have used it to create "CouchApps" which are decentralized CouchDB-based applications that can spread in a viral nature to other CouchDB servers. All you need to know to write CouchApps is Javascript and learn the CouchDB API. You can read this free online book to learn more: http://guide.couchdb.org
The secret sauce to CouchDB is a Master-to-Master replication protocol that lets information spread like a virus. When I attended the first CouchConf, they demonstrated how efficient this is by throwing a "Couch Party" (which is where you have a room full of people replicating to the person next to them simulating an ad hoc network).
Also, all the code that makes a CouchApp work is public by default in special entities known as Design Documents.
P.S. I've been thinking of doing a similar project, but I don't have a lot of time to devote to it at the moment. GOD SPEED MY BOY!
I want to learn how to write a software using a peer to peer networking architecture but i don't know where to start, knowing that I use as a programming languages : c/c++ , lisp, a little of python. any pointer to documentation or tutorials is appreciated.
Why not just start with a Bittorrent API, such as https://github.com/JosephSalisbury/python-bittorrent, rather than writing your own.
After you have gained experience with a well-used P2P network, then you may start to see what you could do better, then start to write your own.
here are some guidelines which i found with google
How do I go about executing a fuzzing strategy to stress a network stack, specifically at the third and fourth layers (network and transport)? I've looked at frameworks to generate fuzzers, like SPIKE, but it seems to me that they are mostly focused on the application layer and above? Is there any well known techniques out there to fuzz well-known protocols in these layers, say, TCP?
Thanks.
Look at Scapy. It allows you to fuzz at the network and transport layers. The fuzz function will fuzz anything you didn't explicitly specify in the IP or TCP layers (you can apply it separately to each). This gives you a range of abilities from just randomly generating ip addresses and port pairs to making and sending nonsense packets.
You may also want to look at Fragroute. This will twist TCP/IP into using all sorts of evasions techniques, but could potentially unveil otherwise hidden bugs/vulnerabilities in your network stack.
Furthermore, if your organization doesn't object, you could set up a Tor exit node and capture traffic from it. I've found it useful for testing correct TCP connection state tracking. Though your end of the connections is well-known and unchanging, there's a huge variety of servers as well as fun network congestion issues. It's basically an endless source of traffic. Be sure to check with your higher ups as your org may object to being a potential source of malicious traffic (even though there is a strong precedent of non-liability). I've gotten around that issue by running it/capturing at home, then bringing in the pcaps.
If you want to fuzz the IP, UDP, or TCP route your packets from your high level services via loopback to a process that reads them, fuzzes them, and forwards them. You need a driver that lets you talk to raw sockets and you need to read/learn what the applicable RFCs say for those protocols.
There is an easy way to do this. Just as Justdelegard recommends, Scapy is probably the best thing to use, in general.
Take a look at Releasing ICMPv4/IP fuzzer prototype by Laurent Gaffié. His Python code, which incidentally he has reposted in more readable fashion at pastebin.com, imports from scapy and uses some methods he defines to do a couple of types of fuzzing. IP and ICMP packets are handled in his sample code. So, this sounds exactly like what you are seeking.
Right now, there seems to be a lot of companies using Tcl/Expect to do custom automated testing of networks. SIP, H.323, layer 2 & 3 protocols, etc.
So if Scapy does not meet your needs, you might be able to make or find something written in Tcl using Expect to do the job. Or, you may wish to do some things in Python, using Scapy - and other things in Tcl, using Expect.
Tcl has long been used for network test and management applications. There was a book on how to use Tcl to do SNMP-based network management way back in the 1990's.
Syntax of Tcl is decidedly odd but the libraries are very powerful. It comes with a framework-like ability to define behavior of custom network behavior atop sockets, similar to what you can do with the standard libraries for the Python programming language.
Unlike Python and other scripting languages, there is an extremely powerful tool for Tcl programs named Expect (see expect man page).
Expect has a handy capability. It can auto generate a Tcl test script. The generated script makes calls to Expect functions. When doing this recording, it functions as a passive man-in-the-middle, recording both sides of the conversation. Kind of the way that you record Macros while you do some editing in MS Word or in Emacs.
Then afterward, you can edit the automatically-generated Expect script to fine tune it, make it behave differently, or creation multiple variations of it. It is very handy for creating regression tests. You should be able to use this to kickstart writing higher layer protocol tests, should you need some. Beats starting from scratch.
I think you can use Tcl/Expect to test standard TCP applications (FTP, HTTP, SMTP, etc.) that use string based commands. It works well for testing character based applications like TELNET that read input from stdin and generate output to stdout too.
So I want to learn all about networks. Well below the socket, down to raw sockets and stuff. And I want to understand hubs, routers, access points, etc. For example, I'd like to be able to write my own software to do this kind of stuff.* Is there a great source for this kind of information?
I know that I'm asking a LOT here, and that to fully explain it all requires from high level down to low level. I guess I'm looking for a source similar in scope and depth to Applied Cryptography, but about networks.
Thanks to anyone who can help to point me (and others like me?) in the right direction.
* Yes, I realize using any of my hand-crafted network stack code would be a huge security issue, and am only looking to do it to learn :)
Similar Question: here. However I'm looking for more than just 'what's below TCP/UDP sockets?'.
Edited for Clarification: The depth I'm talking about is above the driver level. So assuming that the bits can make it to and from the other end of the wire, what next?
I learned IP networking from TCP/IP Illustrated. Highly recommended.
This may not help you learn it, but a packet sniffer like Wireshark will give you some insight into what the data looks like at a pretty low-level protocol (TCP/IP).
As you have obviously recognised, the universe does not start and end with the IP Protocol. Take a look at the OSI 7 Layer Model where IP is a Layer 3 (Network) protocol. Common IP Routers will operate at this level, but there is more complexity you probably should understand in the Data Link and Physical layers before you start coding your own network stacks.
Start with the fundamentals of data communications in all its myriad forms and work your way up the stack until you get to where you need to stop. Data Communications, Computer Networking and Open Systems is a good foundation text, and then look for more detail on each area you need to focus on. Previous answers include good links for IP and TCP/IP, and as mentioned Wireshark will let you look down through some of the layers
CISCO CCNA materials contain a great network fundamentals, but does not affect programming aspect. I'm not sure that there is an official free link, but you can try to find them.
You should equip yourself with a c compiler and the necessary libs and headers for your OS and play around. You may want to read for example:
http://snap.nlc.dcccd.edu/learn/fuller3/chap13/chap13.html
I had some more links in my delicious account, but they all went down the digital drain ;-)
Have you any embedded programming experience ? If so I recommend you buy one of these development boards. They are cheap and allow you work on every part of the networking stack plus all the software tools required are free.
Note that getting going on it isn't easy and I ended up reading the CS8900 IC datasheet to learn how to make it communicate with the ARM7 based processor. But if you enjoy that sort of thing (as I do) then they are great fun.
Hmmm ... have you looked into Computer Networks by Tanenbaum ?
The TCP/IP Guide
I have found the networking chapter in "understanding the linux kernel" and "understanding linux network internals" from oreilly to be very helpful.
The TCP/IP stack is a very good start but there is a lot more and a good understanding of how ethernet works and how ethernet != IP != the-interweb will go a long way.
books on network security often do a decent if not goos job explaining how networks work in a concise context.
what really did the trick for me was taking a job implementing NAT :)
This course worked for me: COS 461 at Princeton. Note that it assumes system-level programming experience with C.
Pretty much all the readings and lectures are available online under "Syllabus". And you can try the assignments too (unfortunately, you won't have access to the Virtual Network System).
Check this.. it is a good collection of information:
http://www.tcpipguide.com/free/t_toc.htm