I want to encrypt the data in a LAN or internet while transmitting. Is it possible?
If yes, how? Can I use IPsec for this, or is there any other protocol to do this?
I want to encrypt the data in a LAN or internet while transmitting. Is it possible?
There are several option (TLS/SSL or IPSec may be the most common). Maybe you should specify your use case to get any better answer.
Basic indutrial protocol today used for encrypting network traffic is SSL/TLS. Imho this is your best and simplest option. Almost every platform or programming language already contain libraries for implementing TLS.
If yes, how? Can I use IPsec for this, or is there any other protocol to do this?
I'd really got for TLS/SSL as operating on higher network layer than IPSec.
I'm not satisfied with some algorithms of TCP, and i know it's not possible to implement TCP in UDP. But i want to make a compatible layer with others server which relies on traditional TCP.
So i ask, can i manipulate IP directly on Linux or other *nix OS?
I know udt and other similar projects. I just need to keep the compatibility so that i don't need to do much works for so large amount of servers.
If you want to remain compatible with other endpoints wich implement standard TCP then I assume you want to use the same protocol on the wire and make incremental improvements to it.
Your kernel's existing TCP implementation is in the kernel. If you want to improve it, I would say you had better make changes there rather than reinvent it. If you want to reinvent it and implement a whole TCP stack in userspace, then, sure, you can do it, but it's going to be A LOT OF WORK.
Quick question: do most chat applications (ie. AIM, Skype, Oovoo) use peer to peer UDP exchange for talking to other users or an echoing TCP connection with a server? Or some combination in-between?
Traditionally, most applications used a TURN-like solution (i.e., communication via a server) to overcome NAT traversal issues. Since chat does not consume much bandwidth, servers could support thousands of communications.
But now that P2P has evolved and the NAT traversal issues are now well understood, some use direct UDP communication provided that the users' NAT allows this (i.e., STUN-like communication). They still need a central server to punch the hole though. Direct communication is also helpful when lots of data needs to be transmitted.
I believe it is fair to say that most modern frameworks use a combination of both.
when you need small fragments of data, such as text messaging, there's no need of using P2P. data can be transmitted from client1 to server, and from server back to the client2.
When you need to transfer data quickly between clients, in cases such as VoIP (voice over IP), or file transfer, you will use P2P.
A pretty standard IM protocol is XMPP. I know it's used by Google Talk, as well as a few other big names in chat.
I usually pass data between my web servers (in different locations) using HTTP requests (sometimes using SSL if it's sensitive). I was wondering if there were any lighter protocols that I might be able to swap HTTP(S) for that would also support public/private keys like SSH or something.
I used PHP sockets to build a SMTP client before so I wouldn't mind doing that if required.
There are lots and lots and lots of protocols. Lots. Start here for a list.
http://en.wikipedia.org/wiki/Internet_Protocol_Suite
SFTP is fun for passing data around. It works well. You'll find that it's not much better than HTTP, however, because HTTP is pretty simple.
http://en.wikipedia.org/wiki/SSH_file_transfer_protocol
SMTP would work. http://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol
SNMP can be made to work. http://en.wikipedia.org/wiki/Simple_Network_Management_Protocol You have to really push the envelope.
All of these, however, involve TCP/IP sockets, which involve a fair amount of overhead because of the negotiation for a connection and the acknowledgement of packets.
If you want real fun with very low overhead, use UDP.
http://en.wikipedia.org/wiki/User_Datagram_Protocol
You might want to use Reliable UDP if you're worried about messages getting dropped.
http://en.wikipedia.org/wiki/Reliable_User_Datagram_Protocol
I'd like to mention XMPP in addition to protocols already listed in other answers.
It's lightweight, and it is used in some "realtime" communication systems (for example, in GTalk).
WebSocket is a good option if you are interested in keeping a connection open to pass multiple messages back and forth. It's useful for issuing updates from the server to clients in real time, for example.
Why don't you simply use FTPS:
http://en.wikipedia.org/wiki/FTPS
or SFTP
http://en.wikipedia.org/wiki/SSH_file_transfer_protocol
I recently checked out the book "UNIX Network Programming, Vol. 1" by Richards Stevens and I found that there is a third transport layer standard besides TCP and UDP: SCTP.
Summary: SCTP is a transport-level protocol that is message-driven like UDP, but reliable like TCP. Here is a short introduction from IBM DeveloperWorks.
Honestly, I have never heard of SCTP before. I can't remember reading about it in any networking books or hearing about it in classes I had taken. Reading other stackoverflow questions that mentions SCTP suggests that I'm not alone with this lack of knowledge.
Why is SCTP so unknown? Why is it not much used?
Indeed, SCTP is used mostly in the telecom area. Traditionally, telecom switches use SS7 (Signaling System No. 7) to interconnect different entities in the telecom network. For example - the telecom provider's subscriber data base(HLR), with a switch (MSC), the subscriber is connected too (MSC).
The telecom area is moving to higher speeds and more reachable environment. One of these changes is to replace SS7 protocol by some more elegant, fast and flexible IP-based protocol.
The telecom area is very conservative. The SS7 network has been used here for decades. It is very a reliable and closed network. This means a regular user has no access to it.
The IP network, in contrast, is open and not reliable, and telecoms will not convert to it if it won't handle at least the load that SS7 handles. This is why SCTP was developed. It tries:
to mimic all advantages of the SS7 network accumulated over the decades.
to create a connection-oriented protocol better than TCP in speed, security, and redundancy
The latest releases of Linux already have SCTP support.
We have been deploying SCTP in several applications now, and encountered significant problem with SCTP support in various home routers. They simply don't handle SCTP correctly. I believe this is primarily a performance issue (the SCTP protocol specification require checksums for the whole packets to be recalculated and not just for headers).
Like many other promising protocols SCTP is sadly dead in the water until D-link and Netgear fixes their broken NAT boxes.
SCTP is not very much known and not used/deployed a lot because:
Widespread: Not widely integrated in TCP/IP stacks (in 2013: still missing natively in latest Mac OSX and Windows. 2020 update: still not in Windows nor Mac OS X)
Libraries: Few high level bindings in easy to use languages (Disclaimer: i'm maintainer of pysctp, SCTP easy stack support for Python)
NAT: Doesn't cross NAT very well/at all (less than 1% internet home & enterprise routers do NAT on SCTP).
Popularity: No general public app use it
Programming paradigm: it changed a bit: it's still a socket, but you can connect many hosts to many hosts (multihoming), datagram is ordered and reliable, erc...
Complexity: SCTP stack is complex to implement (due to above)
Competition: Multipath TCP is coming and should address multihoming needs / capabilities so people refrain from implementing SCTP if possible, waiting for MTCP
Niche: Needs SCTP fills are very peculiar (ordered reliable datagrams, multistream) and not needed by much applications
Security: SCTP evades security controls (some firewalls, most IDSes, all DLPs, does not appear on netstat except CentOS/Redhat/Fedora...)
Audit-ability: Something like 3 companies in the world routinely do audits of SCTP security (Disclaimer: I work in one of them)
Learning curve: Not much toolchain to play with SCTP (check the excellent withsctp that combines nicely with netcat or use socat, 2020 edit: nmap supports it for a few years now )
Under the hood: Used mostly in telecom and everytime you send SMS, start surfing the net on your mobile or make phone calls, you're often triggering messages that flow over SCTP (SIGTRAN/SS7 with GSM/UMTS, Diameter with LTE/IMS/RCS, S1AP/X2AP with LTE), so you actually use it a lot but you never know about it ;-) 2020 edit: it's being removed from the core 5G network (no more Diameter, HTTP/2 instead) and will be only used in the 5G radio access network between antennas and core.
SCTP requires more design within the application to get the best use of it. There are more options than TCP, the Sockets-like API came later, and it is young. However I think most people that take the time to understand it (and who know the shortcomings of TCP) appreciate it -- it is a well designed protocol that builds on our ~30 years of knowledge of TCP and UDP.
One of the aspects that requires some thought is that of streams. Streams provide (usually, I think you can turn it off) an order guarantee within them (much like a TCP connection) but there can be multiple streams per SCTP connection. If your application's data can be sent over multiple streams then you avoid head-of-line blocking where the receiver starves due to one mislaid packet. Effectively different conversations can be had over the same connection without impacting each other.
Another useful addition is that of multi-homing support -- one connection can be across multiple interfaces on both ends and it copes with failures. You can emulate this in TCP, but at the application layer.
Proper link heartbeating, which is the first thing any application using TCP for non-transient connections implements, is there for free.
My personal summary of SCTP is that it doesn't do anything you couldn't do another way (in TCP or UDP) with substantial application support. The thing it provides is the ability to not have to implement that code (badly) yourself.
FYI, SCTP is mandated as supported for Diameter (cf RADIUS next gen). see RFC 3588
Diameter clients MUST support either TCP or SCTP, while agents and
servers MUST support both. Future versions of this specification MAY
mandate that clients support SCTP.
p1. SCTP mapped directly over IPv4 requires support in NAT gateways, which has never been widely deployed anywhere, and without it the typical NAT gateway will only permit one private host per public address to be using SCTP at a time.
p2. SCTP mapped over UDP/IPv4 allows more private hosts per public address, but UDP mappings in IPv4/NAT gateways are notoriously tricky to establish and keep maintained, due to the fact that UDP is a connectionless transport without any explicit state for a NAT to track.
p3. SCTP mapped directly over IPv6 requires... well... IPv6. Have you tried to deploy IPv6? If so, have you tried to buy an IPv6 firewall? Does it support SCTP? How about a load balancer? A SSL accelerator?
p4. Finally, a lot of the Internet is pretty much constrained to what can fit through TCP port 80 and port 443, so SCTP of any flavor tends to lose there. Hence, you see efforts like the MPTCP working group in IETF.
Many of us will be using SCTP soon, since it's used by WebRTC datachannels to create a TCP-like reliable layer on top of UDP -- SCTP over DTLS over UDP: https://datatracker.ietf.org/doc/html/draft-ietf-rtcweb-data-channel-13#section-6
Reading the SCTP Wikipedia page I'd say that the main reason is that SCTP is a very young protocol (proposed in 2000) that is currently unsupported by the mainstream OSs (Windows, OS X, Linux).
If "very young" seems inappropriate to you, think about IPV6: "in December 2008, despite marking its 10th anniversary as a Standards Track protocol, IPv6 was only in its infancy in terms of general worldwide deployment."
SCTP is used extensively in the 4G LTE network where Diameter is used for AAA.
It might not be well known, but it's not unused. Quite recently there was a draft published at the IETF about Using SCTP as a Transport Layer Protocol for HTTP.
In reference to all of the comments about commercial routers being broken or lacking SCTP support, the issue is that SCTP with NAT is still in draft form with the IETF. So there is no RFC specification for them to implement it.
https://datatracker.ietf.org/doc/html/draft-ietf-behave-sctpnat-09
Sctp is born too late, and for many situation TCP is enough.
Also, as I know most of its usage is on telecommunication area.