what's the best way to program a peer-to-peer chat with encryption. is SSL useful for p2p as the clients won't have a certificate signed by verisign,...? is it possible to program such a chat without a master-server that tells the client the ip's of their chat partner?
i'd really like to program such a chat program, because in times of all those privacy issues i want a program i really can trust. i've got some experience with network programming, but only client <-> server.
i don't want any code, just the theory part of a peer-to-peer chat (like skype).
thanks.
First, you can enforce using self signed certificates at both ends OR you can even use 'pre shared key' instead of certificate for authentication. Pre shared key is supposed to be known by all parties involved in advance.
Secondly, SSL is an encryption mechanism so it has no bearing on design of communication model i.e. client/server vs p2p.
Lastly, if both of your p2p nodes have public or live ip's and each of them knows about the other's, then you don't need any kind of central server. The real problem here is that in practice, most of machines are behind firewalls using NAT's and private ip's. So you need some kind of mechanism to traverse through NAT, determine the public ip and port in order to reach the destination machine successfully.
Take a look at the following SO question and it's answer i wrote. It mentions some techniques used by common p2p applications such as skype (knowledge of skype's protocol is limited since it is proprietary), gtalk as well as other XMPP based chat applications.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 7 years ago.
Improve this question
The goal of this question is that I am just trying to better understand the nature of P2P and networking and security / encryption. I am a front-end web developer and my knowledge of the networking stack is not great if we go lower than HTTP requests.
That being said, I am trying to understand how torrent traffic is "sniffed" by ISPs and the content identified. I feel like this question will expose my ignorance, but is it not possible to have some sort of HTTPS-like P2P protocol that would not be so readable?
I grasp that a given packet has to identify its destination to the network along the way, but couldn't torrent packets be configured to show ONLY their destination, so that nobody could identify its purpose along the way, until it arrived at its destination? Why is it apparently an unrectifiable situation that ISPs can just look at P2P traffic and know everything about it, yet SSH is extremely safe?
Every answer here seems to have a different interpretation of the question, or rather, a different assumed purpose of the encryption. Since you compare it to https, it seems like a reasonable assumption is that you're looking for authentication and confidentiality. I'll enumerate a few attempts in decreasing level of "security". This is a bittorrent centric answer, because you tagged the question with bittorrent.
SSL
Starting with the strongest system, it is possible to run bittorrent over SSL (it's not supported by many clients, but in a fully controlled deployment it can be done). This gives you:
Authentication of every peer participating
The ability to pick which peers are let into the swarm by signing their certificate with the swarm root.
SSL encryption of all peer connections + tracker connections
The tracker can authenticate every peer connecting to it, but even if the peer list (or one peer) is leaked or guessed, there's still the peer-to-peer authentication, blocking any unauthorized access.
Bittorrent over SSL has been implemented and deployed.
encrypted torrents
At BitTorrent (in the uTorrent client) we added support for symmetric encryption of torrents at the disk layer:
Everything in the bittorrent engine would operate on encrypted blocks. The data integrity checks (sha-1 hashes of pieces) would be done on encrypted blocks and the .torrent file would have hashes of the encrypted data. An encrypted torrent like this is backwards compatible with clients that don't support the feature, but they won't be able to access the data (just help out the swarm and seed it).
To download the torrent in an unencrypted form, you would add the &key= argument to the magnet link, and uTorrent decrypts and encrypts data at the disk boundary (leaving the data on disk in the clear). Anyone adding the magnet link without the key, would just get encrypted data.
There are some other details involved too, like encrypting some of the metadata in the .torrent file. Such as the list of files etc.
This does not let you pick which peers get to join. You can give access to the peers you want, but since it's a symmetric key, anyone with access can invite anyone else, or publish the key. It does not give you any stronger authentication than you had when you found the magnet link.
It gives you confidentiality among trusted peers and the ability to have untrusted peers help out with seeding.
bittorrent protocol encryption
The bittorrent protocol encryption is probably better described as obfuscation. Its primary intention is not to authenticate or control access to a swarm (it derives the encryption key from the info-hash, so if you can keep that a secret you do get that property). The main purpose is to avoid trivial passive snooping and shaping of traffic. My understanding is that it's less effective to avoid being identified as bittorrent traffic these days. It also provides weak protection against sophisticated and active attacks. For instance, if the DHT is enabled, or tracker connections are not encrypted, it's easy to learn about the info-hash, which is the key.
In the case of private torrents (where DHT and peer exchange are disabled) assuming the tracker runs HTTPS there aren't any obvious holes in it. However, my experience is that it's not uncommon for https trackers to have self signed certificates, and for clients to not authenticate trackers. Which means poisoning the DNS entry for the tracker may be enough to enter the swarm.
Torrent traffic can be encrypted, and there are VPNs/SOCKS proxies that can be used to redirect traffic, i.e., via another country through an encrypted tunnel before connecting to peers. That said, even if you use such services, there are a lot of ways of leaking traffic via side channels (e.g., DNS lookups, insecure trackers, compromised nodes), and most people aren't knowledgeable enough to follow all proper security/anonymity precautions. Furthermore, restricting yourself to communicating only with clients who have also forced encryption will limit the number of peers you can connect to.
The problem you're considering is the difference between point-to-point encryption where there are only two peers in a private context and an unbounded number of peers in a public context.
Decryption by any of the public peers can only be effected if there's a primer somewhere -- a decryption key that is available for all the public peers to use. In the case of protecting from the ISPs, they would also have access to that key unless there was some exclusionary protocol for only sharing the key amongst everyone else. It's not practical to do this.
In a point-to-point connection, a TLS key negotiation eventually creates a session encryption key that is shared by both peers. The key is pseudorandom and session-specific. Data shared on the internet this way would be unusable to clients that didn't participate in the key negotiation.
Bittorrent traffic (specifically the peer-peer protocol used to transfer the bulk of the data) can be encrypted. But it's the kind of encryption that does not provide strong confidentiality/authentication guarantees, similar (but not identical) to HTTP2's opportunistic encryption
Client-Tracker communication can be encrypted with HTTPS.
These two components give you a working, albeit restricted, bittorrent stack that's encrypted and whose contents are not visible to a passive observer.
ISPs may still be able to identify it as "bittorrent, probably" based on side-channel data (packet sizes/traffic patterns, domains contacted, ...) but they won't know exactly what is being transferred.
My knowledge about network programming is limited, so, all the comments are more than welcome. Essentially my question boils down to the following question:
Q1. Is there really such a thing as decentralized asynchronous cross-platform peer-to-peer communication?
Let me explain myself.
If we have two http servers running on computers with actual IP addresses, then clearly the answer is yes, assuming one writes a protocol for the interaction.
To go one step further, if one of them (or both) is (are) behind a router, then, with port forwarding the communication can still be established. However, here the problems start because if someone wants to run such a server on the background, say in a mobile phone, the app that is relying on this server really works when one is at home (we can not really expect to request port forwarding everywhere we go).
But even beyond that,
Q2. do mobile phones obtain an actual IP address from telecommunication companies when someone is not using a wi-fi?
If this is true, then clearly one can have cross-platform asynchronous peer-to-peer communication at the expense of not using wi-fi by running an http server on a smartphone. (I understand that this is not convenient, but it is certainly doable.)
Concluding, the two (perhaps there are more) relevant questions that I can think of are:
Q3. How does Skype really work?
Q4. How does Viber really work?
Based on the answer for Skype, it says: If one of the callee or both of them do not have a public IP, then they send voice traffic to another online Skype node over UDP or TCP.
So, it appears that there is no direct communication in Skype, because they have to use a man-in-the-middle for such a scenario.
Regarding Viber, I could not find a good-thorough answer to this particular question. Do people talk to each other through a Viber centralized server, or, do they establish a direct connection? Of course if they do establish a direct connection, then I really want to know how they manage such a thing since a mobile phone may or may not have a physical address. How is a Viber message routed to my cell phone from a friend of mine even when Viber is not running and I am behind a router?
I guess the answer to Viber is really push notifications, but as far as I can understand, all the variations of push notifications rely on open connections, and then the servers of the applications send the notifications to the clients through such connection(s). So, this approach gives us the feeling that it is asynchronous, but essentially it is not. We are cheating, in the sense that there is a constantly open connection to a server, and moreover, as far as I can understand, the application server has to push the notification through that server. Schematically:
A > Central App Server > Central Server w/ open connection to my cellphone > me
So, this seems to be once again a centralized approach.
Honestly, the only approach that I can think of that is both decentralized and asynchronous (on mobile phones as well) is to run an http server on every platform/device, but this comes at the expense of not using Wi-Fi and assuming that a telecommunication company really assigns a physical IP address to every mobile phone (which I do not know if it is true, do you?).
What about WASTE, darknets, F2Fs, etc? Do they offer advantages in the sense of a more direct asynchronous communication between some interested parties? Are there real-world applications (also including mobile phones) using such approaches for communication.
Really, this is not the actual problem that I would like to work on, but I would like to know what the state of the art is so that I can figure out how I can proceed from there. So, all comments are really more than welcome. If you have references for the state of the art I would like to know about them as well, but a brief description would also be nice.
I appreciate all your time and effort in advance.
You asked many questions, here is the beginning of the answers:
Q1: Yes. For example, take BitTorrent's very successful 10 million+ node network. Aside from the bootstrapping process, the protocol is entirely decentralized and asynchronous. See here for more info.
Q2: Yes! Go to www.whatismyip.com on your mobile telephone, and you will see your assigned IP. However, you are likely to be very filtered (e.g: incoming traffic on port 80 is likely to be blocked).
Q3: It has elements of P2P and clever tricks to get around NAT issues - see here for more info.
Q4: I don't know.
I'm a young professional who's into embedded design, IT networking, control/monitoring systems and much more. Currently, I'm developing a monitoring system using a device from Tibbo Techonology, their DS1102.
http://tibbo.com/products/controllers/ds110x/ds1102
It's a programmable device that covers serial and ethernet communications. For my project, its main tasks are serial data collection and database population. Serial communication is done through RS485 and database used is MySQL 5.5. My database is hosted on a public IP which also runs a webserver for the interface while my device is behind a NAT. It connects to the database directly using the public IP.
I'd like to ask for advices so that I can enhance and upgrade it. Right now these are the
questions I'd like to ask.
Which is better? Having the server on a public IP or using port forwarding?
I'm also using it as webserver for the interface of my monitoring system.
To communicate with the device (rebooting, changing IP etc), I wrote an application in
python using UDP (using port 65535 of device) and also set the device to communicate with the application for specific commands. My concern is I want to encrypt the communication between my python app and the device both ways. The only available function for both encrypting and decrypting on the DS1102 is RC4. What are your thoughts on using RC4 for this application? Also, I'm planning to do port forwarding on port 65535 so that I can use my python app from the outside. Can RC4 be reliable for this too? I really want to learn how to use encryptions properly.
I'm also planning to implement SMTP for alert messages. Tibbo has a sample code from which I based mine. Problem is, it's on AUTH PLAIN LOGIN. I think I want to turn it to STARTTLS later. Can you recommend some lessons on the algorithm of STARTTLS?
What are those details on MAIL FROM:<> and RCPT TO:<>? Because on using the command
DATA, the programmer can write anyway From: and To: which can make his identity someone else.
That's it for now. Suggestions are very welcome.
You can also share some good reading materials and links. I'm always hungry for learning. :)
Thanks for your time.
2.
Encryption substitutes the confidentiality of an arbitrary amount of data (the plaintext) with the confidentiality of a small amount data (the key). In other words, your communication is only as confidential as the key – if the shared secret key leaks out, the encryption is worthless. More on this.
Also note that plain RC4 provides no authenticity (message integrity). An adversary can modify messages as much as he wants. He can even send his own messages which will be considered perfectly valid by the cipher. Verifying the validity of the messages is is up to the code that parses the messages.
If your messages are simple (only a few bytes or so), an adversary could simply send random bytes until they decrypt such that they form a valid message, without knowing anything about the key. This happens on average after only 100 attempts for a 1-byte message for example.
You will obviously have to use some sort of a nonce to prevent trivial replay attacks.
RC4 is also rather quirky per se. I guess you are already aware of the numerous "drop-n" variants and so on.
In short, protocol design is perilous. Even experts often get it wrong (look at WEP for example). The most straightforward way to solve this would be to find hardware that can handle an existing protocol such as TLS.
Quick question: do most chat applications (ie. AIM, Skype, Oovoo) use peer to peer UDP exchange for talking to other users or an echoing TCP connection with a server? Or some combination in-between?
Traditionally, most applications used a TURN-like solution (i.e., communication via a server) to overcome NAT traversal issues. Since chat does not consume much bandwidth, servers could support thousands of communications.
But now that P2P has evolved and the NAT traversal issues are now well understood, some use direct UDP communication provided that the users' NAT allows this (i.e., STUN-like communication). They still need a central server to punch the hole though. Direct communication is also helpful when lots of data needs to be transmitted.
I believe it is fair to say that most modern frameworks use a combination of both.
when you need small fragments of data, such as text messaging, there's no need of using P2P. data can be transmitted from client1 to server, and from server back to the client2.
When you need to transfer data quickly between clients, in cases such as VoIP (voice over IP), or file transfer, you will use P2P.
A pretty standard IM protocol is XMPP. I know it's used by Google Talk, as well as a few other big names in chat.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
If I have a separate system with its own concept of users and presence, what is the most appropriate architecture for creating a bridge to an XMPP server network? As far as I can tell there are three primary ways:
Act as a server. This creates one touchpoint, but I fear it has implications for compatibility, and potentially creates complexity in my system for emulating a server.
Act as a clients. This seems to imply that I need one connection per user in my system, which just isn't going to scale well.
I've heard of an XMPP gateway protocol, but it's unclear if this is any better than the client solution. I also can't tell if this is standard or not.
Any suggestions or tradeoffs would be appreciated. For example, would any of these solutions require running code inside the target XMPP server (not likely something I can do).
The XMPP gateway protocol you've heard of is most likely to do with transports. A transport is a server that connects to both a XMPP server and a non-XMPP server. By running a transport, I can use my Jabber client to talk to someone using, say, MSN Messenger.
A transport typically connects once to the remote network for each JID that it sees as online. That is, it's your option 2 in reverse. This is because there is no special relationship between the transport and the non-XMPP network; the transport is simply acting as a bunch of regular clients. For this to work, XMPP clients must first register with the transport, giving login credentials for the remote network, and allowing the transport to view their presence.
The only reason this has a chance of scaling better is that there can be many transports for the same remote network. For example, my Jabber server could run a transport to MSN, another Jabber server could run another one, and so on, each one providing connections for a different subset of XMPP users. While this spreads out the load on the Jabber side, and load balancing on your system may spread out the load as well, it still requires many connections between the two systems.
In your case, because (I assume) the non-XMPP side of things is cooperating, putting a XMPP server interface on the non-XMPP server is likely your best bet. That server interface is best suited for managing the mapping between XMPP JIDs and how that JID will appear on its own network, rather than forcing XMPP users to register and so on.
In case you haven't seen these, you might find them useful:
http://www.jabber.org/jabber-for-geeks/technology-overview
http://www.xmpp.org/protocols/
http://www.xmpp.org/extensions/
Hope that helps.
I too am working on a similar system.
I am going with the gateway/component route. I have looked at several options and settled with this one.
The gateway is basically a component with the specific purpose of bridging Jabber/XMPP with another network. You will have to build most of the things you take for granted when using XMPP as a client. Stuff like roster control.
There is very little help online on the actual design and building of a component. Like the above answer I found that the xmpp protocols/extensions to be of help. The main ones being:
Basic Client 2008
Basic Server 2008
Intermediate Client 2008
Intermediate Server 2008
Reading through these will show you what XEPs you will be expected to be able to handle. Ignore the stuff that will be handled by the server that your component will be attched to.
It's a shame that Djabberd has such poor documentation as their system of "everything is a module" gave the possibility of backend of the server could interface directly to the other network. I made no headway on this.
There are basically two types of server to server (s2s) connections. The first is either called a gateway or a transport, but they're the same thing. This is probably the kind you're looking for. I couldn't find specific documentation for the non-XMPP side, but how XMPP thinks about doing translations to legacy servers is at http://xmpp.org/extensions/xep-0100.html. The second kind really isn't explained in any additional XEPs -- it's regular XMPP s2s connections. Look for "Server-to-Server Communication" in RFC 3920 or RFC 3920bis for the latest draft update.
Since you have your own users and presence on your server, and it's not XMPP, the concepts aren't going to map completely to the XMPP model. This is where the work of the transport comes in. You have to do the translation from your model to the XMPP model. While this is some work, you do get to make all the decisions.
Which brings us right to one of the key design choices -- you need to really decide which things you are going to map to XMPP from your service and what you aren't. These feature and use case descriptions will drive the overall structure. For example, is this like a transport to talk to AOL or MSN chat services? Then you'll need a way to map their equivalent of rosters, presence, and keep session information along with logins and passwords from your local users to the remote server. This is because your transport will need to pretend to be those users and will need to login for them.
Or, maybe you're just an s2s bridge to someone else's XMPP based chess game, so you don't need a login on the remote server, and can just act similarly to an email server and pass the information back and forth. (With normal s2s connections the only session that would be stored would be SASL authentication used with the remote server, but at the user level s2s just maintains the connection, and not the login session.)
Other factors are scalability and modularity on your end. You nailed some of the scalability concerns. Take a look at putting in multiple transports to balance the load. For modularity, see where you want to make decisions about what to do with each packet or action. For example, how do you handle and keep track of subscription data? You can put it on your transport, but then that makes using multiple transports harder. Or if you make that decision closer to your core server you can have simpler transports and use some common code if you need to talk to services other than XMPP. The trade off is a more complex core server with more vulnerability potential.
What architecture you should use depends on the non-XMPP system.
Do you operate the non-XMPP system? If yes, you should find a way to add an XMPP-S2S interface to that system, in other words, make it act as an XMPP server. AOL is using this approach for AIM. Unfortunately, they have restricted their gateway to GoogleTalk.
You don't operate the non-XMPP system but it has a federation interface that you can use - i. e. your gateway can talk to the other system as a server and has a namespace of its own. In this case, you can build a gateway that acts as a federated server on both sides. For I don't know of any example of a gateway that uses this approach but you could use it if you want to build a public XMPP-to-SIP bridge.
If the non-XMPP system doesn't give you a federation interface, then you have no other option but acting as a bunch of clients. In the XMPP world, this is called a "transport". The differences between a transport and a normal server are basically:
the JIDs of the transport are mapped from another system (e.g. john.doe\40example.net#msngateway.example.org - really ugly!)
XMPP users who want to use the transport need to create an account on the non-XMPP system and give the login credentials of that account to the transport service. The XMPP protocol even has a protocol extension that allows XMPP users to do transport registrations in-band.
One other approach is to work with your XMPP server vendor. Most have internal APIs that make injecting presence possible from third party applications. For example, Jabber XCP provides an API for this that's really easy to use.
(Disclosure: I work for Jabber, Inc, the company behind Jabber XCP)