I have a question regarding the design of these protocols. Why do we use boundary to separate parts of the multipart message instead of introducing a content-length for each part? Using length the parsing would easier. Do I miss some principal reason for using boundaries and not length parameter? Thanks!
Using length the parsing would [be] easier
This is where you are wrong. The authors of multipart MIME had cases in mind where you could not determine beforehand the length of a message part. Think of content encodings that alter message lengths such as base64, UUencode and others. There's also compression, encryption and whatnot. Also: Content-Length is an entity header. This means if you reach it, you've already begun to parse a message part. It comes with literally no advantage over a boundary marker.
If you study older protocols, you will often encounter some marker (usually \0) to indicate the end of a message. Sending the byte count of a message is another solution, but you won't find it a lot in places where message content has to be converted on-the-fly or is to be streamed in some fashion.
Bottom line: The multipart boundary allows some interesting applications with message contents of unpredictable size. HTTP server pushing is a notable example.
Because in good old days the MIME Standard was defined this way.
One of the reasons was probably that content-length has a problem with text/plain data, where the newline might be either CR (old mac), LF (unix) or CR LF (windows, dos). The other might be that it is easier for a human to read, which is IMHO a bad argument but happens a lot when preferring textual representations like HTTP, XML or SOAP instead of the more effective binary ways like ASN.1 or SUN RPC.
You might also view it as a successful attempt of the industry to sell more powerful servers by introducing useless overhead into the protocols :)
I wanted to know whether
A file encryption algorithm can be designed in such a way that it can perform the activities of a file compression as well (any live example?).
Also can I integrate it to a Mobile SMS Service, I mean for text messages?
Also I wanted to know about binary files...if a plain text file is encoded in binary does its size reduces? Also is it better to encode a plain text file to a binary one rather than encoding it to any other format (in case if anyone wants it to encode for any purpose)
In fact all decent encryption programs (take PGP for example) compress data before encryption. They use something mainstream like ZIP for compression. The reason is once data is encrypted it looks like random noise and becomes incompressible, so it can only be compressed before encryption. You likely can't do that for SMS - you have to obey the specifications for the SMS so you'd better check those specifications.
Compression removes redundant information. Redundant information makes breaking an encryption easier. So yes, encryption and compression are compatible. I don't know if there is an algorithm designed to do both things though.
Yes, a binary file will usually be smaller than a plain text file. For instance, the number 34 written out in text takes 2 bytes (at least) whereas in those same 2 bytes you could write a number as large as 65000.
What makes an encoding "better" than another is the purpose to which it is put. If you are optimizing for size, binary is probably better. If you are optimizing for readability or graceful failure, text might be better.
First question: No reason why it shouldn't be possible. Since you can still encrypt compressed data and vise versa, you can test the benefits by trying. I don't know if it can be used for text messaging, but you'd have to explain what you're trying to do if anyone wants to give a reasonable and fitting answer.
Second question: plain text is still binary, it's just readable and encoded in a certain character set. Main difference is that plain text usually conforms to a certain encoding, say UTF-8 or ISO-8859-1.
You can still compress plain text, which makes it "binary" in the traditional sense that you only see weird bytes. :-)
A lot of the examples online show the hash as a hex representation, and they are typically custom implementations. Is there anything wrong with, or less secure about, using the Apache Commons Base64 encoding instead? When reading about encoding, it is usually within the realm of how to represent binary as text in XML, but does not necessarily discuss security concerns... just how the compression works.
On a related issue, why encode it all, since databases have binary types that could probably hold the encryption as binary? So if I'm storing a password, why not just store it in its native type?
An encoding affects only the representation of the data, not its security. So, if you send an unencrypted password and use some form of encoding, you've not made it any more secure; likewise, if you take some highly encrypted text and then represent it in some encoding scheme, that won't make it any less secure, either. Typically, the reason to use this form of encoding is to send binary data using a protocol (such as SMTP), where the protocol is only capable of supporting 7-bit ASCII text. Another use is in URLs, where the set of characters that a URL can support is limited, but you might want to put arbitrarily complicated binary data in that URL (such as a validation signature of some sort).
Not at all. It's just an encoding that represents the same bits in ASCII. It is mostly useful when you have to store or transmit binary data over communications paths designed to handle only text. Modern examples ore email and web interfaces. You also can't send the binary form to a terminal to view it, since it would result in garbage or strange terminal behavior.
If you can safely store the bits in a binary blob in a database there is no reason to encode in base64. But if you don't it would be harder to view it. You would have to convert it back to a text form first.
Well we typically don't do too well reading binary, and hex is a better substitute for that. I wish you had linked to the articles you were referencing, so others could have a direct line on what it is you're forming an opinion from.
I don't understand why they would use Base64 over hex, but I'm assuming it's because hex is 16 digits and Base64 is a few more, thus generating a more compact version of the actual hash ;) ~ We humans tend to do better absorbing a small amount of data at a time.
No, because Base64 is a 1:1 encoding (that is, for every input there is exactly one base64 encoded output, and vice-versa), base64 encoding a SHA1 hash is just as “secure” as a hex-encoded (or binary-encoded, for that matter) hash.
The encoding would only make a hash insecure if the encoding made it possible for multiple hashes to encode to the same string, or multiple strings to decode to the same hash.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
It seems like there used to be way more binary protocols because of the very slow internet speeds of the time (dialup). I've been seeing everything being replaced by HTTP and SOAP/REST/XML.
Why is this?
Are binary protocols really dead or are they just less popular? Why would they be dead or less popular?
You Just Can't Beat the Binary
Binary protocols will always be more space efficient than text protocols. Even as internet speeds drastically increase, so does the amount and complexity of information we wish to convey.
The text protocols you reference are outstanding in terms of standardization, flexibility and ease of use. However, there will always be applications where the efficiency of binary transport will outweigh those factors.
A great deal of information is binary in nature and will probably never be replaced by a text protocol. Video streaming comes to mind as a clear example.
Even if you compress a text-based protocol (e.g. with GZip), a general purpose compression algorithm will never be as efficient as a binary protocol designed around the specific data stream.
But Sometimes You Don't Have To
The reason you are seeing more text-based protocols is because transmission speeds and data storage capacity have indeed grown fast compared to the data size for a wide range of applications. We humans find it much easier to work with text protocols, so we designed our ubiquitous XML protocol around a text representation. Certainly we could have created XML as a binary protocol, if we really had to save every byte, and built common tools to visualize and work with the data.
Then Again, Sometimes You Really Do
Many developers are used to thinking in terms of multi-GB, multi-core computers. Even your typical phone these days puts my first IBM PC-XT to shame. Still, there are platforms such as embedded devices, that have rather strict limitations on processing power and memory. When dealing with such devices, binary may be a necessity.
A parallel with programming languages is probably very relevant.
While hi-level languages are the preferred tools for most programming jobs, and have been made possible (in part) by the increases in CPU speed and storage capactity, they haven't removed the need for assembly language.
In a similar fashion, non-binary protocols introduce more abstraction, more extensibility and are therefore the vehicle of choice particularly for application-level communication. They too have benefited from increases in bandwidth and storage capacity. Yet at lower level it is still impractical to be so wasteful.
Furthermore unlike with programming languages where there are strong incentives to "take the performance hit" in exchange for added simplicity, speed of development etc., the ability to structure communication in layers makes the complexity and "binary-ness" of lower layers rather transparent to the application level. For example so long as the SOAP messages one receives are ok, the application doesn't need to know that these were effectively compressed to transit over the wire.
Facebook, Last.fm, and Evernote use the Thrift binary protocol.
I rarely see this talked about but binary protocols, block protocols especially can greatly simplify the complexity of server architectures.
Many text protocols are implemented in such a way that the parser has no basis upon which to infer how much more data is necessary before a logical unit has been received (XML, and JSON can all provide minimum necessary bytes to finish, but can't provide meaningful estimates). This means that the parser may have to periodically cede to the socket receiving code to retrieve more data. This is fine if your sockets are in blocking mode, not so easy if they're not. It generally means that all parser state has to be kept on the heap, not the stack.
If you have a binary protocol where very early in the receive process you know exactly how many bytes you need to complete the packet, then your receiving operations don't need to be interleaved with your parsing operations. As a consequence, the parser state can be held on the stack, and the parser can execute once per message and run straight through without pausing to receive more bytes.
There will always be a need for binary protocols in some applications, such as very-low-bandwidth communications. But there are huge advantages to text-based protocols. For example, I can use Firebug to easily see exactly what is being sent and received from each HTTP call made by my application. Good luck doing that with a binary protocol :)
Another advantage of text protocols is that even though they are less space efficient than binary, text data compresses very well, so the data may be automatically compressed to get the best of both worlds. See HTTP Compression, for example.
Binary protocols are not dead. It is much more efficient to send binary data in many cases.
WCF supports binary encoding using TCP.
http://msdn.microsoft.com/en-us/library/ms730879.aspx
So far the answers all focus on space and time efficiency. No one has mentioned what I feel is the number one reason for so many text-based protocols: sharing of information. It's the whole point of the Internet and it's far easier to do with text-based, human-readable protocols that are also easily processed by machines. You rid yourself of language dependent, application-specific, platform-biased programming with text data interchange.
Link in whatever XML/JSON/*-parsing library you want to use, find out the structure of the information, and snip out the pieces of data you're interested in.
Some binary protocols I've seen on the wild for Internet Applications
Google Protocol Buffers which are used for internal communications but also on, for example Google Chrome Bookmark Syncing
Flash AMF which is used for communication with Flash and Flex applications. Both Flash and Flex have the capability of communicating via REST or SOAP, however the AMF format is much more efficient for Flex as some benchmarks prove
I'm really glad you have raised this question, as non-binary protocols have multiplied in usage many folds since the introduction of XML. Ten years ago, you would see virtually everybody touting their "compliance" with XML based communications. However, this approach, one of several approaches to binary protocols, has many deficiencies.
One of the values, for example, was readability. But readability is important for debugging, when humans should read the transaction. They are very inefficient when compared with binary transfers. This is due to the fact that XML itself is a binary stream, that has to be translated using another layer into textual fragments ("tokens"), and then back into binary with the contained data.
Another value people found was extensibility. But extensibility can be easily maintained if a protocol version number for the binary stream is used at the beginning of the transaction. Instead of sending XML tags, one could send binary indicators. If the version number is an unknown one, then the receiving end can download the "dictionary" of this unknown version. This dictionary could, for example, be an XML file. But downloading the dictionary is a one time operation, instead of every single transaction!
So efficiency could be kept together with extensibility, and very easily! There are a good number of "compiled XML" protocols out there which do just that.
Last, but not least, I have even heard people say that XML is a good way to overcome little-endian and big-endian types of binary systems. For example, Sun computers vs Intel computers. But this is incorrect: if both sides can accept XML (ASCII) in the right way, surely both sides can accept binary in the right way, as XML and ASCII are also transmitted binarically.......
Hope you find this interesting reading!
Binary protocols will continue to live wherever efficency is required. Mostly, they will live in the lower-levels, where hardware-implementation is more common than software implementations. Speed isn't the only factor - the simplicity of implementation is also important. Making a chip process binary data messages is much easier than parsing text messages.
Surely this depends entirely on the application? There have been two general types of example so far, xml/html related answers and video/audio. One is designed to be 'shared' as noted by Jonathon and the other efficient in its transfer of data (and without Matrix vision, 'reading' a movie would never be useful like reading a HTML document).
Ease of debugging is not a reason to choose a text protocol over a 'binary' one - the requirements of the data transfer should dictate that. I work in the Aerospace industry, where the majority of communications are high-speed, predictable data flows like altitude and radio frequencies, thus they are assigned bits on a stream and no human-readable wrapper is required. It is also highly efficient to transfer and, other than interference detection, requires no meta data or protocol processing.
So certainly I would say that they are not dead.
I would agree that people's choices are probably affected by the fact that they have to debug them, but will also heavily depend on the reliability, bandwidth, data type, and processing time required (and power available!).
They are not dead because they are the underlying layers of every communication system. Every major communication system's data link and network layers are based on some kind of "binary protocol".
Take the internet for example, you are now probably using Ethernet in your LAN, PPPoE to communicate with your ISP, IP to surf the web and maybe FTP to download a file. All of which are "binary protocols".
We are seeing this shift towards text-based protocols in the upper layers because they are much easier to develop and understand when compared to "binary protocols", and because most applications don't have strict bandwidth requirements.
depends on the application...
I think in real time environment (firewire, usb, field busses...) will always be a need for binary protocols
Are binary protocols dead?
Two answers:
Let's hope so.
No.
At least a binary protocol is better than XML, which provides all the readability of a binary protocol combined with all the efficiency of less efficiency than a well-designed ASCII protocol.
Eric J's answer pretty much says it, but here's some more food for thought and facts. Note that the stuff below is not about media protocols (videos, images). Some items may be clear to you, but I keep hearing myths every day so here you go ...
There is no difference in expressiveness between a binary protocol and a text protocol. You can transmit the same information with the same reliability.
For every optimum binary protocol, you can design an optimum text protocol that takes just around 15% more space, and that protocol you can type on your keyboard.
In practice (practical protocols is see every day), the difference is often even less significant due to the static nature of many binary protocols.
For example, take a number that can become very large (e.g., in 32 bit range) but is often very small. In binary, people model this usually as four bytes. In text, it's often done as printed number followed by colon. In this case, numbers below ten become two bytes and numbers below 100 three bytes. (You can of course claim that the binary encoding is bad and that you can use some size bits to make it more space efficient, but that's another thing that you have to document, implement on both sides, and be able to troubleshoot when it comes over your wire.)
For example, messages in binary protocols are often framed by length fields and/or terminators, while in text protocols, you just use a CRC.
In practice, the difference is often less significant due to required redundancy.
You want some level of redundancy, no matter if it's binary or text. Binary protocols often leave no room for error. You have to 100% correctly document every bit that you send, and since most of us are humans, that happens rarely and you can't read it well enough to make a safe conclusion what is correct.
So in summary: Binary protocols are theoretically more space and compute efficient, but the difference is in practice often less than you think and the deal is often not worth it. I am working in the Internet of Things area and have to deal nearly on daily base with custom, badly designed binary protocols which are really hard to troubleshoot, annoying to implement and not more space efficient. If you don't need to absolutely tweak the last milliampere out of your battery and calculate with microcontroller cycles (or transmit media), think twice.