Whilst there are many possible implementations and suggestions how to interpret HTTP-quality-values / -Q-factors floating on the internet, I was unable to find 'correct' interpretations for several cases and why they should be interpreted this way. This would be mandatory for creating a "bulletproof" parsing mechanism.
For example, the MDN-Documentation lists following examples:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
would be read as follows:
value
priority
text/html, application/xhtml+xml
1.0
application/xml
0.9
*/*
0.8
It seems clear to me, that according to RFC 7231 (HTTP/1.1), the first argument would be qualified as q=1.0 due to defaults ('(no value is the same as q=1)'), at least it seems to be true for the second argument, as well, without clearly stating that this is due to the defaulting.
Furthermore it is totally unclear for me, how the following constructed statement should be parsed:
text/html,application/xhtml+xml,application/xml;q=0.9,text/plain,image/png;q=0.95,text/html;q=0.5,image/jpeg;q=0.99,image*;q=1.5
Keeping aside the obvious pointlessness of this statement, it leads to several problems:
Should you consider this totally invalid, due to the several imperfections of this call, even partially breaking the standards at all (q>1 / q<0 is not allowed)?
For example, both MDN: Accept-Language and RFC7231 5.3.5 Accept-Language state, it might be okay to reject the request using HTTP/406 Not Acceptable - but advises against it due to usability reasons.
Should you expect text/plain to be q=1.0 due to its qualification not specified, even though its between two arguments not defined as q=1.0? Or "should" you process it with some kind of state machine, transfering over former values, so it becomes q=0.9?
How should you respond to conflicting information? (text/html has both q=1.0 and q=0.5) Should it get overwritten, or is it just "another" entity in the list, resulting in a duplicate?
How does the qualifier affect the order of preference, when the request could be fully satisfied by the server in all arguments, but it is provided in non-descending or even random order? I would assume, based on the resources given so far, to be expected to sort descending by the q-Value. However, the second example on the MDN page leaves this up to debate.
The latter example lists as follows:
text/html;q=0.8,text/*;q=0.8,*/*;q=0.8
which would expand to
value
priority
text/html
0.8
text/*
0.8
*/*
0.8
In this example, every value has the same q-factor, which yields the question whether the application "should" sort such statements from the most specific to the least specific, or whether it "should" be kept in the order of declarance. If so, what purpose would and could the qualifier serve, especially as - according to the declaration - every content would be "accepted" after all? I assume, when stated on MDN, the example would make sense in some way and not just be hypothetically, but I am kind of uncertain in this case. Most examples out there simply sort by the qualifier, which definitively would result in unexpected behaviour in this scenario.
And as MIME-types tend to consist of two components, according to RFC 2046, any reorder by specificality would have to consider at least two dimensions; this would lead to potential unexpected behaviour for permutations like
text/html;q=0.8,text/*;q=0.8,*/html;q=0.8,*/*;q=0.8
Also, it is unclear to me, whether I should expect parameters to the arguments, like for example
text/html;charset=utf-8;q=0.8,text/plain;charset=utf-8;q=0.8,text/html;q=0.5
as the MDN page for quality values states that there are headers like accept handling additional specifiers resulting in definitions like text/html;level=1, but not going further into detail about that. Considering the RFC7231 ABNF definitions, it seems at least possible, leading to the suggestion that an application developer shouldn't rely on matching solely for only some kind of q=<potential floating point number [\.0-9]+> representation and expecting everything remaining a potential request argument.
There might be solutions for all these hypothetical problems which feel "natural" in some way, however I am unable to find any reliable clues confirming them.
This all concludes to the following question:
Is there even a right way intented, or is this left open to the application developer?
If so, how should an application server react to the stated scenarios and why / where is it documented?
As far as I can see in my research so far, the topic seems to be rather sparse in the RFCs as well as the browser documentation, which might conclude it is designed to be more of a tool than a ruleset. But I am unsure if I am not missing out something "self-evident", as I am not aware of every RFC ever published.
As far as I can tell, the RFC and the MDN example are consistent, no?
Regarding your example:
text/html,application/xhtml+xml,application/xml;q=0.9,text/plain,image/png;q=0.95,text/html;q=0.5,image/jpeg;q=0.99,image*;q=1.5
This parses into
text/html
application/xhtml+xml
application/xml;q=0.9
text/plain,
image/png;q=0.95
text/html;q=0.5
image/jpeg;q=0.99
image*;q=1.5
where the last element is an invalid media range. It's up to you whether you want to ignore just that element or the complete header field.
I have been reading about Etags, and I understand that there are 2 ways of generating an Etag, weak and strong. Weak Etags are computationally easier to generate than strong ones. I have also come to know that Weak Etags are practically enough for most use cases.
from MDN
Weak validators are easy to generate but are far less useful for
comparisons. Strong validators are ideal for comparisons but can be
very difficult to generate efficiently.
another snippet:
Weak Etag values of two representations of the same resources might be
semantically equivalent, but not byte-for-byte identical.
I am finding it hard to understand what does it mean for a resource to be semantically similar but not byte by byte same ? It would be great to see some examples.
EDIT: found an example here, but i don't get it:
Weak Validation: The two resource representations are semantically
equivalent, e.g. some of the content differences are not important
from the business logic perspective e.g. current date displayed on the
page might not be important for updating the entire resource for it.
Is it like while generating the Etag, you can decide that the changes in content are not important for the functionality (for e.g. a css property change for font-size) and respond with 304 ? If yes, then when is the resource updated on the browser, as I guess as long as the Etag is the same , the browser would not get the latest version. In this case it might mean that when a major change happens and a new Etag is created, the css property change would only then be sent to the browser along with the major change.
My suggestion is to look at the specification, RFC 7232, section 2.1. It's only a couple pages long and may answer all of your questions.
You asked for examples, here are some from the specification:
For example, the representation of a weather report that changes in
content every second, based on dynamic measurements, might be grouped
into sets of equivalent representations (from the origin server's
perspective) with the same weak validator in order to allow cached
representations to be valid for a reasonable period of time.
A representation's modification time, if defined with only
one-second resolution, might be a weak validator if it is possible
for the representation to be modified twice during a single second
and retrieved between those modifications.
If the origin server sends the same validator for a representation with
a gzip content coding applied as it does for a representation with no
content coding, then that validator is weak.
That last one represents what is probably the most common use of weak ETags: servers converting strong ETags into weak ones when they gzip the content. Nginx does this, for example.
The specification also explains when to change a weak ETag:
An origin server SHOULD change a weak entity-tag whenever it considers prior
representations to be unacceptable as a substitute for the current
representation.
In other words, it's up to you to decide if two representations of a resource are acceptable substitutions or not. If they are, you can improve caching performance by giving them the same weak ETag.
I wanted to know whether
A file encryption algorithm can be designed in such a way that it can perform the activities of a file compression as well (any live example?).
Also can I integrate it to a Mobile SMS Service, I mean for text messages?
Also I wanted to know about binary files...if a plain text file is encoded in binary does its size reduces? Also is it better to encode a plain text file to a binary one rather than encoding it to any other format (in case if anyone wants it to encode for any purpose)
In fact all decent encryption programs (take PGP for example) compress data before encryption. They use something mainstream like ZIP for compression. The reason is once data is encrypted it looks like random noise and becomes incompressible, so it can only be compressed before encryption. You likely can't do that for SMS - you have to obey the specifications for the SMS so you'd better check those specifications.
Compression removes redundant information. Redundant information makes breaking an encryption easier. So yes, encryption and compression are compatible. I don't know if there is an algorithm designed to do both things though.
Yes, a binary file will usually be smaller than a plain text file. For instance, the number 34 written out in text takes 2 bytes (at least) whereas in those same 2 bytes you could write a number as large as 65000.
What makes an encoding "better" than another is the purpose to which it is put. If you are optimizing for size, binary is probably better. If you are optimizing for readability or graceful failure, text might be better.
First question: No reason why it shouldn't be possible. Since you can still encrypt compressed data and vise versa, you can test the benefits by trying. I don't know if it can be used for text messaging, but you'd have to explain what you're trying to do if anyone wants to give a reasonable and fitting answer.
Second question: plain text is still binary, it's just readable and encoded in a certain character set. Main difference is that plain text usually conforms to a certain encoding, say UTF-8 or ISO-8859-1.
You can still compress plain text, which makes it "binary" in the traditional sense that you only see weird bytes. :-)
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
It seems like there used to be way more binary protocols because of the very slow internet speeds of the time (dialup). I've been seeing everything being replaced by HTTP and SOAP/REST/XML.
Why is this?
Are binary protocols really dead or are they just less popular? Why would they be dead or less popular?
You Just Can't Beat the Binary
Binary protocols will always be more space efficient than text protocols. Even as internet speeds drastically increase, so does the amount and complexity of information we wish to convey.
The text protocols you reference are outstanding in terms of standardization, flexibility and ease of use. However, there will always be applications where the efficiency of binary transport will outweigh those factors.
A great deal of information is binary in nature and will probably never be replaced by a text protocol. Video streaming comes to mind as a clear example.
Even if you compress a text-based protocol (e.g. with GZip), a general purpose compression algorithm will never be as efficient as a binary protocol designed around the specific data stream.
But Sometimes You Don't Have To
The reason you are seeing more text-based protocols is because transmission speeds and data storage capacity have indeed grown fast compared to the data size for a wide range of applications. We humans find it much easier to work with text protocols, so we designed our ubiquitous XML protocol around a text representation. Certainly we could have created XML as a binary protocol, if we really had to save every byte, and built common tools to visualize and work with the data.
Then Again, Sometimes You Really Do
Many developers are used to thinking in terms of multi-GB, multi-core computers. Even your typical phone these days puts my first IBM PC-XT to shame. Still, there are platforms such as embedded devices, that have rather strict limitations on processing power and memory. When dealing with such devices, binary may be a necessity.
A parallel with programming languages is probably very relevant.
While hi-level languages are the preferred tools for most programming jobs, and have been made possible (in part) by the increases in CPU speed and storage capactity, they haven't removed the need for assembly language.
In a similar fashion, non-binary protocols introduce more abstraction, more extensibility and are therefore the vehicle of choice particularly for application-level communication. They too have benefited from increases in bandwidth and storage capacity. Yet at lower level it is still impractical to be so wasteful.
Furthermore unlike with programming languages where there are strong incentives to "take the performance hit" in exchange for added simplicity, speed of development etc., the ability to structure communication in layers makes the complexity and "binary-ness" of lower layers rather transparent to the application level. For example so long as the SOAP messages one receives are ok, the application doesn't need to know that these were effectively compressed to transit over the wire.
Facebook, Last.fm, and Evernote use the Thrift binary protocol.
I rarely see this talked about but binary protocols, block protocols especially can greatly simplify the complexity of server architectures.
Many text protocols are implemented in such a way that the parser has no basis upon which to infer how much more data is necessary before a logical unit has been received (XML, and JSON can all provide minimum necessary bytes to finish, but can't provide meaningful estimates). This means that the parser may have to periodically cede to the socket receiving code to retrieve more data. This is fine if your sockets are in blocking mode, not so easy if they're not. It generally means that all parser state has to be kept on the heap, not the stack.
If you have a binary protocol where very early in the receive process you know exactly how many bytes you need to complete the packet, then your receiving operations don't need to be interleaved with your parsing operations. As a consequence, the parser state can be held on the stack, and the parser can execute once per message and run straight through without pausing to receive more bytes.
There will always be a need for binary protocols in some applications, such as very-low-bandwidth communications. But there are huge advantages to text-based protocols. For example, I can use Firebug to easily see exactly what is being sent and received from each HTTP call made by my application. Good luck doing that with a binary protocol :)
Another advantage of text protocols is that even though they are less space efficient than binary, text data compresses very well, so the data may be automatically compressed to get the best of both worlds. See HTTP Compression, for example.
Binary protocols are not dead. It is much more efficient to send binary data in many cases.
WCF supports binary encoding using TCP.
http://msdn.microsoft.com/en-us/library/ms730879.aspx
So far the answers all focus on space and time efficiency. No one has mentioned what I feel is the number one reason for so many text-based protocols: sharing of information. It's the whole point of the Internet and it's far easier to do with text-based, human-readable protocols that are also easily processed by machines. You rid yourself of language dependent, application-specific, platform-biased programming with text data interchange.
Link in whatever XML/JSON/*-parsing library you want to use, find out the structure of the information, and snip out the pieces of data you're interested in.
Some binary protocols I've seen on the wild for Internet Applications
Google Protocol Buffers which are used for internal communications but also on, for example Google Chrome Bookmark Syncing
Flash AMF which is used for communication with Flash and Flex applications. Both Flash and Flex have the capability of communicating via REST or SOAP, however the AMF format is much more efficient for Flex as some benchmarks prove
I'm really glad you have raised this question, as non-binary protocols have multiplied in usage many folds since the introduction of XML. Ten years ago, you would see virtually everybody touting their "compliance" with XML based communications. However, this approach, one of several approaches to binary protocols, has many deficiencies.
One of the values, for example, was readability. But readability is important for debugging, when humans should read the transaction. They are very inefficient when compared with binary transfers. This is due to the fact that XML itself is a binary stream, that has to be translated using another layer into textual fragments ("tokens"), and then back into binary with the contained data.
Another value people found was extensibility. But extensibility can be easily maintained if a protocol version number for the binary stream is used at the beginning of the transaction. Instead of sending XML tags, one could send binary indicators. If the version number is an unknown one, then the receiving end can download the "dictionary" of this unknown version. This dictionary could, for example, be an XML file. But downloading the dictionary is a one time operation, instead of every single transaction!
So efficiency could be kept together with extensibility, and very easily! There are a good number of "compiled XML" protocols out there which do just that.
Last, but not least, I have even heard people say that XML is a good way to overcome little-endian and big-endian types of binary systems. For example, Sun computers vs Intel computers. But this is incorrect: if both sides can accept XML (ASCII) in the right way, surely both sides can accept binary in the right way, as XML and ASCII are also transmitted binarically.......
Hope you find this interesting reading!
Binary protocols will continue to live wherever efficency is required. Mostly, they will live in the lower-levels, where hardware-implementation is more common than software implementations. Speed isn't the only factor - the simplicity of implementation is also important. Making a chip process binary data messages is much easier than parsing text messages.
Surely this depends entirely on the application? There have been two general types of example so far, xml/html related answers and video/audio. One is designed to be 'shared' as noted by Jonathon and the other efficient in its transfer of data (and without Matrix vision, 'reading' a movie would never be useful like reading a HTML document).
Ease of debugging is not a reason to choose a text protocol over a 'binary' one - the requirements of the data transfer should dictate that. I work in the Aerospace industry, where the majority of communications are high-speed, predictable data flows like altitude and radio frequencies, thus they are assigned bits on a stream and no human-readable wrapper is required. It is also highly efficient to transfer and, other than interference detection, requires no meta data or protocol processing.
So certainly I would say that they are not dead.
I would agree that people's choices are probably affected by the fact that they have to debug them, but will also heavily depend on the reliability, bandwidth, data type, and processing time required (and power available!).
They are not dead because they are the underlying layers of every communication system. Every major communication system's data link and network layers are based on some kind of "binary protocol".
Take the internet for example, you are now probably using Ethernet in your LAN, PPPoE to communicate with your ISP, IP to surf the web and maybe FTP to download a file. All of which are "binary protocols".
We are seeing this shift towards text-based protocols in the upper layers because they are much easier to develop and understand when compared to "binary protocols", and because most applications don't have strict bandwidth requirements.
depends on the application...
I think in real time environment (firewire, usb, field busses...) will always be a need for binary protocols
Are binary protocols dead?
Two answers:
Let's hope so.
No.
At least a binary protocol is better than XML, which provides all the readability of a binary protocol combined with all the efficiency of less efficiency than a well-designed ASCII protocol.
Eric J's answer pretty much says it, but here's some more food for thought and facts. Note that the stuff below is not about media protocols (videos, images). Some items may be clear to you, but I keep hearing myths every day so here you go ...
There is no difference in expressiveness between a binary protocol and a text protocol. You can transmit the same information with the same reliability.
For every optimum binary protocol, you can design an optimum text protocol that takes just around 15% more space, and that protocol you can type on your keyboard.
In practice (practical protocols is see every day), the difference is often even less significant due to the static nature of many binary protocols.
For example, take a number that can become very large (e.g., in 32 bit range) but is often very small. In binary, people model this usually as four bytes. In text, it's often done as printed number followed by colon. In this case, numbers below ten become two bytes and numbers below 100 three bytes. (You can of course claim that the binary encoding is bad and that you can use some size bits to make it more space efficient, but that's another thing that you have to document, implement on both sides, and be able to troubleshoot when it comes over your wire.)
For example, messages in binary protocols are often framed by length fields and/or terminators, while in text protocols, you just use a CRC.
In practice, the difference is often less significant due to required redundancy.
You want some level of redundancy, no matter if it's binary or text. Binary protocols often leave no room for error. You have to 100% correctly document every bit that you send, and since most of us are humans, that happens rarely and you can't read it well enough to make a safe conclusion what is correct.
So in summary: Binary protocols are theoretically more space and compute efficient, but the difference is in practice often less than you think and the deal is often not worth it. I am working in the Internet of Things area and have to deal nearly on daily base with custom, badly designed binary protocols which are really hard to troubleshoot, annoying to implement and not more space efficient. If you don't need to absolutely tweak the last milliampere out of your battery and calculate with microcontroller cycles (or transmit media), think twice.