I am having trouble on finding authoritative information about the behavior with HTTP GET query string duplicate fields, like
http://example.com/page?field=foo&field=bar
and in particular if the order is kept or not. Most web-oriented languages produce an array containing both foo and bar associated to a key "field", but I would like to know if authoritative statement exist (e.g. on a RFC) about this point. RFC 3986 has a section 3.4. Query, which refers to key=value pairs, but nothing is said on how to interpret order and duplicate fields and so on. This makes sense, since it's backend dependent, and not in the scope of that RFC...
Although a de-facto standard exists, I'd like to see an authoritative source for it, just out of curiosity.
There is no spec on this. You may do what you like.
Typical approaches include: first-given, last-given, array-of-all, string-join-with-comma-of-all.
Suppose the raw request is:
GET /blog/posts?tag=ruby&tag=rails HTTP/1.1
Host: example.com
Then there are various options for what request.query['tag'] should yield, depending on the language or the framework:
request.query['tag'] => 'ruby'
request.query['tag'] => 'rails'
request.query['tag'] => ['ruby', 'rails']
request.query['tag'] => 'ruby,rails'
The situation seems to have changed since this question was asked and the accepted answer was written 12 years ago. I believe we now have an authoritative source: The WHATWG URL Standard describes the process of extracting and parsing a query string in detail in section 6.2 (https://url.spec.whatwg.org/#interface-urlsearchparams) and section 5.1 on x-www-form-urlencoded parsing (https://url.spec.whatwg.org/#urlencoded-parsing). The parsing output is "an initially empty list of name-value tuples where both name and value hold a string", where a list is defined as a finite ordered sequence, and the key-value pairs are added to this list in the order they appear in the URL. At first there is no mention of repeated keys, but some methods on the URLSearchParams class in section 6.2 (https://url.spec.whatwg.org/#interface-urlsearchparams) set clear expectations on ordering: "The getAll(name) method steps are to return the values of all name-value pairs whose name is name... in list order"; The sort() method specifies that "The relative order between name-value pairs with equal names must be preserved." (Emphasis mine). Examining the Github issue referenced in the commit where the sort method was added, we see that the original proposal was to sort on values where keys were identical, but this was changed: "The reason for the default sort not affecting the value order is that ordering of the values can be significant. We should not assume that it's ok to move the order of the values around." (https://github.com/whatwg/url/issues/26#issuecomment-271600764)
I can confirm that for PHP (at least in version 4.4.4 and newer) it works like this:
GET /blog/posts?tag=ruby&tag=rails HTTP/1.1
Host: example.com
results in:
request.query['tag'] => 'rails'
But
GET /blog/posts?tag[]=ruby&tag[]=rails HTTP/1.1
Host: example.com
results in:
request.query['tag'] => ['ruby', 'rails']
This behavior is the same for GET and POST data.
yfeldblum's answer is perfect.
Just a note about a fifth behavior I noticed recently: on Windows Phone, opening an application with an uri with a duplicate query key will result in NavigationFailed with:
System.ArgumentException: An item with the same key had already been added.
The culprit is System.Windows.Navigation.UriParsingHelper.InternalUriParseQueryStringToDictionary(Uri uri, Boolean decodeResults).
So the system won't even let you handle it the way you want, it will forbid it. You are left with the only solution to choose your own format (CSV, JSON, XML, ...) and uri-escape-it.
Most (all?) of the frameworks offer no guarantees, so assume they will be returned in random order.
Always take the safest approach.
For example, java HttpServlet interface:
ServletRequest.html#getParameterValues
Even the getParameterMap method leaves out any mention about parameter order (the order of a java.util.Map iterator cannot be relied on either.)
Typically, duplicate parameter values like
http://example.com/page?field=foo&field=bar
result in a single queryString parameter that is an array:
field[0]=='foo'
field[1]=='bar'
I've seen this behavior in ASP, ASP.NET and PHP4.
The ?array[]=value1&array[]=value2 approach is certainly a very popular one.
supported by most Javascript frameworks
supported by Java Spring
supported by PHP
Related
My application has a resource at /foo. Normally, it is represented by an HTTP response payload like this:
{"a": "some text", "b": "some text", "c": "some text", "d": "some text"}
The client doesn't always need all four members of this object. What is the RESTfully semantic way for the client to tell the server what it needs in the representation? e.g. if it wants:
{"a": "some text", "b": "some text", "d": "some text"}
How should it GET it? Some possibilities (I'm looking for correction if I misunderstand REST):
GET /foo?sections=a,b,d.
The query string (called a query string after all) seems to mean "find resources matching this condition and tell me about them", not "represent this resource to me according to this customization".
GET /foo/a+b+d My favorite if REST semantics doesn't cover this issue, because of its simplicity.
Breaks URI opacity, violating HATEOAS.
Seems to break the distinction between resource (the sole meaning of a URI is to identify one resource) and representation. But that's debatable because it's consistent with /widgets representing a presentable list of /widget/<id> resources, which I've never had a problem with.
Loosen my constraints, respond to GET /foo/a, etc, and have the client make a request per component of /foo it wants.
Multiplies overhead, which can become a nightmare if /foo has hundreds of components and the client needs 100 of those.
If I want to support an HTML representation of /foo, I have to use Ajax, which is problematic if I just want a single HTML page that can be crawled, rendered by minimalist browsers, etc.
To maintain HATEOAS, it also requires links to those "sub-resources" to exist within other representations, probably in /foo: {"a": {"url": "/foo/a", "content": "some text"}, ...}
GET /foo, Content-Type: application/json and {"sections": ["a","b","d"]} in the request body.
Unbookmarkable and uncacheable.
HTTP does not define body semantics for GET. It's legal HTTP but how can I guarantee some user's proxy doesn't strip the body from a GET request?
My REST client won't let me put a body on a GET request so I can't use that for testing.
A custom HTTP header: Sections-Needed: a,b,d
I'd rather avoid custom headers if possible.
Unbookmarkable and uncacheable.
POST /foo/requests, Content-Type: application/json and {"sections": ["a","b","d"]} in the request body. Receive a 201 with Location: /foo/requests/1. Then GET /foo/requests/1 to receive the desired representation of /foo
Clunky; requires back-and-forth and some weird-looking code.
Unbookmarkable and uncacheable since /foo/requests/1 is just an alias that would only be used once and only kept until it is requested.
I would suggest the querystring solution (your first). Your arguments against the other alternatives are good arguments (and ones that I've run into in practise when trying to solve the same problem). In particular, the "loosen the constraints/respond to foo/a" solution can work in limited cases, but introduces a lot of complexity into an API from both implementation and consumption and hasn't, in my experience, been worth the effort.
I'll weakly counter your "seems to mean" argument with a common example: consider the resource that is a large list of objects (GET /Customers). It's perfectly reasonable to page these objects, and it's commonplace to use the querystring to do that: GET /Customers?offset=100&take=50 as an example. In this case, the querystring isn't filtering on any property of the listed object, it's providing parameters for a sub-view of the object.
More concretely, I'd say that you can maintain consistency and HATEOAS through these criteria for use of the querystring:
the object returned should be the same entity as that returned from the Url without the querystring.
the Uri without the querystring should return the complete object - a superset of any view available with a querystring at the same Uri. So, if you cache the result of the undecorated Uri, you know you have the full entity.
the result returned for a given querystring should be deterministic, so that Uris with querystrings are easily cacheable
However, what to return for these Uris can sometimes pose more complex questions:
returning a different entity type for Uris differing only by querystring could be undesirable (/foo is an entity but foo/a is a string); the alternative is to return a partially-populated entity
if you do use different entity types for sub-queries then, if your /foo doesn't have an a, a 404 status is misleading (/foo does exist!), but an empty response may be equally confusing
returning a partially-populated entity may be undesirable, but returning part of an entity may not be possible, or may be more confusing
returning a partially populated entity may not be possible if you have a strong schema (if a is mandatory but the client requests only b, you are forced to return either a junk value for a, or an invalid object)
In the past, I have tried to resolve this by defining specific named "views" of required entities, and allowing a querystring like ?view=summary or ?view=totalsOnly - limiting the number of permutations. This also allows for definition of a subset of the entity that "makes sense" to the consumer of the service, and can be documented.
Ultimately, I think that this comes down to an issue of consistency more than anything: you can meet HATEOAS guidance using the querystring relatively easily, but the choices you make need to be consistent across your API and, I'd say, well documented.
I've decided on the following:
Supporting few member combinations: I'll come up with a name for each combination. e.g. if an article has members for author, date, and body, /article/some-slug will return all of it and /article/some-slug/meta will just return the author and date.
Supporting many combinations: I'll separate member names by hyphens: /foo/a-b-c.
Either way, I'll return a 404 if the combination is unsupported.
Architectural constraint
REST
Identifying resources
From the definition of REST:
a resource R is a temporally varying membership function MR(t), which for time t maps to a set of entities, or values, which are equivalent. The values in the set may be resource representations and/or resource identifiers.
A representation being an HTTP body and an identifier being a URL.
This is crucial. An identifier is just a value associated with other identifiers and representations. That's distinct from the identifier→representation mapping. The server can map whatever identifier it wants to any representation, as long as both are associated by the same resource.
It's up to the developer to come up with resource definitions that reasonably describe the business by thinking of categories of things like "users" and "posts".
HATEOAS
If I really care about perfect HATEOAS, I could put a hyperlink somewhere in the /foo representation to /foo/members, and that representation would just contain a hyperlink to every supported combination of members.
HTTP
From the definition of a URL:
The query component contains non-hierarchical data that, along with data in the path component, serves to identify a resource within the scope of the URI's scheme and naming authority (if any).
So /foo?sections=a,b,d and /foo?sections=b are distinct identifiers. But they can be associated within the same resource while being mapped to different representations.
HTTP's 404 code means that the server couldn't find anything to map the URL to, not that the URL is not associated with any resource.
Functionality
No browser or cache will ever have trouble with slashes or hyphens.
Actually it depends on the functionality of the resource.
If for example the resource represents an entity:
/customers/5
Here the '5' represents an id of the customer
Response:
{
"id": 5,
"name": "John",
"surename": "Doe",
"marital_status": "single",
"sex": "male",
...
}
So if we will examine it closely, each json property actually represents a field of the record on customer resource instance.
Let's assume consumer would like to get partial response, meaning, part of the fields. We can look at it as the consumer wants to have the ability to select the various fields via the request, which are interesting to him, but not more (in order to save traffic or performance, if part of the fields are hard to compute).
I think in this situation, the most readable and correct API would be (for example, get only name and surename)
/customers/5?fields=name,surename
Response:
{
"name": "John",
"surename": "Doe"
}
HTTP/1.1
if illegal field name is requested - 404 (Not Found) is returned
if different field names are requested - different responses will be generated, which also aligns with the caching.
Cons: if the same fields are requested, but the order is different between the fields (say: fields=id,name or fields=name,id), although the response is the same, those responses will be cached separately.
HATEOAS
In my opinion pure HATEOAS is not suitable for solving this particular problem. Because in order to achieve that, you need a separate resource for every permutation of field combinations, which is overkill, as it is bloating the API extensively (say you have 8 fields in a resource, you will need permutations!).
if you model resources only for the fields but not all the permutations, it has performance implications, e.g. you want to bring the number of round trips to minimum.
If a,b,c are property of a resource like admin for role property the right way is to use is the first way that you've suggested GET /foo?sections=a,b,d because in this case you would apply a filter to the foo collection. Otherwise if a,b and c are a singole resource of foo collection the the way that would follow is to do a series of GET requests /foo/a /foo/b /foo/c. This approach, as you said, has a high payload for request but it is the correct way to follow the approach Restfull. I would not use the second proposal made by you because plus char in a url has a special meaning.
Another proposal is to abandon use GET and POST and create an action for the foo collection like so: /foo/filter or /foo/selection or any verb that represent an action on the collection. In this way, having a post request body, you can pass a json list of the resource you would.
you could use a second vendor media-type in the request header application/vnd.com.mycompany.resource.rep2, you can't bookmark this however, query-parameters are not cacheable (/foo?sections=a,b,c) you could take a look at matrix-parameters however regarding this question they should be cacheable URL matrix parameters vs. request parameters
I know that in most MVC frameworks, for example, both query string params and form params will be made available to the processing code, and usually merged into one set of params (often with POST taking precedence). However, is it a valid thing to do according to the HTTP specification? Say you were to POST to:
http://1.2.3.4/MyApplication/Books?bookCode=1234
... and submit some update like a change to the book name whose book code is 1234, you'd be wanting the processing code to take both the bookCode query string param into account, and the POSTed form params with the updated book information. Is this valid, and is it a good idea?
Is it valid according HTTP specifications ?
Yes.
Here is the general syntax of URL as defined in those specs
http_URL = "http:" "//" host [ ":" port ] [ abs_path [ "?" query ]]
There is no additional constraints on the form of the http_URL. In particular, the http method (i.e. POST,GET,PUT,HEAD,...) used don't add any restriction on the http URL format.
When using the GET method : the server can consider that the request body is empty.
When using the POST method : the server must handle the request body.
Is it a good idea ?
It depends what you need to do. I suggest you this link explaining the ideas behind GET and POST.
I can think that in some situation it can be handy to always have some parameters like the user language in the query part of the url.
I know that in most MVC frameworks, for example, both query string params and form params will be made available to the processing code, and usually merged into one set of params (often with POST taking precedence).
Any competent framework should support this.
Is this valid
Yes. The POST method in HTTP does not impose any restrictions on the URI used.
is it a good idea?
Obviously not, if the framework you are going to use is still clue-challenged. Otherwise, it depends on what you want to accomplish. The major use case (redirection of a data subset to a new POST target) has been irretrievably broken by browser implementations (all mechanically following the broken lead of Mosaic/Netscape), so the considerations here are mostly theoretical.
I have been reading up on the HTTP/1.1 headers and in some of the sample headers in section 14.1 (Accept) they use accept-extensions (I believe that is what they are) called level=1, level=2, etc.
The problem I am having is that they use these level=X things as if it should be obvious what they do. Is the document just poor at explaining it or am I missing something?
Thanks.
...A Couple Years Later
The user TomWardrop points out that (from comments to this answer):
It use to be part of the text/html media type specification to indicate what version of HTML you wanted, but is wasn't used much (if at all), so it was dropped. Refer to ietf.org/rfc/rfc2854.txt.
Excerpt from the RFC:
Note that [HTML20] included an optional "level" parameter; in
practice, this parameter was never used and has been removed from
this specification. [HTML30] also suggested a "version"
parameter; in practice, this parameter also was never used and has
been removed from this specification.
This appears to be the best answer so far.
I am leaving the original answer below for posterity.
Summary
It appears from looking through RFC-2616 (Request For Comments) over at IETF and W3C, as well as websites elsewhere on the internet, that the level extension is not very well documented or explained. It also does not appear to be in use in headers by anyone, suggesting that it can probably be ignored.
The RFCs
In the RFCs the level parameter is seen in a few examples, but it is never mentioned or clearly expressed exactly what role it plays.
level is used in an example about precedence:
Media ranges can be overridden by more specific media ranges or
specific media types. If more than one media range applies to a given
type, the most specific reference has precedence. For example,
Accept: text/*, text/html, text/html;level=1, */*
have the following precedence:
1) text/html;level=1
2) text/html
3) text/*
4) */*
Source: IEFT RFC-2616 p.100 and W3C RFC2616 section "14.1 Accept"
Seeing the difference between how the types are ordered in the two examples, it looks like text/html;level=1 has greater precedence than text/html, meaning that the level extension must give it that precedence. The last two are obviously ordered further, according to declining specificity.
Now this brings up the quality factor, q. It is explained quite well in the RFCs. It can be anything between 0 and 1. The bigger the value the more precedence the type has. The RFC has an example using both q and level:
The media type quality factor associated with a given type is
determined by finding the media range with the highest precedence
which matches that type. For example,
Accept: text/*;q=0.3, text/html;q=0.7, text/html;level=1,
text/html;level=2;q=0.4, */*;q=0.5
would cause the following values to be associated:
text/html;level=1 = 1
text/html = 0.7
text/plain = 0.3
image/jpeg = 0.5
text/html;level=2 = 0.4
text/html;level=3 = 0.7
Source: IEFT RFC-2616 p.100 and W3C RFC2616 section "14.1 Accept"
From this it appears that level's value is used in decreasing precedence (1 has highest precedence, 2 second highest, etc.). That makes sense, but when taken together with the Accept: header it makes no sense at all:
First, the q parameters have magically disappeared from the types, in the associations (below). It is like the author assumed that it is obvious why they have been omitted, but forgot to tell us why it is obvious.
Second, the types in the Accept: header, and the types shown in the associations (below), are not the same types. E.g the image/jpeg type is never mentioned in the header, and the text/* type is missing from the associations.
I am at a loss to explain what it all means.
Zend Framework Discussion
On an answer to a question asking about the level parameter there is some interesting stuff.
q and level compliment each other
[...] The q-factor is the one most looked at. However, you can also specify a "level", and these CAN also act like priorities, but operate in the order of decreasing precedence (i.e., level 1 is higher priority than level 2). The examples they have in the spec are contrived, and, tbh, confusing at best [...]
The q parameter is what you should use, and the level parameter can be used. Okay, but it still does not clear up exactly what it does, and how it is supposed to affect the priorities of types.
Used for supported type/spec version
Another documented use case for the "level" is to indicate the _version_ of the type. As an example, a "level 1" might indicate the first version of that spec available. In such cases, it wouldn't be a priority, but instead a _descriptor_ [...]
So, a way of indicating what version of the type is supported. Now, this actually makes more sense:
text/html;level=5;q=1
text/html;level=4.01;q=0.9
; Etc.
Somehow I doubt level was supposed to be used for this, however. If it was then it would probably have been called something more descriptive, such as version, or simply v (like q).
Conflicting meanings
Unfortunately, the "level" selector has conflicting meanings, so I'm not 100% sure we should support it by default; "q", on the other hand, is well documented.
Yeah. Conflicting meanings and not very well documented. The level property has very little going for it.
Other stuff I found
Looking for questions/answers elsewhere on the Internet I found:
A seriously old-school document that does not mention level at all. It actually mentions two other ones, mxs and mxb.
A Moz Dev page listing common Accept headers from various browsers. Nowhere is level used. In fact, I have never seen it used outside the RFCs.
Conclusion
In conclusion I think it is safe to say that level is a waste of time. It is poorly documented, not used much in practice (if used at all), and it is more confusing than it is worth.
"level" is just an example for a media type parameter. It's not involved in computing the preference.
(The relevant spec nowadays is http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p2-semantics-26.html#header.accept)
Expanding on Julian Reschke's correct answer a little. The answer mentions "media type parameter" which comes directly from the specification:
Accept = #( media-range [ accept-params ] )
media-range = ( "*/*" / ( type "/" "*" ) / ( type "/" subtype )) *( OWS ";" OWS parameter )
accept-params = weight *( accept-ext )
accept-ext = OWS ";" OWS token [ "=" ( token / quoted-string ) ]
Where parameter is defined as:
parameter = token "=" ( token / quoted-string )
That is, a name/value pair (see token). So, in the examples, "level=1" is just an example of a legal "parameter". In that sense, there's nothing really special about it. If you take the time to parse the Backus-Naur of the definition, you'll see that, in all of the examples, the "level=x" must be a media-type parameter.
With regard to why the example specifically uses "level", I think the "level" in the HTTP spec examples refers to HTML levels. In all of the examples, level modifies the text/html media type. The best definition I've found for HTML levels is:
"There are four official levels [0-4], or versions of HTML conformance. Each encompasses a set of tags and higher levels include tags from all those below it."
http://scholar.lib.vt.edu/reports/soasis-slides/slide4.html
I.e., a request could use Accept to ask for a lower level of HTML if it wanted a subset of HTML elements/attributes. As an example, HTML level 0 defines the DOCTYPE element, but does not define the "Version" attribute until level 3; so that HTML levels 0-2 would/should not contain a Version attribute in a <!DOCTYPE> declaration.
http://www.december.com/html/spec/level0.html
http://www.december.com/html/spec/level3.html
"level" seems like an old aspect/feature of HTML. I can find very few references to it, none of them are clear (e.g., is there a difference between HTML level and major version?), and most of those I can find are old. However, the latest Apache httpd server docs still say they support "level" in content negotiation, and suggest that it may be synonymous with version, at least in that implementation:
"4. Select the variants with the highest 'level' media parameter (used to give the version of text/html media types)."
Note that the IANA registry doesn't mention anything about "level" as a text/html parameter. If nothing else, that says to me: "don't use it."
Interestingly, the DOM specification still uses the word "level".
http://www.w3.org/TR/DOM-Level-2-HTML/
http://www.w3.org/TR/DOM-Level-3-Core/
etc..
Should the following URLs be considered functionally equivalent?
http://example.com/foo?a=&b=
http://example.com/foo?a&b
This came about when a user of a Drupal module I wrote which parses apart and then rewrites URIs noticed that the code sometimes causes the query string parts to change in unexpected ways due to how some of the underlying PHP functions behave. For example:
parse_str("a&b", $values); print http_build_query($values);
a=&b=
Is this something I should bother worrying about?
Edit so SO stops complaining that this question is similar to another one: The question is whether it's safe to assume that "no value for X" and "empty value for X" are equivalent, not whether the "no value" style is syntactically correct (which it is).
RFC 3986 Uniform Resource Identifier (URI): Generic Syntax doesn't have anything to say about the structure of the query string aside from how characters like ? should be dealt with. So strictly speaking, your two example URLs are different. Of course, the application which receives those query strings may treat them as functionally equivalent, but this isn't something you can determine from the URL alone.
As per RFC6570 empty query parameters are allowed. Please refer to section 3.2.9
Example Template Expansion
{&x,y,empty} &x=1024&y=768&empty=
I was wondering if it's acceptable to put custom data in an HTTP authorization header. We're designing a RESTful API and we may need a way to specify a custom method of authorization. As an example, let's call it FIRE-TOKEN authentication.
Would something like this be valid and allowed according to the spec: Authorization: FIRE-TOKEN 0PN5J17HBGZHT7JJ3X82:frJIUN8DYpKDtOLCwo//yllqDzg=
The first part of the second string (before the ':') is the API key, the second part is a hash of query string.
The format defined in RFC2617 is credentials = auth-scheme #auth-param. So, in agreeing with fumanchu, I think the corrected authorization scheme would look like
Authorization: FIRE-TOKEN apikey="0PN5J17HBGZHT7JJ3X82", hash="frJIUN8DYpKDtOLCwo//yllqDzg="
Where FIRE-TOKEN is the scheme and the two key-value pairs are the auth parameters. Though I believe the quotes are optional (from Apendix B of p7-auth-19)...
auth-param = token BWS "=" BWS ( token / quoted-string )
I believe this fits the latest standards, is already in use (see below), and provides a key-value format for simple extension (if you need additional parameters).
Some examples of this auth-param syntax can be seen here...
https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-p7-auth-19#section-4.4
https://developers.google.com/youtube/2.0/developers_guide_protocol_clientlogin
https://developers.google.com/accounts/docs/AuthSub#WorkingAuthSub
Put it in a separate, custom header.
Overloading the standard HTTP headers is probably going to cause more confusion than it's worth, and will violate the principle of least surprise. It might also lead to interoperability problems for your API client programmers who want to use off-the-shelf tool kits that can only deal with the standard form of typical HTTP headers (such as Authorization).
No, that is not a valid production according to the "credentials" definition in RFC 2617. You give a valid auth-scheme, but auth-param values must be of the form token "=" ( token | quoted-string ) (see section 1.2), and your example doesn't use "=" that way.
Old question I know, but for the curious:
Believe it or not, this issue was solved ~2 decades ago with HTTP BASIC, which passes the value as base64 encoded username:password. (See http://en.wikipedia.org/wiki/Basic_access_authentication#Client_side)
You could do the same, so that the example above would become:
Authorization: FIRE-TOKEN MFBONUoxN0hCR1pIVDdKSjNYODI6ZnJKSVVOOERZcEtEdE9MQ3dvLy95bGxxRHpnPQ==