cookie-octet reality check

[bagder]

Hello,

In draft-10 section 4.1.1 we see:

cookie-octet      = %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E
                   / %x80-FF
                     ; octets excluding CTLs,
                     ; whitespace DQUOTE, comma, semicolon,
                     ; and backslash

This means that space, comma and double-quotes for example are invalid contents in cookie values and names. Why?

In RFC 6265 the same section says:

cookie-octet      = %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E
                   ; US-ASCII characters excluding CTLs,
                   ; whitespace DQUOTE, comma, semicolon,
                   ; and backslash

(the difference is %x80-FF which now is explicitly allowed)

Firefox does not ignore all those and claims "parity with Chrome" on this.

If we ignore such cookies, we break compatibility with two major browsers. If we don't, we don't follow the spec.

[reschke]

99a9d4a

(so 0x80-FF are gone again)

[bagder]

Ah thanks for pointing that out. I think they are slightly less problematic than the more common ASCII bytes though.

[sbingler]

Hi @bagder,

Section 4 defines how a well behaved server should format their cookies. Unfortunately not all servers are well behaved so in the interest of compatibility UAs are encouraged to implement a more permissive syntax as mentioned in Section 5.

This section specifies the Cookie and Set-Cookie header fields in
sufficient detail that a user agent implementing these requirements
precisely can interoperate with existing servers (even those that do
not conform to the well-behaved profile described in Section 4).

A user agent could enforce more restrictions than those specified
herein (e.g., restrictions specified by its cookie policy, described
in Section 7.2). However, such additional restrictions may reduce
the likelihood that a user agent will be able to interoperate with
existing servers.

Section-5.4 goes into more detail.

This means that space, comma and double-quotes for example are invalid contents in cookie values and names. Why?

I would imagine it's due to potential parsing errors and other ambiguities, but that's just an educated guess.
Set-Cookie: foo=bar, "Set-Cookie: foo2=bar2" seems pretty foot gunny to me.

[bagder]

What's the point in specifying a format that isn't followed by implementations? Isn't it much smarter to align that with what is actually in use in the world so that the spec can help implementers?

[sbingler]

What's the point in specifying a format that isn't followed by implementations?

By which implementations? The spec exists for UAs consuming cookies as well as the servers producing them.

Note that I wasn't around when the original decision to split the formats in this manner was made, so take this all with a grain of salt. But with my knowledge of the space I believe it's because different parts of the cookie ecosystem have different goals and need somewhat different guidance to reach those goals. Generally speaking though I assume that implementers want their work to be compatible with their counterparts' implementation:

• Are you a web-dev/server-dev looking to be as widely compatible as possible across UAs and versions? You're better off producing the more restrictive syntax to minimize the possibility that your cookie string is misparsed by any UA.

• Are you a UA developer looking to be as widely compatible as possible across websites/servers/origins? You're better off accepting a more relaxed syntax to minimize the number of cookies you can't parse from older/misconfigured servers.

Each side, server and UA, should ideally try to align on their own format within the spec but they can't always assume their counterpart will. E.x.: Maybe a user is running some ancient browser to visit a site or maybe someone on the latest Firefox version wants to visit a site that hasn't been updated since the 90s.

(And finally, if you don't care about being widely compatible you can do whatever you want.)

[bagder]

The split-description is incomprehensible and just makes readers of the spec to get it wrong and not read both descriptions (I know I'm not alone in repeatedly doing this mistake). There is just one Set-Cookie: field; it has one defined format. The spec should then explain what format it has. If that format is not followed, compliant receivers should ignore it (as the spec says).

Right now, the spec specifies a way how servers should send the cookie, but none of the popular browsers care about that format and happily accept cookies not following it, because you need to read the second format for the same header to see that a receiver can be much more liberal. So the single header field has... two formats?

I care deeply about compatibility. That's why I submitted this issue. I think the current way of specifying it makes us all a disservice.

[bagder]

By which implementations

The very reason existing cookie receivers don't use the same strict checks as is documented for the server side is of course because many servers don't comply. Otherwise there wouldn't be a need to write them differently.

[bagder]

It looks like #1759 is a fix for this.

[sbingler]

To make sure I understand:
Is your proposal to fix this issue to make the UA parser requirements more obvious (in some way) to readers of the spec? And therefore reduce the chance that a UA will implement the server syntax accidentally?

[bagder]

I think the way it currently is phrased easily misleads readers of the spec. I think it should be improved to reduce that risk. I don't think there is a server-syntax and a separate client-syntax of the same header field.

And basically, since the server-syntax clearly isn't obeyed in the wild, it should be fixed to better match reality.

[sbingler]

So then is your proposal to remove the syntax as described currently in section 4 and replace its usage with the syntax as described in Section 5.4? I.e.: To recommend that all implementers of cookies (UA and servers alike) use the section 5.4 syntax?

[bagder]

Yes. In spirit with how HTTP header fields are documented/specified in other HTTP related RFCs: yes. One header field, one syntax in one place.

[sbingler]

I agree that having a single syntax to refer to would be ideal, unfortunately the web has not been ideal with respect to cookies with many erroneous and incomplete implementations having been released on both sides along with simply outdated versions (likely with many still in use today).

Many of the designs and considerations that have gone into the spec have been to maximize compatibility, including backwards compatibility, between servers and UAs in the real world and that has required some concessions which includes the server/UA syntax differences. ("Lax-Allowing-Unsafe" enforcement is another example.)

The chief benefit of using a single syntax appears to be ease of implementation, mainly locating the correct grammar within the document, with a minor benefit (for the servers) of a slightly expanded character set. These seem appealing but I expect there will be subtle problems arising from allowing more characters:
While current versions of Chrome, Firefox, and likely Safari all implement the more lax syntax, there are other UAs that may not. These other UAs would begin to drop cookies (with the expanded character set) but such breakage would be hidden from the server operator and likely never fixed. This is obviously bad for compatibility. These effects are dispersed and difficult to quantify but are still important to consider.

Both of the benefits of a single syntax seem to me to be minor compared to the compatibility risks. I acknowledge that there are existing servers that do not follow the stricter syntax, but I feel we should not encourage even more to do so.

I understand that a UA inadvertently implementing the server syntax is frustrating, maybe there is something we can do to reduce the chances of that happening. Perhaps a more obvious note or some other strong indication that there are different requirements for different implementors.

[bagder]

While current versions of Chrome, Firefox, and likely Safari all implement the more lax syntax, there are other UAs that may not

That's exactly what brought me to file this issue to begin with. The current spec combined with the current state of widely used implementations cause cookie interop problems. I argue these problems already exist and I propose a way that could work to reduce them. I cannot agree with your risk assessment.