Incorrect content length due to eol appending

[bbkr]

HTTP::Message in Str method is silently appending EOL:

https://github.com/sergot/http-useragent/blob/cfdfc48c55d312049f90f1db61042718c1de9908/lib/HTTP/Message.pm6#L231

This causes Content-Length calculated by user to be incorrect:

$response = HTTP::Response.new;
$response.set-code( 200 );

my $content = 'foo';   # 3 bytes
$response.add-content( $content );
$response.field( Content-Length => $content.encode.bytes );

$socket.print( $response.Str ); # will send 2 bytes more than declared

Why is this EOL appended? User content should not be tampered with. Is this behavior described somewhere in RFC?

[zjmarlow]

the current relevant code: lib/HTTP/Message.rakumod#L224
t/020-message.rakutest might need updating.

rfc 2616 seems to be the relevant document.

rfc2616 section 4.1 Message Types prohibits extra CRLF in certain situations.

RFC 2616 section 4.4 Message Length, item 3 mentions "The Content-Length header field MUST NOT be sent if..." entity length differs from transfer length and also mentions that the Content-Length header field MUST be ignored if a Transfer-Encoding header is also present.

I'm still reading through the RFC and will post again if I figure out what the correct behavior is.

[lizmat]

Thanks. A PR would also be very much appreciated.

[zjmarlow]

It looks like the only change in Message.rakumod needed is removing the extra eol from line 224. After that, I'll review t/020-message.rakutest, t/040-request.rakutest, and t/050-response.rakutest. After making the changes, I'll submit a pull request and include notes and references to the RFC.

It's going to take more reading about Chunked Transfer Coding to see if the planned changes are correct.

[zjmarlow]

The application the fix is needed for currently only requires correcting how eol is handled in the response. However, the RFC permits chunked content in requests as well. Fixing that would require additional changes. I am okay with making whichever set of edits is needed. Please let me know whether to address requests in addition to responses (or if more detail is needed).