Add support for incoming block-wise transfers

[felixletkemann]

When I do a request to my rails application using postman, I get a hash in the parameters variable, looking like:
=> <ActionController::Parameters {"book"=>{"author"=>"ernest hemingway", "isbn"=>"1234567"}, "format"=>"json", "controller"=>"api/v1/book", "action"=>"create"} permitted: false>

When I do the same request over CoAP using Copper in Firefox, I get:

=> <ActionController::Parameters {"{ book: { author: \"ernest hemingway\", isbn: \"1234567"=>nil, "format"=>"json", "controller"=>"api/v1/book", "action"=>"create"} permitted: false>

Something must be doing strange things with the params variable.

[felixletkemann]

The same issue exists when using sinatra. The parameters variable will be converted to this very strange format mentioned above. Current example:
{ position: { lat: 3.14, lng: 49.2332, imei: 9FJ489FJW938FJW39F8JW39F } }
result:
{"{ position: { lat: 3.14, lng: 49.2332, imei: 9FJ489FJW938FJW39F8"=>nil}

[nning]

Thanks for reporting!

I think, I saw this issue some time ago and it popped up again in connection with the update to Rack 2, now. Are you using Rack 2?

I will look into this as soon as I have some time; I'm currently quite short of it.

[felixletkemann]

Yes, I do use rack 2 and I would not like to downgrade to 1.6.8 since that breaks the compatibility with some other gems I use.
Maybe you could give me a hint where you would expect the issue to be and I will try to have a look into it.

[felixletkemann]

Maybe the error is happening outside of david. I have noticed, that the function dispatch already gets the data corrupted. It looks like the UDPSocket is delivering corrupted data. Also, this does not happen if the payload is short enough! The payload is okay, as long as its length is 64 chars or below. Any longer payload will be cut of, which results in a fucked up json which can cause parsing errors later on.

In the attachment you will find a picture, that shows the part of code I was talking about.

[felixletkemann]

Maybe the bug is caused by the recvmsg_nonblock method? Maybe that limits the number of chars that can be read at a time.

[nning]

Are you on MRI, JRuby or Rubinius?

In your position example, the payload seems to be cut off after 64 bytes; did you set the block size to 64 bytes in the "Behaviour" menu of Copper?

Did you activate block-wise transfer? (CoAP message payloads SHOULD not be bigger than 1024 bytes (see RFC 7252, Section 4.6).

(Oddly, my recently updated Copper (1.0.1) does not send outgoing block-wise transfers anymore.)

I do not know of any limitation of recvmsg_nonblock.

[nning]

Oh, I just remembered, David currently does not support incoming block-wise transfers.

[nning]

Would you like to add this feature?

[felixletkemann]

Your guess was correct, the maximum block size set by copper was the reason for all the trouble I have had. Unfortunately, I do not have the deep knowledge about CoAP required to add this feature. But I think, most IoT applications will not need any messages larger than 1024 bytes anyway.
Thank you very much for your support!

[felixletkemann]

I have noticed that there is still some strange escaping behaviour, even if there is no problem with block wise transfer. After spending quite some time I noticed, that the string is okay inside the env variable in david but is fucked up when I am trying to read it in the sinatra application. This means, that rack must have some trouble dealing with the json string.

At the moment I am using a workaround inside my sinatra controller:

  private 

  def params
    JSON.parse(request.env['rack.input'].read)
  end

[nning]

I think, that's expected behaviour; does sinatra even deserialize te request body automatically?

[felixletkemann]

This

{"{ position: { lat: 3.14, lng: 49.2332, imei: 9FJ489FJW938FJW39F8"=>nil}

is probably not the serialized version of

{ position: { lat: 3.14, lng: 49.2332, imei: 9FJ489FJW938FJW39F8JW39F } }

So, I guess Sinatra does that but fails in this case, because it is not properly serialized.

[nning]

Does it fail to deserialize, because the value for imei is not valid JSON?

Could you provide an example as code for debugging, please?