Add mode to generate event per line of command output

[Jahor]

Hi,

I needed a mode in which external app will produce several lines of output that can be indexed in Elasticsearch separately.

In my case it will be a JSON objects per line which in conjunction with decode_json_fields processor will spare me time implementing custom beat for getting metrics from RabbitMQ API.

It produces events like this:

{
  "@timestamp": "2017-03-05T05:23:04.003Z",
  "beat": {
    "hostname": "host",
    "name": "name",
    "version": "3.1.1"
  },
  "line": {
    "command": "bash",
    "exitCode": 0,
    "line": "{\"test1\": {\"message\": \"hello\"}}",
    "line_number": 0,
    "source": "stdout"
  },
  "type": "execbeat"
}

{
  "@timestamp": "2017-03-05T05:23:04.003Z",
  "beat": {
    "hostname": "host",
    "name": "name",
    "version": "3.1.1"
  },
  "line": {
    "command": "bash",
    "exitCode": 0,
    "line": "{\"test2\": {\"message\": \"world\"}}",
    "line_number": 1,
    "source": "stdout"
  },
  "type": "execbeat"
}

from

{"test1": {"message": "hello"}}
{"test2": {"message": "world"}}

[jautz]

@christiangalsterer, what do you think about this feature in general? That is, some possibility to produce multiple documents from one execbeat command invocation? I am just curious about the "roadmap".

[christiangalsterer]

Hi @jautz,

sorry for the late feedback. In general I think this is a very nice feature. I think it would address #22 isn't it?

I will look into it the next days.

[christiangalsterer]

Here some ideas where it would be great if you can share your thoughts.

Instead of creating a new line field what about returning the existing exec field but adding the line_number as an additional optional field?

You also return the output in the line field and denote the source with a new source field. In the line _node=false mode the result is returned in two different fields. As I would like to keep the output consistent was there a special reason for this or was this just personell preference?

[jautz]

Thanks for your reply. Yes, it seems to be a solution for #22.
To be honest, I do not use beats anymore for the specific problem I wanted to address with this multi-event feature. Therefore anyone else who needs this should speak up and let Christian know about the demand.

[StyXman]

Any news about this? I do think it's useful, for instance for easily obtaining metrics from things for which there are no support yet, like SMART for disks, custom systems, and more.