Event Time discrepency

[vwbusguy]

The eventtime is being parsed way in the future. Here's an example:

50135-01-04 16:40:05.000000000 -0800 tag-for-app: {"sourceLine":0,"@timestamp":"2018-03-01T13:49:00.005-0800","level":"INFO","logger":"o.m.a.p.LoggerMessageProcessor","sourceMethod":"<unknown>","sourceClass":"<unknown>","loggerFull":"org.mule.api.processor.LoggerMessageProcessor","thread":"[mule-research-fluentd-scott].get:/fluentd:apiConfig.stage1.02","message":"Hey Fluentd!\n","sourceFile":"<unknown>","host":"[REDACTED]","tag":"tag-for-appt"}

I believe the problem is that we're send the event time in millseconds instead of "second since epoch" here: https://github.com/wywy/log4j-plugin-fluency/blob/master/src/com/wywy/log4j/appender/FluencyAppender.java#L210

This was my workaround in fluentd (1.0) to reconcile the two:

<filter tag-for-app>
  @type parser
  key_name @timestamp
  reserve_time false
  reserve_data true
  <parse>
    @type regexp
    expression /^(?<time>.*)$/
    time_format %Y-%m-%dT%H:%M:%S.%L%z
  </parse>
</filter>

[lobeck]

Thanks for the report, i guess this is caused by the new fluentd version. We just tested the plugin against 0.12 but apparently the changes in 0.14+ are causing issues now.

Unfortunately I can't help you with a fix as I currently don't have the resources to support this project.

[vwbusguy]

Ah, that's right. They changed the time format between the two versions. Ref. https://www.fluentd.org/blog/fluentd-v0.14.0-has-been-released
Perhaps the best way to move forward would be to introduce a conditional for the EventTime format and default to 0.12 behavior so we don't break existing deployments.

[vwbusguy]

I'm not sure if this is related to the other Java "fluency" project, but here's where the new nanosecond time-format is referenced - https://github.com/komamitsu/fluency#emit-event