Input kafka: the decorate_events option overwrites @timestamp

Elastic Stack Logstash
1

Version: logstash-input-kafka (8.3.1)

  • Operating System: Red Hat 7.2
  • Logstash: 6.8.1
    I am using decorate_event in my kafka input section. It works nicely but unfortunatly, in combination with codec json, I lost @timestamp as time of indexation. My @timestamp is the time of recording in Kafka, same as [@metadata][kafka][timestamp] but I have lost the timestamp of indexing in logstash. So I have to rebuild my code in order not to use codec json in the kafka input section. Is it a normal behavior ?
    Thanks for your tips
2

If a json codec sees a field in the JSON called @timestamp it will overwrite @timestamp with it. You could use a ruby filter to capture the ingestion time

ruby { code => 'event.set("ingestedAt", LogStash::Timestamp.new(Time.now))' }
3

it is not the case and if I set decorate_event to false, @timestamp is correct

4

If you look at the code, decorate_events just controls whether fields are added to [@metadata][kafka].

5

already checked the code and you're right but if you run a logstash configuration, @timestamp is overwritten

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.