Assume i do have a log in the following format:
[ISO8601_timestamp] [log_message]
what i want to do is to have 3 timestamp after i store my log in Elasticsearch. right now im reading logs with filebeat and then send them to logstash and then store it in elasticsearch.
filebeat ---> logstash ---> elasticsearch
i want when i store the documents in elasticsearch have 3 timestampe, namely:
filebeat_time
logstash_time
log_time
filebeat_ time is when filebeat read the log. logstash_time is when logstash received the log and log_time is the ISO8601_timestamp which already is in the file.
apparently filebeat itself adds a @tiemestam filed. also logstash itself adds a @timestamp filed too. althught i don't know what happen when logstash recieve a message that already has a @timestamp field. what crossed my mine was if there was a way that i could change the name of @timestamp field that filebeat adds to filebeat_time using filebeat setting before message reaching logstash and then using logstash date plugin to store @timestamp automatically added by logstash in the field logstash_time that way maybe i could solve my problem.
the problem is i can't find a way using filebeat to create a filebeat_time filed based on @timestamp and then delete @timestamp. i tried using processors
- add_fields:
target: ''
fields:
filebeat_time: '%{@timestamp}'
and
- timestamp:
field: @timestamp
layouts:
- '2006-01-02T15:04:05.999Z'
target_field: filebeat_time
but they did not work. i could not find a way to create a filebeat_time based on @timestamp field that filebeat already adds to a document.