Is filebeat able to index in elasticsearch with the date of the log line it reads, not the the date of the day?

[Orsius (Gautier Franchini)]

Hello, I'm wondering if filebeat is able to index in elasticsearch with the date of the log line it reads, not the the date of the day?

• -- What I want to put in place:
  *.log (from august 2016) > filebeat > elastisearch > kibana (the goal is to see the log with the date of the event not with the day where filebeat reads the log)

• -- key points:
  I want to keep my setup as simple as possible (skip logstash step);

• -- Here is my filebeat conf file:

filebeat:
prospectors:
-
paths:
- /applications/myapp/api*.log
input_type: log
exclude_lines: ["(.*)METHOD(.*)PATH(.*)health(.*)REQUEST(.*)"]
multiline:
pattern: "^[[:digit:]]{2}.[[:digit:]]{2}.[[:digit:]]{4}"
negate: true
match: after
registry_file: /var/lib/filebeat/registry
output:
elasticsearch:
hosts: ["http://elastic-01:9200", "http://elastic-02:9200" , "http://elastic-03:9200"]
index: "filebeat-query-processor"
template:
name: "filebeat"
path: "filebeat.template.json"
shipper:
logging:
to_syslog: false
to_files: true
files:
path: /var/log/filebeat
name: filebeat-query-processor.log
rotateeverybytes: 10485760 # = 10MB
level: debug
``

Thanks in advance for your answers.

kr,
Orsius.

[magnusbaeck (Magnus Bäck)]

Filebeat doesn't do this kind of parsing. You need Logstash for that.

[Orsius (Gautier Franchini)]

That's what I thought; Thanks for your quick answer.

[ruflin]

You might be interested in the elasticsearch ingest feature: https://www.elastic.co/guide/en/elasticsearch/reference/master/grok-processor.html