Is filebeat able to index in elasticsearch with the date of the log line it reads, not the the date of the day?


(Gautier Franchini) #1

Hello, I'm wondering if filebeat is able to index in elasticsearch with the date of the log line it reads, not the the date of the day?

  • -- What I want to put in place:
    *.log (from august 2016) > filebeat > elastisearch > kibana (the goal is to see the log with the date of the event not with the day where filebeat reads the log)

  • -- key points:
    I want to keep my setup as simple as possible (skip logstash step);

  • -- Here is my filebeat conf file:

filebeat:
prospectors:
-
paths:
- /applications/myapp/api*.log
input_type: log
exclude_lines: ["(.*)METHOD(.*)PATH(.*)health(.*)REQUEST(.*)"]
multiline:
pattern: "^[[:digit:]]{2}.[[:digit:]]{2}.[[:digit:]]{4}"
negate: true
match: after
registry_file: /var/lib/filebeat/registry
output:
elasticsearch:
hosts: ["http://elastic-01:9200", "http://elastic-02:9200" , "http://elastic-03:9200"]
index: "filebeat-query-processor"
template:
name: "filebeat"
path: "filebeat.template.json"
shipper:
logging:
to_syslog: false
to_files: true
files:
path: /var/log/filebeat
name: filebeat-query-processor.log
rotateeverybytes: 10485760 # = 10MB
level: debug
``

Thanks in advance for your answers.

kr,
Orsius.


(Magnus B├Ąck) #2

Filebeat doesn't do this kind of parsing. You need Logstash for that.


(Gautier Franchini) #3

That's what I thought; Thanks for your quick answer.


(ruflin) #4

You might be interested in the elasticsearch ingest feature: https://www.elastic.co/guide/en/elasticsearch/reference/master/grok-processor.html


(system) #5

This topic was automatically closed after 21 days. New replies are no longer allowed.