Filebeat sending multiple lines as one event

Elastic Stack Beats
1

Hi there,

I´m sending apache access logs to elasticsearch using filebeat -> logstash.
The configuration is like that

filebeat.yml

filebeat.prospectors:

filebeat.config.inputs:
enabled: true
path: conf.d/*.yml
reload.enabled: true
reload.period: 10s

output.logstash:
hosts: ["10.248.114.155:5044"]
bulk_max_size: 5120

conf.d/apache-access-logs.yml

  • type: log
    paths:

    • /www/vofapl-int-/logs/**/liintra31.access_log

    encoding: utf-8

    multiline.pattern: '^\b\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\b\s'
    multiline.negate: false
    multiline.match: after

    fields:
    type: apache-access-logs

    fields_under_root: true

The problem (in kibana)

"message": "10.100.138.91 vofapl-int-405.bmwgroup.net qx0000 [24/Oct/2018:08:52:02 +0200] \"GET /vofapl_bc/api/v1/planorder_generator/jobs/progress?length=10&type=&status=&scenarioIds= HTTP/1.1\" 200 4016 \"https://vofapl-int-405.bmwgroup.net/vofapl_ui/?conversationid=rDbwV0n1E83xi5h7vqsP&deviceType=1&env=workplace&lang=en&locale=en-US&mwpOrigin=https%3A%2F%2Fworkplace-int4s.bmwgroup.net&role=PFX_PPA&strongAuth=1\" \"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36\"\n10.100.138.91 vofapl-int-405.bmwgroup.net qx0000 [24/Oct/2018:08:52:06 +0200] "GET /vofapl_bc/api/v1/planorder_generator/jobs/progress?length=10&type=&status=&scenarioIds= HTTP/1.1" 200 4016 "https://vofapl-int-405.bmwgroup.net/vofapl_ui/?conversationid=H6LgGyqd2p4NR10utsKQ&deviceType=1&env=workplace&lang=en&locale=en-US&mwpOrigin=https%3A%2F%2Fworkplace-int4s.bmwgroup.net&role=PFX_PPA&strongAuth=1\" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36""

As you can see in the message there are 2 lines (but I´ve already see 4 and more) in the message.

Can you give me some hints how I can debug such a problem ?

Many Thanks in advance and greetings,

Cyberseppi

2

As it looks like each new event will start with an IP address, this should as far as I can see behave similar to the timestamp example in the docs. In that example multiline.negate is set to true, so I suspect that may be why you are seeing the current behaviour.

3

strange thing is, that in most cases it works fine .... there are just a few such cases :frowning:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.