How to write more than one multiline patterns for two different type of Logs

faiz · November 12, 2019, 3:40pm

HI,
I need to ship logs of my tomcat alongs with my applications logs.
My application logs and tomcat logs are of different time stamp, There fore I need to write more that one multiline patterns in the filebeat prospector.

How could I ship logs from the same filebeat to logstash.

Any suggestion would be helpful.

Thanks,

Slavik_Fursov · November 12, 2019, 11:20pm

I have never tried,
But you can have multiple patterns on the ingestion pipelines.
For example, existing Apache pipeline does it:

github.com

elastic/beats/blob/master/filebeat/module/apache/access/ingest/default.json#L7-L11


"patterns": [
    "%{IPORHOST:destination.domain} %{IPORHOST:source.ip} - %{DATA:user.name} \\[%{HTTPDATE:apache.access.time}\\] \"(?:%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}|-)?\" %{NUMBER:http.response.status_code:long} (?:%{NUMBER:http.response.body.bytes:long}|-)( \"%{DATA:http.request.referrer}\")?( \"%{DATA:user_agent.original}\")?",
    "%{IPORHOST:source.address} - %{DATA:user.name} \\[%{HTTPDATE:apache.access.time}\\] \"(?:%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}|-)?\" %{NUMBER:http.response.status_code:long} (?:%{NUMBER:http.response.body.bytes:long}|-)( \"%{DATA:http.request.referrer}\")?( \"%{DATA:user_agent.original}\")?",
    "%{IPORHOST:source.address} - %{DATA:user.name} \\[%{HTTPDATE:apache.access.time}\\] \"-\" %{NUMBER:http.response.status_code:long} -",
    "\\[%{HTTPDATE:apache.access.time}\\] %{IPORHOST:source.address} %{DATA:apache.access.ssl.protocol} %{DATA:apache.access.ssl.cipher} \"%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}\" (-|%{NUMBER:http.response.body.bytes:long})"

But multiline setting have to be defined on the filebeat side. For that, I guess you will need to find settings, which works for both App and Tomcat

faiz · November 18, 2019, 5:15pm

Hi thanks for the response, but I am comfortable on the ingestion side, i am facing blocker on the filebeat side, if i could add mulitple multiline pattern. ;(

Any example of the same would be much appreciable.

system · December 16, 2019, 5:15pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
I want to run two multiline pattern using filebeat Beats filebeat	1	310	June 10, 2020
Parsing multiline logs of various formats Beats filebeat	2	1026	February 15, 2017
Multi line pattern for different services to one logstash Beats filebeat	8	737	November 30, 2018
Filebeat multiline Pattern on Apache Logs does not work as expected (Filebeat -> Logstash -> ES) Beats filebeat	3	373	May 5, 2022
Help on multiline Beats filebeat	12	2712	November 15, 2017

How to write more than one multiline patterns for two different type of Logs

Related topics