Multiple logs are concatenated to single event by Logstash

Elastic Stack Logstash
1 
Hi all,
I am using Filebeat to send Multiline logs to Logstash ,
Filebeat input is a file ( logs.txt), In case multiple events are published to (Logs.txt )   in a millisecond, they all are added as 1 log event to ES. Please find below the configuration:

**==================Filebeat Configuration=============================**
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:\store\Logs\Logs.txt
  multiline.type: pattern
  multiline.pattern: '^((\-{40}.*\s)Message: (?P<Message>.*\s*)ID: (?P<ID>.*\s*)Severity: (?P<Severity>.*\s*)Timestamp: (?P<Timestamp>.*\s*)ExtendedProperties: (?P<ExtendedProperties>[\s\S]*?)(\-{40}))'
  multiline.negate: true 
  multiline.match: before
  tags: ["multilinelogs"]
**===================================================================**

**==================Logstash Configuration=============================**
input {

    beats {
		port => 5000
	}
}
filter {
        grok {
			match => { "message" => "^((\-{40}.*\s)Message: (?<Message>.*\s*)ID: (?<ID>.*\s*)Severity: (?<Severity>.*\s*)Timestamp: (?<Timestamp>.*\s*)ExtendedProperties: (?<ExtendedProperties>[\s\S]*?)(\-{40}))"
			
		}
	}
	
}

output {
	elasticsearch {
		hosts => ["http://localhost:9200"]
		index => "multilinelogs"
	}	
}

**===================================================================**

**==================Logs.txt=============================**
----------------------------------------
Message:  successfully created
ID: 1
Severity: Information
Timestamp: 2020-09-13T01:31:18.344+05:32
Extended Properties: MsgCreateTime - 2020-09-13T01:18:09.262+05:30
ForceLog - True
----------------------------------------
----------------------------------------
Message:  successfully created
ID: 2
Severity: Information
Timestamp: 2020-09-13T01:31:18.344+05:32
Extended Properties: MsgCreateTime - 2020-09-13T01:18:09.262+05:30
ForceLog - True
----------------------------------------
----------------------------------------
Message:  successfully created
ID: 3
Severity: Information
Timestamp: 2020-09-13T01:31:18.344+05:32
Extended Properties: MsgCreateTime - 2020-09-13T01:18:09.262+05:30
ForceLog - True
----------------------------------------

**===================================================================**

Issue1:
In kibana -> it puts all logs together in one event. 

**FileBeat Image**
![FB|690x77](upload://swJZdm595qHmsd58MDRlL3pqPQJ.png)

**Logstash image**
![Logstash|690x142](upload://9KhXMrCx1KoFvwXLbLx9oSYaRsY.png) 

**Kibana image**
![kibana|690x424](upload://ebO27zcPCtundaLnMbohMEXDFLt.png) 


**Expected Output**
All the three logs should be read as 3 separate events .
How do we acheive that in Logstash.. where is the problem? in Logstash or Filebeat? Is there issue with read rate of logstash? sometime it aggregates 2 events or 5 events together.
2

This Issue happens with Multi-line logs specifically.

3

`
Resolved the issue ... I was supposed to use multiline.flush_pattern: '^(-{40})$'
This line arks the endpoint of my log .
So now Filebeat is able to flush each log as separate event. :sweat_smile:

`

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.