# How can I merge two events in logstash?
**URL:** https://discuss.elastic.co/t/how-can-i-merge-two-events-in-logstash/159520
**Category:** Logstash
**Created:** [December 5, 2018, 11:47am UTC](https://discuss.elastic.co/t/how-can-i-merge-two-events-in-logstash/159520 "2018-12-05T11:47:06Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@vinodanumarla](https://discuss.elastic.co/u/vinodanumarla)
#### Post date: [December 5, 2018, 11:47am UTC](https://discuss.elastic.co/t/how-can-i-merge-two-events-in-logstash/159520/1 "2018-12-05T11:47:06Z")
I'm trying to parse a log file into elasticsearch through logstash.
I want to send the following log as single event(i.e. as a single document) into elasticsearch.
```auto
######################################################
ETL Wrapper Initializing - 09/27/2018 06:33:57
######################################################
------------------------------------------------------
Wrapper Information - 09/27/2018 06:33:57
------------------------------------------------------
------------------------------------------------------
Reading Component Log Port Files - 09/27/2018 06:34:53
------------------------------------------------------
-- > -- > Found 3 files and only merge non-zero byte files
------------------------------------------------------
Renaming Reject Files - 09/27/2018 06:34:56
------------------------------------------------------
######################################################
Sending Notifications - 09/27/2018 06:34:56
######################################################
------------------------------------------------------
Setting Exit Status - 09/27/2018 06:34:56
------------------------------------------------------
######################################################
ETL Wrapper Finalizing - 09/27/2018 06:34:56
######################################################
------------------------------------------------------
```
**Here is my logstash configuration:**
```auto
input {
file {
path => "D:/logs/file.log"
start_position => "beginning"
}
}
filter{
grok {
match => {"message" => "ETL Wrapper Initializing - %{DATESTAMP:JobStartTime}"}
match => {"message" => "ETL Wrapper Finalizing - %{DATESTAMP:JobEndTime}"}
}
if "_grokparsefailure" in [tags]{
drop{}
}
if [message] =~ /^$/ {
drop { }
}
mutate{
remove_field => ["@version","host","message"]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200/"
index => "success_index"
}
stdout { codec => rubydebug }
}
```
**Output of above configuration:**
```auto
{
"JobStartTime" => "09/27/2018 09:33:41",
"@timestamp" => 2018-12-05T10:55:44.698Z,
"path" => "D:/logs/file.log"
}
{
"JobEndTime" => "09/27/2018 09:34:16",
"@timestamp" => 2018-12-05T10:55:44.784Z,
"path" => "D:/logs/file.log"
}
```
**My expected output:**
```auto
{
"JobStartTime" => "09/27/2018 09:33:41",
"@timestamp" => 2018-12-05T10:55:44.698Z,
"path" => "D:/logs/file.log"
"JobEndTime" => "09/27/2018 09:34:16"
}
```
How can I merge "JobStartTime" and "JobEndTime" into single document?
Any help is appreciable..
Thanks in advance.!
-Vinod
---
### Author:  [@Michel99\_7](https://discuss.elastic.co/u/Michel99_7)
#### Post date: [December 6, 2018, 9:44am UTC](https://discuss.elastic.co/t/how-can-i-merge-two-events-in-logstash/159520/2 "2018-12-06T09:44:09Z")
Hello vinodanumarla,
Just have a look to the Aggregate Filter plugin which allows to do such things.
---
### Author:  [@vinodanumarla](https://discuss.elastic.co/u/vinodanumarla)
#### Post date: [December 6, 2018, 11:04am UTC](https://discuss.elastic.co/t/how-can-i-merge-two-events-in-logstash/159520/3 "2018-12-06T11:04:08Z")
Hello Michel99\_7
Thanks for the response.
I have tried with Aggregate Filter. Below is my filter,
aggregate {
task\_id =\> "%{path}"
code =\> "
map['path'] = event.get('path')
map['JobStartTime'] ||=
map['JobStartTime'] \<\< {'JobStartTime' =\> event.get('JobStartTime')}
map['JobEndTime'] ||=
map['JobEndTime'] \<\< {'JobEndTime' =\> event.get('JobEndTime')}
event.cancel()
"
push\_previous\_map\_as\_event =\> true
timeout =\> 3
}
Below is the output:
{
"JobEndTime" =\> [
[0] {
"JobEndTime" =\> nil
},
[1] {
"JobEndTime" =\> "09/27/2018 09:34:16"
}
],
"@timestamp" =\> 2018-12-06T09:32:33.412Z,
"JobStartTime" =\> [
[0] {
"JobStartTime" =\> "09/27/2018 09:33:41"
},
[1] {
"JobStartTime" =\> nil
}
],
"path" =\> "D:/logs/20180927\_093341\_PDCDWG1040\_lylty\_acct\_extract.log
.SUCCESS"
}
Able to combine two docs as above. But, can we do it without creating lists like above i.e without separate mappings.
Is it possible to get the output as below?
{
"JobStartTime" =\> "09/27/2018 09:33:41",
"@timestamp" =\> 2018-12-05T10:55:44.698Z,
"path" =\> "D:/logs/file.log"
"JobEndTime" =\> "09/27/2018 09:34:16"
}
Thanks!
---
### Author:  [@DyraSan](https://discuss.elastic.co/u/DyraSan)
#### Post date: [December 6, 2018, 11:43am UTC](https://discuss.elastic.co/t/how-can-i-merge-two-events-in-logstash/159520/4 "2018-12-06T11:43:48Z")
I'm trying to parse a log file into elasticsearch through logstash.
I want to send the following log as single event(i.e. as a single document) into elasticsearch.
Here is my log file looks like:
## ###################################################### ETL Wrapper Initializing - 09/27/2018 06:33:57 ######################################################
## Wrapper Information - 09/27/2018 06:33:57
* * *
## Reading Component Log Port Files - 09/27/2018 06:34:53
## -- \> -- \> Found 3 files and only merge non-zero byte files
## Renaming Reject Files - 09/27/2018 06:34:56
## ###################################################### Sending Notifications - 09/27/2018 06:34:56 ######################################################
## Setting Exit Status - 09/27/2018 06:34:56
## ###################################################### ETL Wrapper Finalizing - 09/27/2018 06:34:56 ######################################################
**Here is my logstash configuration:**
input {
file {
path =\> "D:/logs/file.log"
start\_position =\> "beginning"
}
}
filter{
grok {
match =\> {"message" =\> "ETL Wrapper Initializing - %{DATESTAMP:JobStartTime}"}
match =\> {"message" =\> "ETL Wrapper Finalizing - %{DATESTAMP:JobEndTime}"}
}
if "\_grokparsefailure" in [tags]{
drop{}
}
if [message] =~ /^$/ {
drop { }
}
mutate{
remove\_field =\> ["@version","host","message"]
}
}
output {
elasticsearch {
hosts =\> "[http://localhost:9200/](http://localhost:9200/)"
index =\> "success\_index"
}
stdout { codec =\> rubydebug }
}
**Output of above configuration:**
{
"JobStartTime" =\> "09/27/2018 09:33:41",
"@timestamp" =\> 2018-12-05T10:55:44.698Z,
"path" =\> "D:/logs/file.log"
}
{
"JobEndTime" =\> "09/27/2018 09:34:16",
"@timestamp" =\> 2018-12-05T10:55:44.784Z,
"path" =\> "D:/logs/file.log"
}
**My expected output:**
{
"JobStartTime" =\> "09/27/2018 09:33:41",
"@timestamp" =\> 2018-12-05T10:55:44.698Z,
"path" =\> "D:/logs/file.log"
"JobEndTime" =\> "09/27/2018 09:34:16"
}
How can I merge "JobStartTime" and "JobEndTime" into single document?
Any help is appreciable.. Thanks in advance!
Thanks,
[kissanime](https://www.kissanime.vip/)
---
### Author:  [@Michel99\_7](https://discuss.elastic.co/u/Michel99_7)
#### Post date: [December 6, 2018, 12:42pm UTC](https://discuss.elastic.co/t/how-can-i-merge-two-events-in-logstash/159520/5 "2018-12-06T12:42:33Z")
Hi Vino,
You can also use a multiline command in the input section:
input {
file {
path =\> "C:/test/\*.log"
#sincedb\_path =\> "/dev/null"
start\_position =\> "beginning"
discover\_interval =\> 5
close\_older =\> 60
mode =\> "tail"
file\_sort\_by =\> "path"
codec =\> multiline {
pattern =\> "^ETL Wrapper Initializing"
negate =\> true
what =\> previous
}
}
}
filter {
grok { match =\> {"message" =\> "^ETL Wrapper Initializing - %{DATESTAMP:JobStartTime}"} }
grok { match =\> {"message" =\> "ETL Wrapper Finalizing - %{DATESTAMP:JobEndTime}"} }
mutate { remove\_field =\> ["message"] }
}
output {
stdout {codec =\> rubydebug}
}
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 3, 2019, 12:52pm UTC](https://discuss.elastic.co/t/how-can-i-merge-two-events-in-logstash/159520/6 "2019-01-03T12:52:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.