# Multiline logs into one event using logstash?
**URL:** https://discuss.elastic.co/t/multiline-logs-into-one-event-using-logstash/320005
**Category:** Logstash
**Created:** [November 29, 2022, 7:55am UTC](https://discuss.elastic.co/t/multiline-logs-into-one-event-using-logstash/320005 "2022-11-29T07:55:25Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@kurdit](https://discuss.elastic.co/u/kurdit)
#### Post date: [November 29, 2022, 7:55am UTC](https://discuss.elastic.co/t/multiline-logs-into-one-event-using-logstash/320005/1 "2022-11-29T07:55:25Z")
hi all! I have these logs:
```auto
Nov 23 18:57:14 mx.host.cloud 18:57:14.756 2 SIPS-072111 SIPDATA-124634 REGISTER sip:111.222.333.444:65110 from udp[555.666.777.888]:65111
Nov 23 18:57:14 mx.host.cloud 18:57:14.756 2 SIPS-072111 SIGNAL-154480 created
Nov 23 18:57:16 mx.host.cloud 18:57:16.000 2 SIPS-072111 SIPDATA-124636 404-REGISTER(final) sent [0.0.0.0]:65100 -> udp[555.666.777.888]:65111
```
as you can see, for one event (action), several log lines are generated on the server, but each line has an event ID - 072111
how can i bind such multi-line logs into one event using logstash?
I didn't see the multiline codec in the elastic documentation, but after studying it in more detail, I realized that this is not exactly what I need. what options do i have?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [November 29, 2022, 5:40pm UTC](https://discuss.elastic.co/t/multiline-logs-into-one-event-using-logstash/320005/2 "2022-11-29T17:40:52Z")
You could try using an [aggregate](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html) filter.
---
### Author:  [@kurdit](https://discuss.elastic.co/u/kurdit)
#### Post date: [November 29, 2022, 7:40pm UTC](https://discuss.elastic.co/t/multiline-logs-into-one-event-using-logstash/320005/3 "2022-11-29T19:40:31Z")
yes, I stumbled upon it after a lot of searching for a solution, but I'm not able to apply it correctly for my situation. can you help?
my filter looks like this:
```auto
if [ident] {
aggregate {
task_id => "%{ident}"
code => "
map['main_hostname'] = event.get('main_hostname')
map['client_hostname'] = event.get('client_hostname')
"
}
}
```
my original log:
> Nov 23 18:57:14 mx.host.cloud 18:57:14.756 2 SIPS-072111 SIPDATA-124634 REGISTER sip:111.222.333.444:65110 from udp[555.666.777.888]:65111
> Nov 23 18:57:14 mx.host.cloud 18:57:14.756 2 SIPS-072111 SIGNAL-154480 created
> Nov 23 18:57:16 mx.host.cloud 18:57:16.000 2 SIPS-072111 SIPDATA-124636 404-REGISTER(final) sent [0.0.0.0]:65100 -\> udp[555.666.777.888]:65111
and the main grok:
`%{SYSLOGTIMESTAMP:date} %{IPORHOST:hostname} %{TIME:timestamp} %{INT} %{DATA:protocol}-%{INT:ident} %{GREEDYDATA} %{IPORHOST:main_hostname}:%{INT} %{GREEDYDATA:msg} \[%{IPORHOST:client_hostname}\]:%{INT}`
as a result, the events are still not merged...
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [November 29, 2022, 8:43pm UTC](https://discuss.elastic.co/t/multiline-logs-into-one-event-using-logstash/320005/4 "2022-11-29T20:43:15Z")
Firstly, I understand the need to redact your actual IP address, but please do not replace them with something that is not a valid address. 55.66.77.88 would be much better than 555.666.777.888.
Second, your grok pattern does not match any of your log lines because neither the main\_hostname pattern nor the client\_hostname are preceded by a space in the logs.
Third, adding something to the map does not modify the events.
One option would be to collect parts of all the event log messages in the event, event.cancel the individual events and create a new event using the [push\_map\_as\_event\_on\_timeout](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html#plugins-filters-aggregate-example3) option.
Another possibility is to copy fields from one event to subsequent events for the same task\_id. Something like
```
code => '
if !map["main_hostname"]
map["main_hostname"] = event.get("main_hostname")
else
event.set("main_hostname", map["main_hostname"])
end
...
'
```
I have previously posted [many examples](https://discuss.elastic.co/search?q=push_map_as_event_on_timeout%20%20%40Badger) of using push\_map\_as\_event\_on\_timeout.
---
### Author:  [@kurdit](https://discuss.elastic.co/u/kurdit)
#### Post date: [November 29, 2022, 9:06pm UTC](https://discuss.elastic.co/t/multiline-logs-into-one-event-using-logstash/320005/5 "2022-11-29T21:06:59Z")
thanks a lot for your answer 🙂
I corrected the grok and other data to remove confidential information and could make a mistake somewhere, this is true :^(
---
### Author:  [@kurdit](https://discuss.elastic.co/u/kurdit)
#### Post date: [November 29, 2022, 9:46pm UTC](https://discuss.elastic.co/t/multiline-logs-into-one-event-using-logstash/320005/6 "2022-11-29T21:46:51Z")
I tried to follow your advice, but did not get a positive result ☹
```auto
...
if [id] {
aggregate {
task_id => "%{id}"
code => '
if !map["client_hostname"]
map["client_hostname"] = event.get("client_hostname")
else
event.set("client_hostname", map["client_hostname"])
end
'
}
}
if [id] {
aggregate {
task_id => "%{id}"
code => '
if !map["main_hostname"]
map["main_hostname"] = event.get("main_hostname")
else
event.set("main_hostname", map["main_hostname"])
end
'
}
}
...
```
Am I using the tool incorrectly? now I will try your advice with push\_map\_as\_event\_on\_timeout
---
### Author:  [@kurdit](https://discuss.elastic.co/u/kurdit)
#### Post date: [November 30, 2022, 11:21am UTC](https://discuss.elastic.co/t/multiline-logs-into-one-event-using-logstash/320005/7 "2022-11-30T11:21:23Z")
I didn't get a positive result with push\_map\_as\_event\_on\_timeout either ☹
---
### Author:  [@kurdit](https://discuss.elastic.co/u/kurdit)
#### Post date: [December 1, 2022, 1:50pm UTC](https://discuss.elastic.co/t/multiline-logs-into-one-event-using-logstash/320005/8 "2022-12-01T13:50:24Z")
hi, can u help me plz?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 1, 2022, 5:22pm UTC](https://discuss.elastic.co/t/multiline-logs-into-one-event-using-logstash/320005/9 "2022-12-01T17:22:21Z")
I would split the grok into two so that the first one matches the common prefix and the second parses the hostnames that are present in the first log line. With a single grok two of the lines will not match so they will not have an [ident] field.
```
grok { match => { "message" => "%{SYSLOGTIMESTAMP:date} %{IPORHOST:hostname} %{TIME:timestamp} %{INT} %{DATA:protocol}-%{INT:ident} %{GREEDYDATA:restOfLine}" } }
grok { match => { "restOfLine" => "%{GREEDYDATA}%{IPORHOST:main_hostname}:%{INT} %{GREEDYDATA:msg}\[%{IPORHOST:client_hostname}\]:%{INT}" } }
aggregate {
task_id => "%{ident}"
code => '
if !map["client_hostname"]
map["client_hostname"] = event.get("client_hostname")
else
event.set("client_hostname", map["client_hostname"])
end
if !map["main_hostname"]
map["main_hostname"] = event.get("main_hostname")
else
event.set("main_hostname", map["main_hostname"])
end
'
}
```
---
### Author:  [@kurdit](https://discuss.elastic.co/u/kurdit)
#### Post date: [December 2, 2022, 10:14am UTC](https://discuss.elastic.co/t/multiline-logs-into-one-event-using-logstash/320005/10 "2022-12-02T10:14:08Z")
its work! thank u very much
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 30, 2022, 10:14am UTC](https://discuss.elastic.co/t/multiline-logs-into-one-event-using-logstash/320005/11 "2022-12-30T10:14:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.