# Logs getting Merged/clubbed with each other in some cases
**URL:** https://discuss.elastic.co/t/logs-getting-merged-clubbed-with-each-other-in-some-cases/329588
**Category:** Logstash
**Created:** [April 7, 2023, 5:17pm UTC](https://discuss.elastic.co/t/logs-getting-merged-clubbed-with-each-other-in-some-cases/329588 "2023-04-07T17:17:37Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@shadu88](https://discuss.elastic.co/u/shadu88)
#### Post date: [April 7, 2023, 5:17pm UTC](https://discuss.elastic.co/t/logs-getting-merged-clubbed-with-each-other-in-some-cases/329588/1 "2023-04-07T17:17:37Z")
Hello Dear ELKs,
I'm using logstash7.10 for forward the logs to Qradar and Azure sentinel. Have noticed some irregularities with some log source type.
Log flow : heterogenous logs -\> file --\> logstash( file input) --\> Qradar on tcp 514 + Azure sentinel using sentinel output plugin
Deviations :
1. Fortigate logs merged with checkpoint logs
2. timestamp is missing in some logs
3. some are truncated logs
Ask : is there any way to fine tune logstash config to TC above issues
Request for help!! Thank you in advance.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [April 7, 2023, 5:50pm UTC](https://discuss.elastic.co/t/logs-getting-merged-clubbed-with-each-other-in-some-cases/329588/2 "2023-04-07T17:50:43Z")
> [@shadu88](#):
>
> Log flow : heterogenous logs -\> file --\> logstash( file input) --\> Qradar on tcp 514 + Azure sentinel using sentinel output plugin
Are you using the `pipelines.yml` to configure [multiple pipelines](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html)? What does your `pipelines.yml` looks like.
If you didn't configure logstash to use multiple pipelines with `pipelines.yml` then you have just only one pipeline and unless you have conditionals in this pipeline, the data from all inputs will pass through all filters and go to all outputs.
> [@shadu88](#):
>
> - Fortigate logs merged with checkpoint logs
> - timestamp is missing in some logs
> - some are truncated logs
Also, share some evidence about this issues, share logs and the output you are getting and the expected output.
---
### Author:  [@shadu88](https://discuss.elastic.co/u/shadu88)
#### Post date: [April 10, 2023, 5:53am UTC](https://discuss.elastic.co/t/logs-getting-merged-clubbed-with-each-other-in-some-cases/329588/3 "2023-04-10T05:53:31Z")
Thank @leandrojmp for your quick response.
Yes, i have pipelines.yml:
Problematic logs:
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [April 10, 2023, 11:55am UTC](https://discuss.elastic.co/t/logs-getting-merged-clubbed-with-each-other-in-some-cases/329588/4 "2023-04-10T11:55:14Z")
You need to share your pipelines configurations, it is impossible to know what your Logstash is doing without it.
Also, avoid sharing plain text as screenshots as it is not possible to copy to try to replicate your pipelines, share them as text using the preformatted text option, the `` button.
---
### Author:  [@shadu88](https://discuss.elastic.co/u/shadu88)
#### Post date: [April 10, 2023, 12:16pm UTC](https://discuss.elastic.co/t/logs-getting-merged-clubbed-with-each-other-in-some-cases/329588/5 "2023-04-10T12:16:39Z")
```auto
# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
# https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html
#- pipeline.id: main
# path.config: "/etc/logstash/conf.d/*.conf"
- pipeline.id: main
path.config: "/etc/logstash/conf.d/main.conf"
- pipeline.id: sentinel
path.config: "/etc/logstash/conf.d/sentinel.conf"
#- pipeline.id: sentinel-win-fortigate
# path.config: "/etc/logstash/conf.d/sentinel-win-fortigate.conf"
```
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [April 10, 2023, 12:22pm UTC](https://discuss.elastic.co/t/logs-getting-merged-clubbed-with-each-other-in-some-cases/329588/6 "2023-04-10T12:22:57Z")
You need to share your configuration, the content of the files `main.conf` and `sentinel.conf`.
---
### Author:  [@shadu88](https://discuss.elastic.co/u/shadu88)
#### Post date: [April 10, 2023, 12:35pm UTC](https://discuss.elastic.co/t/logs-getting-merged-clubbed-with-each-other-in-some-cases/329588/7 "2023-04-10T12:35:48Z")
cat /etc/logstash/conf.d/main.conf
input {
file {
path =\> "/logpath/mainlog.log"
start\_position =\> "beginning"
sincedb\_path =\> "/etc/logstash/sincedb/null"
}
}
output {
tcp { host =\> ["10.2.5.1"]
port =\> 9600
codec =\> line { format =\> "%{message}" }
}
pipeline { send\_to =\> "sentinel" }
}
* * *
cat /etc/logstash/conf.d/sentinel.conf
input {
pipeline { address =\> "sentinel" }
}
filter {
if [message] =~ "10.1.2.3" and [message] =~ "[localhost] sudo: pam\_unix" { drop { } }
if [message] =~ "10.1.2.3" and [message] =~ "kernel: " { drop { } }
if [message] =~ "diskUuid" { drop { } } #filter from Logstash 3/4
}
output {
if [message] =~ "zpa-lss" {
microsoft-sentinel-logstash-output-plugin {
client\_app\_Id =\> "abcb"
client\_app\_secret =\> "\*\*\*\*\*\*\*\*\*
tenant\_id =\> "\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*"
data\_collection\_endpoint =\> "[https://abc](https://abc)"
dcr\_immutable\_id =\> "\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*"
dcr\_stream\_name =\> "Custom-ZPA\_CL"
#create\_sample\_file=\> true
#sample\_file\_path =\> "/tmp/logstash\_samplefile"
}
}
else if [message] =~ "zscaler-nss" {
microsoft-sentinel-logstash-output-plugin {
client\_app\_Id =\> "abc"
client\_app\_secret =\> "\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*"
tenant\_id =\> "\*\*\*_**"
data\_collection\_endpoint =\> "[https://abc](https://abc)"
dcr\_immutable\_id =\> "**_"
dcr\_stream\_name =\> "Custom-ZScalar\_NSSStream"
#create\_sample\_file=\> true
#sample\_file\_path =\> "/tmp/logstash\_samplefile"
}
}
else if [message] =~ "Check Point" {
microsoft-sentinel-logstash-output-plugin {
client\_app\_Id =\> "abc"
client\_app\_secret =\> "\*\*\*\*\*\*\*\*\*\*\*\*\*\*"
tenant\_id =\> "\***8"
data\_collection\_endpoint =\> "[https://abc](https://abc)"
dcr\_immutable\_id =\> "**"
dcr\_stream\_name =\> "Custom-CheckPointStream"
#create\_sample\_file=\> true
#sample\_file\_path =\> "/tmp/logstash\_samplefile"
}
}
else if ([message] =~ "devname=" and [message] =~ "devid=" and [message] =~ "date=" and [message] =~ "time=") {
microsoft-sentinel-logstash-output-plugin {
client\_app\_Id =\> "bbc"
client\_app\_secret =\> "\*\*\*\*\*\*\*\*\*\*\*\*\*\***"
tenant\_id =\> "_8"
data\_collection\_endpoint =\> "[https://abc](https://abc)"
dcr\_immutable\_id =\> "_**"
dcr\_stream\_name =\> "Custom-FortigateStream"
#create\_sample\_file=\> true
#sample\_file\_path =\> "/tmp/logstaplefile"
}
}
else {
microsoft-sentinel-logstash-output-plugin {
client\_app\_Id =\> "**"
client\_app\_secret =\> "**\*\*\*"
tenant\_id =\> "\*\*\*\*_**88"
data\_collection\_endpoint =\> "[https://abc](https://abc)"
dcr\_immutable\_id =\> "**_"
dcr\_stream\_name =\> "Custom-SyslogStream"
#create\_sample\_file=\> true
#sample\_file\_path =\> "/tmp/logstash\_samplefile"
}
}
}
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [April 10, 2023, 12:44pm UTC](https://discuss.elastic.co/t/logs-getting-merged-clubbed-with-each-other-in-some-cases/329588/8 "2023-04-10T12:44:01Z")
> [@shadu88](#):
>
> ```auto
> input {
> file {
> path => "/data/logcollector/prodlogs/mainlog.log"
> start_position => "beginning"
> sincedb_path => "/etc/logstash/sincedb/null"
>
> }
> }
>
> ```
Well, I'm not sure what is your issue, you have just only **one** source of logs, your Check point and Fortigate logs are coming from the same file.
If you have logs merged in logstash it means that they are merged in the source file, if something is missing from the logs in logstash, it means that they are missing also in the source file, you need to check on how you are creating this source file.
---
### Author:  [@shadu88](https://discuss.elastic.co/u/shadu88)
#### Post date: [April 10, 2023, 1:59pm UTC](https://discuss.elastic.co/t/logs-getting-merged-clubbed-with-each-other-in-some-cases/329588/9 "2023-04-10T13:59:45Z")
Hello @leandrojmp,
Thanks for checking. I have validated the source file, logs look ok to me. I suspect considering the huge log volume logs are getting merged when bandwidth is high. Do you suggest any performance tuning or any parameter for controlling the traffic.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [April 10, 2023, 2:51pm UTC](https://discuss.elastic.co/t/logs-getting-merged-clubbed-with-each-other-in-some-cases/329588/10 "2023-04-10T14:51:20Z")
> [@shadu88](#):
>
> Do you suggest any performance tuning or any parameter for controlling the traffic.
It really depends on how this log is generated, but this is not an issue with Logstash, it is an issue on how you are creating this log, Logstash will consume the log as it is.
---
### Author:  [@shadu88](https://discuss.elastic.co/u/shadu88)
#### Post date: [April 11, 2023, 7:52am UTC](https://discuss.elastic.co/t/logs-getting-merged-clubbed-with-each-other-in-some-cases/329588/11 "2023-04-11T07:52:05Z")
I shall check the input file and get back.
Thanks Much!!!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 9, 2023, 7:53am UTC](https://discuss.elastic.co/t/logs-getting-merged-clubbed-with-each-other-in-some-cases/329588/12 "2023-05-09T07:53:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.