# Log Filtration Issue with Filebeat and Logstash Configuration
**URL:** https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609
**Category:** Logstash
**Created:** [August 11, 2023, 5:05am UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609 "2023-08-11T05:05:00Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Priyaansh\_Dwivedi](https://discuss.elastic.co/u/Priyaansh_Dwivedi)
#### Post date: [August 11, 2023, 5:05am UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/1 "2023-08-11T05:05:01Z")
Hello everyone,
I'm using Filebeat to send three different logs to Logstash, where I'm applying parsing through filters. In the parsing process, I've taken into consideration the logs that are visible in the observation section of Kibana. While the logs are successfully being transferred to Elasticsearch, I'm encountering an issue where no filtration seems to occur while everything is working fine. Indices and data sync.
To elaborate, the format of the logs remains the same both before and after applying the filter section. I've attached my Logstash and Filebeat configuration files for reference.
Any insights or suggestions on why the filtration might not work as expected would be greatly appreciated.
**Logstash Conf**
```auto
input {
beats {
port => 5045
}
}
filter {
if "jicofo" in [tags] {
grok {
match => { "event.original" => "Jicofo %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level}: \[%{POSINT:process_id}\] \[room=%{DATA:meeting_name}@%{DATA}(?:\s*meeting_id=%{UUID:meeting_id})?\] %{GREEDYDATA:message}" }
}
}
}
output {
if [tags] and "jicofo" in [tags] {
elasticsearch {
hosts => ["http://elasticSearch_ip:9200"]
index => "jicofo-%{+YYYY}"
}
}
}
```
Thank you in advance for your help!
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [August 11, 2023, 6:08am UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/2 "2023-08-11T06:08:34Z")
Can you provide a log sample? (2-3 lines)
---
### Author:  [@Priyaansh\_Dwivedi](https://discuss.elastic.co/u/Priyaansh_Dwivedi)
#### Post date: [August 11, 2023, 6:35am UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/3 "2023-08-11T06:35:13Z")
```auto
@timestamp:
Aug 10, 2023 @ 15:34:47.730
@version:
1
agent.ephemeral_id:
048aeaca-ac5e-4582-99be-d524189902de
agent.id:
48835a24-a1f1-4948-9741-870975a42f7b
agent.name:
filebeat -ip-address
agent.type:
filebeat
agent.version:
8.9.0
app:
jitsi-meet-jicofo
ecs.version:
8.0.0
event.original:
Jicofo 2023-08-10 10:04:47.463 INFO: [5097] [room=roomname@conference.filebeat-ip meeting_id=9d6998ec-1cf5-49d1-bdd2-5d27b3ce5bbf]
```
```auto
Jicofo 2023-08-10 10:04:39.909 INFO: [5097] [room=roomname@conferencece.filebeat-ip meeting_id=9d6998ec-1cf5-49d1-bdd2-5d27b3ce5bbf] ColibriV2SessionManager.removeParticipant#123: Removing 3ea9b689
```
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [August 11, 2023, 6:40am UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/4 "2023-08-11T06:40:38Z")
Lines from the jicofo.log file, pls.
---
### Author:  [@Priyaansh\_Dwivedi](https://discuss.elastic.co/u/Priyaansh_Dwivedi)
#### Post date: [August 11, 2023, 6:57am UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/6 "2023-08-11T06:57:57Z")
> [@Rios](#):
>
> Can you provide a log sample? (2-3 lines)
above logs are of jicofo
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [August 11, 2023, 7:42am UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/7 "2023-08-11T07:42:08Z")
I don't see "jicofo" in tags. You should use [event][original] or the `message` field and make something like this:
```auto
if ([event][original] =~ /^Jicofo/ ) {
grok {
match => { "[event][original]" => "Jicofo %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level}: \[%{POSINT:process_id}\] \[room=%{DATA:meeting_name}@%{DATA}(?:\s*meeting_id=%{UUID:meeting_id})?\] %{GREEDYDATA:message}" }
overwrite => ["message"]
}
}
```
Also maybe is better to use `GREEDYDATA:logmessage` insted of overwriting `GREEDYDATA:message`. It's up to you.
---
### Author:  [@Priyaansh\_Dwivedi](https://discuss.elastic.co/u/Priyaansh_Dwivedi)
#### Post date: [August 11, 2023, 8:24am UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/8 "2023-08-11T08:24:25Z")
jicofo tag is used in filebeat.yml
I am sharing that too with you and this Grok pattern is working fine in Grok debugger.
```auto
filebeat.inputs:
- type: log
paths:
- /var/log/jitsi/jicofo.log
fields:
app: jitsi-meet-jicofo
fields_under_root: true
tags: ["jicofo"]
- type: log
paths:
- /var/log/jitsi/jvb.log
fields:
app: jitsi-meet-jvb
fields_under_root: true
tags: ["jvb"]
- type: log
paths:
- /var/log/nginx/access.log
fields:
app: nginx
fields_under_root: true
tags: ["nginx-access"]
- type: log
paths:
- /var/log/nginx/error.log
fields:
app: nginx
fields_under_root: true
tags: ["nginx-error"]
output.logstash:
hosts: ["logstash_ip.in:5045"]
when.contains:
tags: "jicofo"
index: "jicofo-%{+yyyy}"
output.logstash:
hosts: ["logstash_ip.in:5044"]
```
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [August 11, 2023, 8:28am UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/9 "2023-08-11T08:28:11Z")
Yes, you can use tags, it's up to you
---
### Author:  [@Priyaansh\_Dwivedi](https://discuss.elastic.co/u/Priyaansh_Dwivedi)
#### Post date: [August 11, 2023, 8:30am UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/10 "2023-08-11T08:30:10Z")
So where is the issue, why my logs are not getting filtered? no error nothing in the logs file too.
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [August 11, 2023, 9:06am UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/11 "2023-08-11T09:06:06Z")
> [@Priyaansh\_Dwivedi](#):
>
> `event.original`
Should be `[event][original]`. However should check my post above
---
### Author:  [@Priyaansh\_Dwivedi](https://discuss.elastic.co/u/Priyaansh_Dwivedi)
#### Post date: [August 11, 2023, 9:18am UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/12 "2023-08-11T09:18:07Z")
I did that still getting the same output on kibana discover.
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [August 11, 2023, 10:43am UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/13 "2023-08-11T10:43:54Z")
Can you show what you get? Kibana or rubydebug
---
### Author:  [@Priyaansh\_Dwivedi](https://discuss.elastic.co/u/Priyaansh_Dwivedi)
#### Post date: [August 11, 2023, 11:25am UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/14 "2023-08-11T11:25:34Z")
same log without any change
```auto
Jicofo 2023-08-11 09:34:36.453 INFO: [5795] [room=test8@conference.ipAddress meeting_id=44a3405d-270c-4469-9104-bf8b3b34af4d] JitsiMeetConferenceImpl.acceptSession#1302: Accepted initial sources from 1aac59c0: [audio=[2727804029], video=[3597009703, 868618095, 2972627672, 2019927256, 72401307, 4049052442], groups=[FID[3597009703, 868618095], SIM[3597009703, 2972627672, 2019927256], FID[2972627672, 72401307], FID[2019927256, 4049052442]]]
```
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [August 11, 2023, 11:49am UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/15 "2023-08-11T11:49:42Z")
> [@Priyaansh\_Dwivedi](#):
>
> I did that still getting the same output on kibana discover.
Can you share the **entire** document you are receiving? Go on discover, expand one of the documents, click on the JSON to see the Json structure and copy and share everything to make it clear what you are receiving.
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [August 11, 2023, 12:11pm UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/16 "2023-08-11T12:11:56Z")
As Leandro said, we need more details. Here is .conf which parse a line.
```auto
input {
generator {
message => "Jicofo 2023-08-11 09:34:36.453 INFO: [5795] [room=test8@conference.ipAddress meeting_id=44a3405d-270c-4469-9104-bf8b3b34af4d] JitsiMeetConferenceImpl.acceptSession#1302: Accepted initial sources from 1aac59c0: [audio=[2727804029], video=[3597009703, 868618095, 2972627672, 2019927256, 72401307, 4049052442], groups=[FID[3597009703, 868618095], SIM[3597009703, 2972627672, 2019927256], FID[2972627672, 72401307], FID[2019927256, 4049052442]]]"
count => 1
}
}
filter {
if ([event][original] =~ /^Jicofo/ ) {
grok {
match => { "[event][original]" => "Jicofo %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level}: \[%{POSINT:process_id}\] \[room=%{DATA:meeting_name}@%{DATA}(?:\s*meeting_id=%{UUID:meeting_id})?\] %{GREEDYDATA:message}" }
overwrite => ["message"]
}
}
}
output {
stdout { codec => rubydebug{} }
}
```
Result:
```auto
{
"log_level" => "INFO",
"event" => {
"original" => "Jicofo 2023-08-11 09:34:36.453 INFO: [5795] [room=test8@conference.ipAddress meeting_id=44a3405d-270c-4469-9104-bf8b3b34af4d] JitsiMeetConferenceImpl.acceptSession#1302: Accepted initial sources from 1aac59c0: [audio=[2727804029], video=[3597009703, 868618095, 2972627672, 2019927256, 72401307, 4049052442], groups=[FID[3597009703, 868618095], SIM[3597009703, 2972627672, 2019927256], FID[2972627672, 72401307], FID[2019927256, 4049052442]]]",
"sequence" => 0
},
"meeting_id" => "44a3405d-270c-4469-9104-bf8b3b34af4d",
"meeting_name" => "test8",
"@timestamp" => 2023-08-11T12:08:45.964922400Z,
"message" => "JitsiMeetConferenceImpl.acceptSession#1302: Accepted initial sources from 1aac59c0: [audio=[2727804029], video=[3597009703, 868618095, 2972627672, 2019927256, 72401307, 4049052442], groups=[FID[3597009703, 868618095], SIM[3597009703, 2972627672, 2019927256], FID[2972627672, 72401307], FID[2019927256, 4049052442]]]",
"process_id" => "5795",
"timestamp" => "2023-08-11 09:34:36.453"
}
```
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [August 11, 2023, 12:14pm UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/17 "2023-08-11T12:14:22Z")
> [@Priyaansh\_Dwivedi](#):
>
> ```auto
> output.logstash:
> hosts: ["logstash_ip.in:5045"]
> when.contains:
> tags: "jicofo"
> index: "jicofo-%{+yyyy}"
>
> output.logstash:
> hosts: ["logstash_ip.in:5044"]
>
> ```
Just saw this in your filebeat config, this is not supported, filebeat can have **only** one output, you need to remove one of those outputs.
I'm not sure which one Filebeat will use, if I'm not wrong it will use the last one. What is the pipeline listening on port 5044?
---
### Author:  [@rl0ne](https://discuss.elastic.co/u/rl0ne)
#### Post date: [August 15, 2023, 10:54am UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/18 "2023-08-15T10:54:28Z")
Exactly, one of mentioned outputs of filebeat is elasticsearch which bypass logstash and all it's filters. Seems like that is the issue.
---
### Author:  [@Priyaansh\_Dwivedi](https://discuss.elastic.co/u/Priyaansh_Dwivedi)
#### Post date: [August 16, 2023, 5:25am UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/19 "2023-08-16T05:25:18Z")
Port 5045 is dedicated to the Jicofo log with the "jicofo" tag, and port 5044 is assigned for other logs like Nginx and JVB. In the index management of the Kibana dashboard, I am observing complete outputs for both Jicofo and the others, indicating that both outputs are functioning properly. Additionally, I haven't come across any errors or warnings in the log files.
jicofo.conf file(1st conf file)
```auto
input {
beats {
port => 5045
}
}
filter {
if "jicofo" in [tags] {
grok {
match => { "message" => "Jicofo %{TIMESTAMP_ISO8601:my_timestamp} %{LOGLEVEL:my_log_level}: \[%{POSINT:my_process_id}\] \[room=%{DATA:meeting_name}@%{DATA}(?:\s*meeting_id=%{UUID:meeting_id})?\] %{GREEDYDATA:message}" }
}
}
}
output {
if [tags] and "jicofo" in [tags] {
elasticsearch {
hosts => ["http://elastic-search-ip:9200"]
index => "jicofo-%{+YYYY}"
}
}
}
```
Logstash.conf(second conf file)
```auto
input {
beats {
port => 5044
}
}
filter {
if "jvb" in [tags] {
mutate {
add_field => { "index_prefix" => "jvb-" }
}
} else if "nginx-access" in [tags] {
mutate {
add_field => { "index_prefix" => "nginx-access-" }
}
} else if "nginx-error" in [tags] {
mutate {
add_field => { "index_prefix" => "nginx-error-" }
}
}
}
output {
if [index_prefix] {
elasticsearch {
hosts => ["http://elasticSearch_ip:9200"]
index => "%{[index_prefix]}%{+YYYY}"
}
}
}
```
---
### Author:  [@Priyaansh\_Dwivedi](https://discuss.elastic.co/u/Priyaansh_Dwivedi)
#### Post date: [August 16, 2023, 6:02am UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/20 "2023-08-16T06:02:59Z")
Hey, It worked. I am really thankful to all of you for your time and respond.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 13, 2023, 6:03am UTC](https://discuss.elastic.co/t/log-filtration-issue-with-filebeat-and-logstash-configuration/340609/21 "2023-09-13T06:03:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.