# Lumberjack with Filebeat pipelines not working
**URL:** https://discuss.elastic.co/t/lumberjack-with-filebeat-pipelines-not-working/278304
**Category:** Logstash
**Created:** [July 9, 2021, 7:02pm UTC](https://discuss.elastic.co/t/lumberjack-with-filebeat-pipelines-not-working/278304 "2021-07-09T19:02:14Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@PraveenKT](https://discuss.elastic.co/u/PraveenKT)
#### Post date: [July 9, 2021, 7:02pm UTC](https://discuss.elastic.co/t/lumberjack-with-filebeat-pipelines-not-working/278304/1 "2021-07-09T19:02:14Z")
I am testing a setup which is Filebeat --\> Logstash(1) --\> logstash(2) --\> Elasticsearch with custom indices.
In logstash-2 i am using below config file (simplified). For logstash to logstash communication i am using Lumberjack. Now I want to configure Logstash-2 to use the pipelines ([Reference](https://www.elastic.co/guide/en/logstash/current/use-ingest-pipelines.html)) in below config file. Can anyone please suggest
```auto
input {
beats {
codec => json
port => 5045
}
}
filter {
if "FORTINET" in [tags] {
mutate { add_field => { "[@metadata][target_index]" => "firewall" } }
} else {
mutate { add_field => { "[@metadata][target_index]" => "unknown" } }
}
}
output {
elasticsearch {
hosts => ["xxx.xxx.xx.xx:9200"]
index => "%{[@metadata][target_index]}"
}
}
}
```
Thank you
praveen
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 9, 2021, 7:08pm UTC](https://discuss.elastic.co/t/lumberjack-with-filebeat-pipelines-not-working/278304/2 "2021-07-09T19:08:18Z")
If you want logstash to send events through an ingestion pipeline you have to [configure](https://www.elastic.co/guide/en/logstash/current/use-ingest-pipelines.html#CO21-1) that.
---
### Author:  [@PraveenKT](https://discuss.elastic.co/u/PraveenKT)
#### Post date: [July 9, 2021, 7:11pm UTC](https://discuss.elastic.co/t/lumberjack-with-filebeat-pipelines-not-working/278304/3 "2021-07-09T19:11:22Z")
Hello Badger, for default beat indices it is working. But here i am using the custom indieces as show in original post. How do i mix the pipline config you shared with custom indices configuration as show in original post.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 9, 2021, 7:32pm UTC](https://discuss.elastic.co/t/lumberjack-with-filebeat-pipelines-not-working/278304/4 "2021-07-09T19:32:11Z")
The documentation I linked to has an example of using sprintf references for both the index and pipeline options.
---
### Author:  [@PraveenKT](https://discuss.elastic.co/u/PraveenKT)
#### Post date: [July 9, 2021, 7:51pm UTC](https://discuss.elastic.co/t/lumberjack-with-filebeat-pipelines-not-working/278304/5 "2021-07-09T19:51:07Z")
According to the docuement you shared, we must add the """mutate { add\_field =\> { "[@metadata][target\_index]" =\> "firewall" } }""""" to pipeline output elasticsearch section. But there is no opetion MUTATE option in Logstash elasticsearch output section.
**How do we add custom inidices to logstash pipeline configuration(shared doc by you)????**
Could you please help me?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 9, 2021, 7:57pm UTC](https://discuss.elastic.co/t/lumberjack-with-filebeat-pipelines-not-working/278304/6 "2021-07-09T19:57:51Z")
No, the mutate goes in the filter section.
---
### Author:  [@PraveenKT](https://discuss.elastic.co/u/PraveenKT)
#### Post date: [July 9, 2021, 8:21pm UTC](https://discuss.elastic.co/t/lumberjack-with-filebeat-pipelines-not-working/278304/7 "2021-07-09T20:21:39Z")
I tried below configuration but no luck, Firewall logs not going through pipeline **but found the non parsed firewall logs** in Kibana discovery. Where is my mistake?
```auto
input {
beats {
codec => json
port => 5045
}
}
filter {
if "FW" in [tags] {
mutate { add_field => { "[@metadata][target_index]" => "firewall" } }
} else {
mutate { add_field => { "[@metadata][target_index]" => "unknown" } }
}
}
output {
if [@metadata][pipeline] {
if "FW" in [tags] {
elasticsearch {
hosts => ["http://x.x.x.x.x:9200"]
#manage_template => false
index => "firewall-%{+YYYY.MM.dd}"
pipeline => "%{[@metadata][pipeline]}"
}
}
} else {
elasticsearch {
hosts => ["http://x.x.x.x.x:9200"]
index => "%{[@metadata][target_index]}"
}
}
}
```
---
### Author:  [@PraveenKT](https://discuss.elastic.co/u/PraveenKT)
#### Post date: [July 9, 2021, 8:45pm UTC](https://discuss.elastic.co/t/lumberjack-with-filebeat-pipelines-not-working/278304/8 "2021-07-09T20:45:30Z")
I tried by removing filter section also Logs are not parsing.
---
### Author:  [@ritchierich](https://discuss.elastic.co/u/ritchierich)
#### Post date: [July 12, 2021, 7:00am UTC](https://discuss.elastic.co/t/lumberjack-with-filebeat-pipelines-not-working/278304/9 "2021-07-12T07:00:45Z")
@PraveenKT
Try changing the output to stdout or file to view the data.
---
### Author:  [@PraveenKT](https://discuss.elastic.co/u/PraveenKT)
#### Post date: [July 15, 2021, 10:27pm UTC](https://discuss.elastic.co/t/lumberjack-with-filebeat-pipelines-not-working/278304/10 "2021-07-15T22:27:26Z")
Hello Ric,
Please find below stdout data
```auto
Jul 15 15:20:06 usla-pap-elk04 logstash: {
Jul 15 15:20:06 usla-pap-elk04 logstash: "agent" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "hostname" => "inhy-pap-elk02.officeuaredinc.com",
Jul 15 15:20:06 usla-pap-elk04 logstash: "name" => "inhy-pap-elk02.officeuaredinc.com",
Jul 15 15:20:06 usla-pap-elk04 logstash: "type" => "filebeat",
Jul 15 15:20:06 usla-pap-elk04 logstash: "ephemeral_id" => "663630e0-6b5f-4e05-9ab8-b591698df553",
Jul 15 15:20:06 usla-pap-elk04 logstash: "id" => "5a980d47-4921-4ded-8755-ed16183583a9",
Jul 15 15:20:06 usla-pap-elk04 logstash: "version" => "7.13.2"
Jul 15 15:20:06 usla-pap-elk04 logstash: },
Jul 15 15:20:06 usla-pap-elk04 logstash: "fileset" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "name" => "firewall"
Jul 15 15:20:06 usla-pap-elk04 logstash: },
Jul 15 15:20:06 usla-pap-elk04 logstash: "service" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "type" => "fortinet"
Jul 15 15:20:06 usla-pap-elk04 logstash: },
Jul 15 15:20:06 usla-pap-elk04 logstash: "@version" => "1",
Jul 15 15:20:06 usla-pap-elk04 logstash: "input" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "type" => "udp"
Jul 15 15:20:06 usla-pap-elk04 logstash: },
Jul 15 15:20:06 usla-pap-elk04 logstash: "ecs" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "version" => "1.9.0"
Jul 15 15:20:06 usla-pap-elk04 logstash: },
Jul 15 15:20:06 usla-pap-elk04 logstash: "event" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "module" => "fortinet",
Jul 15 15:20:06 usla-pap-elk04 logstash: "dataset" => "fortinet.firewall"
Jul 15 15:20:06 usla-pap-elk04 logstash: },
Jul 15 15:20:06 usla-pap-elk04 logstash: "message" => "<141>date=2021-07-16 time=03:49:19 devname=\"inhy1cr-sec-fw01\" devid=\"FG3H1E5819902619\" eventtime=1626387560346422106 tz=\"+0530\" logid=\"0000000013\" type=\"traffic\" subtype=\"forward\" level=\"notice\" vd=\"root\" srcip=10.250.10.65 identifier=14284 srcintf=\"root-isp11\" srcintfrole=\"undefined\" dstip=10.10250.116 dstintf=\"trust\" dstintfrole=\"lan\" srcuuid=\"f15c1844-3615-51ea-067c-b16fcb267fb0\" dstuuid=\"f15c1844-3615-51ea-067c-b16fcb267fb0\" srccountry=\"Reserved\" dstcountry=\"Reserved\" sessionid=176467979 proto=1 action=\"accept\" policyid=3 policytype=\"policy\" poluuid=\"c730e8c2-3617-51ea-9926-48860b9f6f66\" policyname=\"INTER-any_to_any_RFC1918\" service=\"PING\" trandisp=\"noop\" duration=60 sentbyte=36 rcvdbyte=36 sentpkt=1 rcvdpkt=1 appcat=\"unscanned\" masterdstmac=\"00:a2:ee:6c:2b:4b\" dstmac=\"00:a2:ee:6c:2b:4b\" dstserver=0",
Jul 15 15:20:06 usla-pap-elk04 logstash: "tags" => [
Jul 15 15:20:06 usla-pap-elk04 logstash: [0] "FW",
Jul 15 15:20:06 usla-pap-elk04 logstash: [1] "beats_input_codec_plain_applied",
Jul 15 15:20:06 usla-pap-elk04 logstash: [2] "_geoip_lookup_failure",
Jul 15 15:20:06 usla-pap-elk04 logstash: [3] "beats_input_codec_json_applied"
Jul 15 15:20:06 usla-pap-elk04 logstash: ],
Jul 15 15:20:06 usla-pap-elk04 logstash: "@timestamp" => 2021-07-15T22:19:36.564Z,
Jul 15 15:20:06 usla-pap-elk04 logstash: "log" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "source" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "address" => "10.10.25.1:19834"
Jul 15 15:20:06 usla-pap-elk04 logstash: }
Jul 15 15:20:06 usla-pap-elk04 logstash: },
Jul 15 15:20:06 usla-pap-elk04 logstash: "host" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "containerized" => false,
Jul 15 15:20:06 usla-pap-elk04 logstash: "hostname" => "inhy-pap-elk02.officeuaredinc.com",
Jul 15 15:20:06 usla-pap-elk04 logstash: "architecture" => "x86_64",
Jul 15 15:20:06 usla-pap-elk04 logstash: "mac" => [
Jul 15 15:20:06 usla-pap-elk04 logstash: [0] "00:15:5d:10:0b:63"
Jul 15 15:20:06 usla-pap-elk04 logstash: ],
Jul 15 15:20:06 usla-pap-elk04 logstash: "os" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "platform" => "centos",
Jul 15 15:20:06 usla-pap-elk04 logstash: "kernel" => "4.18.0-240.1.1.el8_3.x86_64",
Jul 15 15:20:06 usla-pap-elk04 logstash: "name" => "CentOS Linux",
Jul 15 15:20:06 usla-pap-elk04 logstash: "type" => "linux",
Jul 15 15:20:06 usla-pap-elk04 logstash: "version" => "8",
Jul 15 15:20:06 usla-pap-elk04 logstash: "family" => "redhat"
Jul 15 15:20:06 usla-pap-elk04 logstash: },
Jul 15 15:20:06 usla-pap-elk04 logstash: "name" => "inhy-pap-elk02.officeuaredinc.com",
Jul 15 15:20:06 usla-pap-elk04 logstash: "id" => "f33e99c170c143aeb413c3cf396b0b55",
Jul 15 15:20:06 usla-pap-elk04 logstash: "ip" => [
Jul 15 15:20:06 usla-pap-elk04 logstash: [0] "10.10250.76",
Jul 15 15:20:06 usla-pap-elk04 logstash: [1] "fe80::37dc:7f2e:5eff:28ea"
Jul 15 15:20:06 usla-pap-elk04 logstash: ]
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 12, 2021, 10:28pm UTC](https://discuss.elastic.co/t/lumberjack-with-filebeat-pipelines-not-working/278304/11 "2021-08-12T22:28:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.