Lumberjack with Filebeat pipelines not working

Elastic Stack Logstash
1

I am testing a setup which is Filebeat --> Logstash(1) --> logstash(2) --> Elasticsearch with custom indices.

In logstash-2 i am using below config file (simplified). For logstash to logstash communication i am using Lumberjack. Now I want to configure Logstash-2 to use the pipelines (Reference) in below config file. Can anyone please suggest

input {
  beats {
    codec => json
    port => 5045
  }
}
filter {
  if "FORTINET" in [tags] {
    mutate { add_field => { "[@metadata][target_index]" => "firewall" } }
  }  else {
    mutate { add_field => { "[@metadata][target_index]" => "unknown" } }
  }
}
output {
     elasticsearch {
    hosts => ["xxx.xxx.xx.xx:9200" ]
    index => "%{[@metadata][target_index]}"
  }
}
}

Thank you
praveen

2

If you want logstash to send events through an ingestion pipeline you have to configure that.

3

Hello Badger, for default beat indices it is working. But here i am using the custom indieces as show in original post. How do i mix the pipline config you shared with custom indices configuration as show in original post.

4

The documentation I linked to has an example of using sprintf references for both the index and pipeline options.

5

According to the docuement you shared, we must add the """mutate { add_field => { "[@metadata][target_index]" => "firewall" } }""""" to pipeline output elasticsearch section. But there is no opetion MUTATE option in Logstash elasticsearch output section.

How do we add custom inidices to logstash pipeline configuration(shared doc by you)????
Could you please help me?

6

No, the mutate goes in the filter section.

7

I tried below configuration but no luck, Firewall logs not going through pipeline but found the non parsed firewall logs in Kibana discovery. Where is my mistake?

input {
  beats {
    codec => json
    port => 5045
  }
}
filter {
  if "FW" in [tags] {
    mutate { add_field => { "[@metadata][target_index]" => "firewall" } }
  }  else {
    mutate { add_field => { "[@metadata][target_index]" => "unknown" } }
  }
}
output {
  if [@metadata][pipeline] {
    if "FW" in [tags] {
    elasticsearch {
      hosts => ["http://x.x.x.x.x:9200"]
      #manage_template => false
      index => "firewall-%{+YYYY.MM.dd}"
      pipeline => "%{[@metadata][pipeline]}"
    }
  }
} else {
    elasticsearch {
    hosts => ["http://x.x.x.x.x:9200" ]
    index => "%{[@metadata][target_index]}"
  }
}
}
8

I tried by removing filter section also Logs are not parsing.

9

@PraveenKT

Try changing the output to stdout or file to view the data.

10

Hello Ric,

Please find below stdout data

Jul 15 15:20:06 usla-pap-elk04 logstash: {
Jul 15 15:20:06 usla-pap-elk04 logstash: "agent" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "hostname" => "inhy-pap-elk02.officeuaredinc.com",
Jul 15 15:20:06 usla-pap-elk04 logstash: "name" => "inhy-pap-elk02.officeuaredinc.com",
Jul 15 15:20:06 usla-pap-elk04 logstash: "type" => "filebeat",
Jul 15 15:20:06 usla-pap-elk04 logstash: "ephemeral_id" => "663630e0-6b5f-4e05-9ab8-b591698df553",
Jul 15 15:20:06 usla-pap-elk04 logstash: "id" => "5a980d47-4921-4ded-8755-ed16183583a9",
Jul 15 15:20:06 usla-pap-elk04 logstash: "version" => "7.13.2"
Jul 15 15:20:06 usla-pap-elk04 logstash: },
Jul 15 15:20:06 usla-pap-elk04 logstash: "fileset" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "name" => "firewall"
Jul 15 15:20:06 usla-pap-elk04 logstash: },
Jul 15 15:20:06 usla-pap-elk04 logstash: "service" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "type" => "fortinet"
Jul 15 15:20:06 usla-pap-elk04 logstash: },
Jul 15 15:20:06 usla-pap-elk04 logstash: "@version" => "1",
Jul 15 15:20:06 usla-pap-elk04 logstash: "input" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "type" => "udp"
Jul 15 15:20:06 usla-pap-elk04 logstash: },
Jul 15 15:20:06 usla-pap-elk04 logstash: "ecs" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "version" => "1.9.0"
Jul 15 15:20:06 usla-pap-elk04 logstash: },
Jul 15 15:20:06 usla-pap-elk04 logstash: "event" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "module" => "fortinet",
Jul 15 15:20:06 usla-pap-elk04 logstash: "dataset" => "fortinet.firewall"
Jul 15 15:20:06 usla-pap-elk04 logstash: },
Jul 15 15:20:06 usla-pap-elk04 logstash: "message" => "<141>date=2021-07-16 time=03:49:19 devname=\"inhy1cr-sec-fw01\" devid=\"FG3H1E5819902619\" eventtime=1626387560346422106 tz=\"+0530\" logid=\"0000000013\" type=\"traffic\" subtype=\"forward\" level=\"notice\" vd=\"root\" srcip=10.250.10.65 identifier=14284 srcintf=\"root-isp11\" srcintfrole=\"undefined\" dstip=10.10250.116 dstintf=\"trust\" dstintfrole=\"lan\" srcuuid=\"f15c1844-3615-51ea-067c-b16fcb267fb0\" dstuuid=\"f15c1844-3615-51ea-067c-b16fcb267fb0\" srccountry=\"Reserved\" dstcountry=\"Reserved\" sessionid=176467979 proto=1 action=\"accept\" policyid=3 policytype=\"policy\" poluuid=\"c730e8c2-3617-51ea-9926-48860b9f6f66\" policyname=\"INTER-any_to_any_RFC1918\" service=\"PING\" trandisp=\"noop\" duration=60 sentbyte=36 rcvdbyte=36 sentpkt=1 rcvdpkt=1 appcat=\"unscanned\" masterdstmac=\"00:a2:ee:6c:2b:4b\" dstmac=\"00:a2:ee:6c:2b:4b\" dstserver=0",
Jul 15 15:20:06 usla-pap-elk04 logstash: "tags" => [
Jul 15 15:20:06 usla-pap-elk04 logstash: [0] "FW",
Jul 15 15:20:06 usla-pap-elk04 logstash: [1] "beats_input_codec_plain_applied",
Jul 15 15:20:06 usla-pap-elk04 logstash: [2] "_geoip_lookup_failure",
Jul 15 15:20:06 usla-pap-elk04 logstash: [3] "beats_input_codec_json_applied"
Jul 15 15:20:06 usla-pap-elk04 logstash: ],
Jul 15 15:20:06 usla-pap-elk04 logstash: "@timestamp" => 2021-07-15T22:19:36.564Z,
Jul 15 15:20:06 usla-pap-elk04 logstash: "log" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "source" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "address" => "10.10.25.1:19834"
Jul 15 15:20:06 usla-pap-elk04 logstash: }
Jul 15 15:20:06 usla-pap-elk04 logstash: },
Jul 15 15:20:06 usla-pap-elk04 logstash: "host" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "containerized" => false,
Jul 15 15:20:06 usla-pap-elk04 logstash: "hostname" => "inhy-pap-elk02.officeuaredinc.com",
Jul 15 15:20:06 usla-pap-elk04 logstash: "architecture" => "x86_64",
Jul 15 15:20:06 usla-pap-elk04 logstash: "mac" => [
Jul 15 15:20:06 usla-pap-elk04 logstash: [0] "00:15:5d:10:0b:63"
Jul 15 15:20:06 usla-pap-elk04 logstash: ],
Jul 15 15:20:06 usla-pap-elk04 logstash: "os" => {
Jul 15 15:20:06 usla-pap-elk04 logstash: "platform" => "centos",
Jul 15 15:20:06 usla-pap-elk04 logstash: "kernel" => "4.18.0-240.1.1.el8_3.x86_64",
Jul 15 15:20:06 usla-pap-elk04 logstash: "name" => "CentOS Linux",
Jul 15 15:20:06 usla-pap-elk04 logstash: "type" => "linux",
Jul 15 15:20:06 usla-pap-elk04 logstash: "version" => "8",
Jul 15 15:20:06 usla-pap-elk04 logstash: "family" => "redhat"
Jul 15 15:20:06 usla-pap-elk04 logstash: },
Jul 15 15:20:06 usla-pap-elk04 logstash: "name" => "inhy-pap-elk02.officeuaredinc.com",
Jul 15 15:20:06 usla-pap-elk04 logstash: "id" => "f33e99c170c143aeb413c3cf396b0b55",
Jul 15 15:20:06 usla-pap-elk04 logstash: "ip" => [
Jul 15 15:20:06 usla-pap-elk04 logstash: [0] "10.10250.76",
Jul 15 15:20:06 usla-pap-elk04 logstash: [1] "fe80::37dc:7f2e:5eff:28ea"
Jul 15 15:20:06 usla-pap-elk04 logstash: ]
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.