Elastic Search is receiving data from beat, but no documents appear

Elastic Stack Elasticsearch
1

We have a beat with the panw module enabled collecting data from Palo Alto Firewalls. The data is being received by Elasticsearch but there are no documents. We have confirmed with a tcpdump the data is arriving from the firewall to the beat. I enabled DEBUG logging in Filebeat and confirmed events were published:

Feb 06 12:13:01 systemname filebeat[2425]: 2020-02-06T12:13:01.701-0500 DEBUG [processors] processing/processors.go:186 Publish event: {

Feb 06 12:13:02 systemname filebeat[2425]: 2020-02-06T12:13:02.423-0500 DEBUG [elasticsearch] elasticsearch/client.go:348 PublishEvents: 2 events have been published to elasticsearch in
Feb 06 12:13:02 systemname filebeat[2425]: 2020-02-06T12:13:02.423-0500 DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [20: 0, 2]
Feb 06 12:13:02 systemname filebeat[2425]: 2020-02-06T12:13:02.423-0500 DEBUG [publisher] memqueue/eventloop.go:535 broker ACK events: count=2, start-seq=44, end-seq=45
Feb 06 12:13:02 systemname filebeat[2425]: 2020-02-06T12:13:02.423-0500 DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:2
Feb 06 12:13:02 systemname filebeat[2425]: 2020-02-06T12:13:02.423-0500 DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
Feb 06 12:13:02 systemname filebeat[2425]: 2020-02-06T12:13:02.423-0500 DEBUG [acker] beater/acker.go:69 stateless ack {"count": 2}
Feb 06 12:13:08 systemname filebeat[2425]: 2020-02-06T12:13:08.499-0500 DEBUG [input] input/input.go:152 Run input

We ran a packet capture on the filebeat node and confirmed the Elasticsearch cluster was receiving the message and responding with an HTTP 200. From the Packet Capture's HTTP Stream:

POST /_bulk HTTP/1.1
Host: elasticSearchNodeHere:9200
User-Agent: Go-http-client/1.1
Content-Length: 2862
Accept: application/json
Content-Type: application/json; charset=UTF-8
Accept-Encoding: gzip

{"create":{"_index":"filebeat-7.5.2","pipeline":"filebeat-7.5.2-panw-panos-pipeline"}}
{"@timestamp":"2020-02-06T15:12:37.000Z","event":{"timezone":"-05:00","created":"2020/02/06 10:12:37","severity":6,"module":"panw","dataset":"panw.panos"},"fileset":{"name":"panos"},"input":{"type":"syslog"},"ecs":{"version":"1.1.0"},"agent":{"ephemeral_id":"8c77978b-b0b1-4df9-ac7f-ec835acaf2b6","hostname":"systemname","id":"d1a2322e-a921-4dff-8b15-7c527ed20097","version":"7.5.2","type":"filebeat"},"hostname":"dataHere","log":{"source":{"address":"someaddress:someport"}},"tags":["pan-os"],"host":{"name":"systemname","hostname":"systemname","architecture":"x86_64","os":{"platform":"rhel","version":"7.7 (Maipo)","family":"redhat","name":"Red Hat Enterprise Linux Server","kernel":"3.10.0-1062.12.1.el7.x86_64","codename":"Maipo"},"id":"78dc6a03fa634f6b883b97250d1a90fe","containerized":false},"_temp_":{"message_type":"SYSTEM","message_subtype":"general","generated_time":"2020/02/06 10:12:37"},"syslog":{"facility":1,"facility_label":"user-level","priority":14,"severity_label":"Informational"},"service":{"type":"panw"},"message":"1,2020/02/06 10:12:37,002201001018,SYSTEM,general,0,2020/02/06 10:12:37,,general,,0,0,general,informational,\"FqdnRefresh job enqueued. Enqueue time=2020/02/06 10:12:37. JobId=204605.  . Type: Full\",2950462,0x0,0,0,0,0,,ifw-pa5050-fo","observer":{"serial_number":"002201001018"}}
{"create":{"_index":"filebeat-7.5.2","pipeline":"filebeat-7.5.2-panw-panos-pipeline"}}
{"@timestamp":"2020-02-06T15:12:37.000Z","syslog":{"priority":14,"severity_label":"Informational","facility":1,"facility_label":"user-level"},"event":{"created":"2020/02/06 10:12:37","severity":6,"dataset":"panw.panos","module":"panw","timezone":"-05:00"},"fileset":{"name":"panos"},"tags":["pan-os"],"agent":{"id":"d1a2322e-a921-4dff-8b15-7c527ed20097","version":"7.5.2","type":"filebeat","ephemeral_id":"8c77978b-b0b1-4df9-ac7f-ec835acaf2b6","hostname":"systemname"},"ecs":{"version":"1.1.0"},"host":{"name":"systemname","hostname":"systemname","architecture":"x86_64","os":{"codename":"Maipo","platform":"rhel","version":"7.7 (Maipo)","family":"redhat","name":"Red Hat Enterprise Linux Server","kernel":"3.10.0-1062.12.1.el7.x86_64"},"id":"78dc6a03fa634f6b883b97250d1a90fe","containerized":false},"message":"1,2020/02/06 10:12:37,002201001018,SYSTEM,general,0,2020/02/06 10:12:37,,general,,0,0,general,informational,\"FqdnRefresh job started processing. Dequeue time=2020/02/06 10:12:37. Job Id=204605.   \",2950463,0x0,0,0,0,0,,ifw-pa5050-fo","hostname":"systemname","log":{"source":{"address":"someaddress:someport"}},"service":{"type":"panw"},"input":{"type":"syslog"},"observer":{"serial_number":"002201001018"},"_temp_":{"generated_time":"2020/02/06 10:12:37","message_type":"SYSTEM","message_subtype":"general"}}

Elastic Searches Response:
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-encoding: gzip
content-length: 188

I noticed it was using the "filebeat-7.5.2-panw-panos-pipeline" pipeline, and this seems to process without issue (I guess?). I checked to make sure the pipeline was intact:
"filebeat-7.5.2-panw-panos-pipeline" : {
"on_failure" : [
{
"set" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
},
{
"remove" : {
"ignore_missing" : true,
"field" : [
"temp"
]
}
}
],

In Elasticsearch, I can see there is an index:
# curl -X GET localhost:9200/_cat/indices
green open filebeat-7.5.2-2020.02.06-000001 foEkBgCNTe-AhdmXKWhC-w 1 1 0 0 566b 283b
green open .kibana_task_manager_1 at0InUYVTzaJiqRpXoATng 1 1 2 1 32.5kb 16.2kb
green open .apm-agent-configuration GM60UyQvSGuaVmYuibxCtQ 1 1 0 0 566b 283b
green open .kibana_1 bV0kNdxbRmuKTpGvkwBe5w 1 1 1059 47 1.2mb 598.3kb

But looking in the index there are no documents:
GET /filebeat-7.5.2-2020.02.06-000001/_search
{
"query": {
"match_all": {}
}
}

and this returned:
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : null,
"hits" :
}
}

I cant see it in kibana or doing a search within Elasticsearch. The document count remains at 0, though there is clearly data coming in. I checked to make sure filebeat, kibana, and Elasticsearch are all on the same version, they are. I shut everything down, removed all data inside Elasticsearch's data directory, started Elasticsearch, kibana, filebeat, confirmed file beat loaded the pipeline, then re-added the dashboards in Kibana but the same problem persists. I am at a loss. Can you provide assistance. Thanks.

I cant see it in kibana or doing a search within Elasticsearch. The document count remains at 0, though there is clearly data coming in. I checked to make sure filebeat, kibana, and Elasticsearch are all on the same version, they are. I shut everything down, removed all data inside Elasticsearch's data directory, started Elasticsearch, kibana, filebeat, confirmed file beat loaded the pipeline, then re-added the dashboards in Kibana but the same problem persists. I am at a loss. Can you provide assistance. Thanks.

2

Unfortunately a 200 OK response might still mean there were document-level failures; HTTP does not have a status code that means "some parts of this request succeeded and other parts failed". You must look at the body of the response to see the document-by-document status.

3

Thanks David, I didn't paste the entire response from the packet capture. On a side note, is there an easier way I can get this? The response in the capture is as follows:

HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-encoding: gzip
content-length: 183
{"took":0,"ingest_took":1,"errors":false,"items":[{"create":{"_index":"filebeat-7.5.2","_type":"_doc","_id":"auto-generated","_version":-4,"result":"noop","_shards":{"total":0,"successful":0,"failed":0},"status":200}}]}
4

It looks like the pipeline has decided that nothing needs doing for this document.

5

I don't understand why the pipeline would not run on this data. How can I see what the pipe line is getting, what it is processing to determine this, and at what point did it decide nothing needed to be done?
FWIW - We just installed filebeat on a separate system, enabled the system module, loaded the template, dashboards, and confirmed filebeat was publishing to elastic search (In filebeat's logs). It was, however the index is still empty. So this problem is reproduceable with multiple pipelines. Doesn't think mean the issue is related to the index itself and not the pipeline? Or something that happens before the pipelines? How can I get more data to see what is going on?

6

I think it's not that the pipeline isn't running, it's that it is running and deliberately dropping the data. That's a thing that pipelines can do: https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest-conditionals.html

I'm not familiar with this particular pipeline, however. Where has it come from? Can you share its definition here?

7

Ah I see, it's something that comes with filebeat? I think it's probably a good idea to ask this on the Beats forum, since it's more likely there will be people there who know what this pipeline's supposed to do and why it seems to be dropping this data.

8

Thanks David. This is the part that blows my mind. This is happening to everything I setup using filebeat. In the example I posted above I installed file beat on a system, enabled the panw module:
Filebeat - panw module
This one wasn't working, so I opened this thread. I then stood up a different test system to send data to elastic search, installed and configured filebeat, enabled the system module:
Filebeat- System Module
Walked through the configuration to load the templates, pipelines, dashboards, etc.... I had debug enabled on filebeat and watched the logging, there were no errors at this step. Everything should have been good to go. Yet, there are still no documents in the filebeat index. Two different sources (Filebeat's panw and system modules) from two different systems with two different pipelines are sending data but nothing is appearing. I'm wondering if there is a step before pipelines thats broken and I'm missing, but I don't know what that would be. Also, I shut the system down, deleted the contents of the data directory, brought ES back up, checked its logs, re-setup the beats, but the problem persists. I'll post both pipelines in question shortly, but I'm guessing the problem is between elastic search is getting the data and elastic search is running the pipeline. Could it be something at the index level itself I'm missing? I didn't change anything in regards to aliases etc (I think).

9

Thanks David. I'll open a thread on the Beats now. I'm still trying to figure out how I can see in elastic search's logs 1) What It got before doing anyting 2) What it did once it got it, and 3) What process it followed. Is it possible to increase logging such that elastic search grabs this and the json its producing? I had to do a packet capture to get the output I posted here. I'm wondering if there is a better way.

10

I've looked through the source and I do not see any obvious additional logging that can be enabled to trace a single document through an ingest pipeline, sorry.

Although I'm not familiar with how Beats configures Elasticsearch, I had a quick look at the source and I see a single drop processor here:

Is this the pipeline you're running? If so, it looks like it drops messages that do not have _temp_.message_type set to TRAFFIC or THREAT and in the request above this field is SYSTEM.

11

Hey Daniel,
This looks to be the same pipeline. For reference, here is the pipeline I pulled from ES. Please note, this issue is affecting multiple filebeat modules (the system and panw module), deployed on multiple systems, and each module is using its own ingest pipline. The only commonality I see between these is the index itself (or something I'm missing). Wouldn't this imply the problem lies outside of an individual pipeline?

12

Panw pipline (Part 1 of 2):

            {
              "filebeat-7.5.2-panw-panos-pipeline" : {
                "on_failure" : [
                  {
                    "set" : {
                      "field" : "error.message",
                      "value" : "{{ _ingest.on_failure_message }}"
                    }
                  },
                  {
                    "remove" : {
                      "ignore_missing" : true,
                      "field" : [
                        "_temp_"
                      ]
                    }
                  }
                ],
                "description" : "Pipeline for Palo Alto Networks PAN-OS Logs",
                "processors" : [
                  {
                    "rename" : {
                      "field" : "message",
                      "target_field" : "log.original"
                    }
                  },
                  {
                    "date" : {
                      "if" : "ctx.event.timezone == null",
                      "field" : "_temp_.generated_time",
                      "formats" : [
                        "yyyy/MM/dd HH:mm:ss"
                      ],
                      "on_failure" : [
                        {
                          "append" : {
                            "value" : "{{ _ingest.on_failure_message }}",
                            "field" : "error.message"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "date" : {
                      "if" : "ctx.event.timezone != null",
                      "field" : "_temp_.generated_time",
                      "formats" : [
                        "yyyy/MM/dd HH:mm:ss"
                      ],
                      "timezone" : "{{ event.timezone }}",
                      "on_failure" : [
                        {
                          "append" : {
                            "field" : "error.message",
                            "value" : "{{ _ingest.on_failure_message }}"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "date" : {
                      "field" : "event.created",
                      "target_field" : "event.created",
                      "formats" : [
                        "yyyy/MM/dd HH:mm:ss"
                      ],
                      "on_failure" : [
                        {
                          "append" : {
                            "value" : "{{ _ingest.on_failure_message }}",
                            "field" : "error.message"
                          }
                        }
                      ],
                      "if" : "ctx.event.timezone == null && ctx.event.created != null "
                    }
                  },
                  {
                    "date" : {
                      "on_failure" : [
                        {
                          "append" : {
                            "field" : "error.message",
                            "value" : "{{ _ingest.on_failure_message }}"
                          }
                        }
                      ],
                      "if" : "ctx.event.timezone != null && ctx.event.created != null ",
                      "field" : "event.created",
                      "target_field" : "event.created",
                      "formats" : [
                        "yyyy/MM/dd HH:mm:ss"
                      ],
                      "timezone" : "{{ event.timezone }}"
                    }
                  },
                  {
                    "date" : {
                      "target_field" : "event.start",
                      "formats" : [
                        "yyyy/MM/dd HH:mm:ss"
                      ],
                      "on_failure" : [
                        {
                          "append" : {
                            "field" : "error.message",
                            "value" : "{{ _ingest.on_failure_message }}"
                          }
                        }
                      ],
                      "if" : "ctx.event.timezone == null && ctx.event.start != null",
                      "field" : "event.start"
                    }
                  },
                  {
                    "date" : {
                      "if" : "ctx.event.timezone != null && ctx.event.start != null",
                      "field" : "event.start",
                      "target_field" : "event.start",
                      "timezone" : "{{ event.timezone }}",
                      "formats" : [
                        "yyyy/MM/dd HH:mm:ss"
                      ],
                      "on_failure" : [
                        {
                          "append" : {
                            "field" : "error.message",
                            "value" : "{{ _ingest.on_failure_message }}"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "convert" : {
                      "ignore_missing" : true,
                      "field" : "client.bytes",
                      "type" : "long"
                    }
                  },
                  {
                    "convert" : {
                      "type" : "long",
                      "ignore_missing" : true,
                      "field" : "client.packets"
                    }
                  },
                  {
                    "convert" : {
                      "type" : "long",
                      "ignore_missing" : true,
                      "field" : "client.port"
                    }
                  },
                  {
                    "convert" : {
                      "ignore_missing" : true,
                      "field" : "server.bytes",
                      "type" : "long"
                    }
                  },
                  {
                    "convert" : {
                      "type" : "long",
                      "ignore_missing" : true,
                      "field" : "server.packets"
                    }
                  },
                  {
                    "convert" : {
                      "field" : "server.port",
                      "type" : "long",
                      "ignore_missing" : true
                    }
                  },
                  {
                    "convert" : {
                      "type" : "long",
                      "ignore_missing" : true,
                      "field" : "source.bytes"
                    }
                  },
                  {
                    "convert" : {
                      "type" : "long",
                      "ignore_missing" : true,
                      "field" : "source.packets"
                    }
                  },
                  {
                    "convert" : {
                      "type" : "long",
                      "ignore_missing" : true,
                      "field" : "source.port"
                    }
                  },
                  {
                    "convert" : {
                      "type" : "long",
                      "ignore_missing" : true,
                      "field" : "destination.bytes"
                    }
                  },
                  {
                    "convert" : {
                      "field" : "destination.packets",
                      "type" : "long",
                      "ignore_missing" : true
                    }
                  },
                  {
                    "convert" : {
                      "type" : "long",
                      "ignore_missing" : true,
                      "field" : "destination.port"
                    }
                  },
                  {
                    "convert" : {
                      "type" : "long",
                      "ignore_missing" : true,
                      "field" : "network.bytes"
                    }
                  },
                  {
                    "convert" : {
                      "type" : "long",
                      "ignore_missing" : true,
                      "field" : "network.packets"
                    }
                  },
                  {
                    "convert" : {
                      "type" : "long",
                      "ignore_missing" : true,
                      "field" : "event.duration"
                    }
                  },
                  {
                    "convert" : {
                      "ignore_missing" : true,
                      "field" : "_temp_.labels",
                      "type" : "long"
                    }
                  },
                  {
                    "convert" : {
                      "type" : "long",
                      "ignore_missing" : true,
                      "field" : "panw.panos.sequence_number"
                    }
                  },
                  {
                    "convert" : {
                      "ignore_missing" : true,
                      "field" : "source.nat.port",
                      "type" : "long"
                    }
                  },
                  {
                    "convert" : {
                      "ignore_missing" : true,
                      "field" : "destination.nat.port",
                      "type" : "long"
                    }
                  },
                  {
                    "remove" : {
                      "if" : "ctx?.panw?.panos?.network?.pcap_id == \"0\"",
                      "field" : [
                        "panw.panos.network.pcap_id"
                      ]
                    }
                  },
                  {
                    "script" : {
                      "lang" : "painless",
                      "if" : "ctx?._temp_?.labels != null && ctx._temp_.labels != 0",
                      "params" : {
                        "container_page" : 32768,
                        "symmetric_return" : 2048,
                        "ssl_decrypted" : 16777216,
                        "temporary_match" : 8192,
                        "ipv6_session" : 33554432,
                        "x_forwarded_for" : 524288,
                        "http_proxy" : 262144,
                        "captive_portal" : 2097152,
                        "nat_translated" : 4194304,
                        "pcap_included" : 2147483648,
                        "url_filter_denied" : 8388608
                      },
                      "source" : """def labels = ctx?.labels; if (labels == null) {
               labels = new HashMap();
               ctx['labels'] = labels;
            } long value = ctx._temp_.labels; for (entry in params.entrySet()) {
               if ((value & entry.getValue()) != 0) {
                   labels[entry.getKey()] = true;
               }
            }
            """
                    }
                  },
                  {
                    "script" : {
                      "params" : {
                        "NANOS_IN_A_SECOND" : 1000000000
                      },
                      "source" : """long nanos = ctx['event']['duration'] * params.NANOS_IN_A_SECOND; ctx['event']['duration'] = nanos; def start = ctx.event?.start; if (start != null) {
               ctx.event['end'] = ZonedDateTime.parse(start).plusNanos(nanos);
            }
            """,
                      "lang" : "painless",
                      "if" : "ctx?.event?.duration != null"
                    }
                  },
                  {
                    "set" : {
                      "value" : "inbound",
                      "if" : "ctx?._temp_?.message_type == \"TRAFFIC\" && ctx?.panw?.panos?.source?.zone == \"untrust\" && ctx?.panw?.panos?.destination?.zone == \"trust\"",
                      "field" : "network.direction"
                    }
                  },
                  {
                    "set" : {
                      "value" : "outbound",
                      "if" : "ctx?._temp_?.message_type == \"TRAFFIC\" && ctx?.panw?.panos?.source?.zone == \"trust\" && ctx?.panw?.panos?.destination?.zone == \"untrust\"",
                      "field" : "network.direction"
                    }
                  },
                  {
                    "set" : {
                      "field" : "network.direction",
                      "value" : "internal",
                      "if" : "ctx?._temp_?.message_type == \"TRAFFIC\" && ctx?.panw?.panos?.source?.zone == \"trust\" && ctx?.panw?.panos?.destination?.zone == \"trust\""
                    }
                  },
13

Panw Pipline (part 2 of 2):

         {
                "set" : {
                  "value" : "external",
                  "if" : "ctx?._temp_?.message_type == \"TRAFFIC\" && ctx?.panw?.panos?.source?.zone == \"untrust\" && ctx?.panw?.panos?.destination?.zone == \"untrust\"",
                  "field" : "network.direction"
                }
              },
              {
                "set" : {
                  "value" : "unknown",
                  "if" : """ctx?._temp_?.message_type == "TRAFFIC" && ((ctx?.panw?.panos?.source?.zone != "trust" && ctx?.panw?.panos?.source?.zone != "untrust") || (ctx?.panw?.panos?.destination?.zone != "trust" && ctx?.panw?.panos?.destination?.zone != "untrust"))""",
                  "field" : "network.direction"
                }
              },
              {
                "set" : {
                  "if" : """ctx?._temp_?.message_type == "THREAT" && (ctx?._temp_?.direction == "0" || ctx?._temp_?.direction == "client-to-server")""",
                  "field" : "network.direction",
                  "value" : "inbound"
                }
              },
              {
                "set" : {
                  "field" : "network.direction",
                  "value" : "outbound",
                  "if" : """ctx?._temp_?.message_type == "THREAT" && (ctx?._temp_?.direction == "1" || ctx?._temp_?.direction == "server-to-client")"""
                }
              },
              {
                "set" : {
                  "field" : "network.direction",
                  "value" : "unknown",
                  "if" : """ctx?._temp_?.message_type == "THREAT" && ctx?.network?.direction == null"""
                }
              },
              {
                "set" : {
                  "field" : "network.type",
                  "value" : "ipv4",
                  "if" : """ctx?._temp_?.message_type == "TRAFFIC" && ctx?.labels?.ipv6_session == null"""
                }
              },
              {
                "set" : {
                  "field" : "network.type",
                  "value" : "ipv6",
                  "if" : """ctx?._temp_?.message_type == "TRAFFIC" && ctx?.labels?.ipv6_session != null"""
                }
              },
              {
                "set" : {
                  "field" : "event.category",
                  "value" : "network_traffic",
                  "if" : "ctx?._temp_?.message_type == \"TRAFFIC\""
                }
              },
              {
                "set" : {
                  "field" : "event.category",
                  "value" : "security_threat",
                  "if" : "ctx?._temp_?.message_type == \"THREAT\""
                }
              },
              {
                "drop" : {
                  "if" : "ctx?.event?.category == null"
                }
              },
              {
                "set" : {
                  "field" : "event.action",
                  "value" : "flow_started",
                  "if" : "ctx?._temp_?.message_subtype == \"start\""
                }
              },
              {
                "set" : {
                  "if" : "ctx?._temp_?.message_subtype == \"end\"",
                  "field" : "event.action",
                  "value" : "flow_terminated"
                }
              },
              {
                "set" : {
                  "field" : "event.action",
                  "value" : "flow_dropped",
                  "if" : "ctx?._temp_?.message_subtype == \"drop\""
                }
              },
              {
                "set" : {
                  "field" : "event.action",
                  "value" : "flow_denied",
                  "if" : "ctx?._temp_?.message_subtype == \"deny\""
                }
              },
              {
                "set" : {
                  "field" : "event.action",
                  "value" : "data_match",
                  "if" : "ctx?._temp_?.message_subtype == \"data\""
                }
              },
              {
                "set" : {
                  "value" : "file_match",
                  "if" : "ctx?._temp_?.message_subtype == \"file\"",
                  "field" : "event.action"
                }
              },
              {
                "set" : {
                  "field" : "event.action",
                  "value" : "flood_detected",
                  "if" : "ctx?._temp_?.message_subtype == \"flood\""
                }
              },
              {
                "set" : {
                  "field" : "event.action",
                  "value" : "packet_attack",
                  "if" : "ctx?._temp_?.message_subtype == \"packet\""
                }
              },
              {
                "set" : {
                  "field" : "event.action",
                  "value" : "scan_detected",
                  "if" : "ctx?._temp_?.message_subtype == \"scan\""
                }
              },
              {
                "set" : {
                  "if" : "ctx?._temp_?.message_subtype == \"spyware\"",
                  "field" : "event.action",
                  "value" : "spyware_detected"
                }
              },
              {
                "set" : {
                  "if" : "ctx?._temp_?.message_subtype == \"url\"",
                  "field" : "event.action",
                  "value" : "url_filtering"
                }
              },
              {
                "set" : {
                  "if" : "ctx?._temp_?.message_subtype == \"virus\"",
                  "field" : "event.action",
                  "value" : "virus_detected"
                }
              },
              {
                "set" : {
                  "field" : "event.action",
                  "value" : "exploit_detected",
                  "if" : "ctx?._temp_?.message_subtype == \"vulnerability\""
                }
              },
              {
                "set" : {
                  "value" : "wildfire_veredict",
                  "if" : "ctx?._temp_?.message_subtype == \"wildfire\"",
                  "field" : "event.action"
                }
              },
              {
                "set" : {
                  "field" : "event.action",
                  "value" : "wildfire_virus_detected",
                  "if" : "ctx?._temp_?.message_subtype == \"wildfire-virus\""
                }
              },
              {
                "set" : {
                  "value" : 1,
                  "field" : "event.severity",
                  "if" : "ctx.log.level == \"critical\""
                }
              },
              {
                "set" : {
                  "field" : "event.severity",
                  "if" : "ctx.log.level == \"high\"",
                  "value" : 2
                }
              },
              {
                "set" : {
                  "field" : "event.severity",
                  "if" : "ctx.log.level == \"medium\"",
                  "value" : 3
                }
              },
              {
                "set" : {
                  "if" : "ctx.log.level == \"low\"",
                  "value" : 4,
                  "field" : "event.severity"
                }
              },
              {
                "set" : {
                  "field" : "event.severity",
                  "if" : "ctx.log.level == \"informational\"",
                  "value" : 5
                }
              },
              {
                "set" : {
                  "field" : "event.outcome",
                  "value" : "drop-icmp",
                  "if" : "ctx?.event?.outcome == \"drop icmp\" || ctx?.event?.outcome == \"drop ICMP\""
                }
              },
              {
                "set" : {
                  "field" : "event.outcome",
                  "value" : "reset-both",
                  "if" : "ctx?.event?.outcome == \"reset both\""
                }
              },
              {
                "set" : {
                  "field" : "event.outcome",
                  "value" : "reset-client",
                  "if" : "ctx?.event?.outcome == \"reset client\""
                }
              },
              {
                "set" : {
                  "field" : "event.outcome",
                  "value" : "reset-server",
                  "if" : "ctx?.event?.outcome == \"reset server\""
                }
              },
              {
                "append" : {
                  "if" : "ctx?.source?.ip != null",
                  "field" : "related.ip",
                  "value" : [
                    "{{source.ip}}"
                  ]
                }
              },
              {
                "append" : {
                  "value" : [
                    "{{destination.ip}}"
                  ],
                  "if" : "ctx?.destination?.ip != null",
                  "field" : "related.ip"
                }
              },
              {
                "append" : {
                  "field" : "related.ip",
                  "value" : [
                    "{{source.nat.ip}}"
                  ],
                  "if" : "ctx?.source?.nat?.ip != null"
                }
              },
              {
                "append" : {
                  "value" : [
                    "{{destination.nat.ip}}"
                  ],
                  "if" : "ctx?.destination?.nat?.ip != null",
                  "field" : "related.ip"
                }
              },
              {
                "geoip" : {
                  "if" : "ctx?.source?.ip != null",
                  "field" : "source.ip",
                  "target_field" : "source.geo"
                }
              },
              {
                "geoip" : {
                  "target_field" : "destination.geo",
                  "if" : "ctx?.destination?.ip != null",
                  "field" : "destination.ip"
                }
              },
              {
                "geoip" : {
                  "ignore_missing" : true,
                  "database_file" : "GeoLite2-ASN.mmdb",
                  "field" : "source.ip",
                  "target_field" : "source.as",
                  "properties" : [
                    "asn",
                    "organization_name"
                  ]
                }
              },
              {
                "geoip" : {
                  "database_file" : "GeoLite2-ASN.mmdb",
                  "field" : "destination.ip",
                  "target_field" : "destination.as",
                  "properties" : [
                    "asn",
                    "organization_name"
                  ],
                  "ignore_missing" : true
                }
              },
              {
                "rename" : {
                  "field" : "source.as.asn",
                  "target_field" : "source.as.number",
                  "ignore_missing" : true
                }
              },
              {
                "rename" : {
                  "field" : "source.as.organization_name",
                  "target_field" : "source.as.organization.name",
                  "ignore_missing" : true
                }
              },
              {
                "rename" : {
                  "field" : "destination.as.asn",
                  "target_field" : "destination.as.number",
                  "ignore_missing" : true
                }
              },
              {
                "rename" : {
                  "field" : "destination.as.organization_name",
                  "target_field" : "destination.as.organization.name",
                  "ignore_missing" : true
                }
              },
              {
                "append" : {
                  "value" : [
                    "{{panw.panos.network.nat.community_id}}"
                  ],
                  "if" : "ctx?.panw?.panos?.network?.nat?.community_id != null && ctx.panw.panos.network.nat.community_id != ctx?.network?.community_id",
                  "field" : "network.community_id"
                }
              },
              {
                "grok" : {
                  "patterns" : [
                    "%{GREEDYDATA:panw.panos.threat.name}\\(\\s*%{GREEDYDATA:panw.panos.threat.id}\\s*\\)"
                  ],
                  "if" : "ctx?.panw?.panos?.threat?.name != null",
                  "field" : "panw.panos.threat.name",
                  "ignore_failure" : true
                }
              },
              {
                "set" : {
                  "field" : "panw.panos.threat.name",
                  "value" : "URL-filtering",
                  "if" : "ctx?.panw?.panos?.threat?.id == \"9999\""
                }
              },
              {
                "remove" : {
                  "ignore_missing" : true,
                  "field" : [
                    "_temp_"
                  ]
                }
              }
            ]
          }
        }
14

(I reformatted your posts to preserve whitespace with the </> button)

Yep, looks like this pipeline drops messages of type SYSTEM.

15

I understand. Thank you for your assistance. My takeaway is I need to learn more about pipelines and how to take apart what they are doing.

2 Likes
16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.