# Elastic Search is receiving data from beat, but no documents appear
**URL:** https://discuss.elastic.co/t/elastic-search-is-receiving-data-from-beat-but-no-documents-appear/218230
**Category:** Elasticsearch
**Created:** [February 6, 2020, 5:53pm UTC](https://discuss.elastic.co/t/elastic-search-is-receiving-data-from-beat-but-no-documents-appear/218230 "2020-02-06T17:53:57Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@dsdameron](https://discuss.elastic.co/u/dsdameron)
#### Post date: [February 6, 2020, 5:53pm UTC](https://discuss.elastic.co/t/elastic-search-is-receiving-data-from-beat-but-no-documents-appear/218230/1 "2020-02-06T17:53:57Z")
We have a beat with the panw module enabled collecting data from Palo Alto Firewalls. The data is being received by Elasticsearch but there are no documents. We have confirmed with a tcpdump the data is arriving from the firewall to the beat. I enabled DEBUG logging in Filebeat and confirmed events were published:
> Feb 06 12:13:01 systemname filebeat[2425]: 2020-02-06T12:13:01.701-0500 DEBUG [processors] processing/processors.go:186 Publish event: {
>
> Feb 06 12:13:02 systemname filebeat[2425]: 2020-02-06T12:13:02.423-0500 DEBUG [elasticsearch] elasticsearch/client.go:348 PublishEvents: 2 events have been published to elasticsearch in
> Feb 06 12:13:02 systemname filebeat[2425]: 2020-02-06T12:13:02.423-0500 DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [20: 0, 2]
> Feb 06 12:13:02 systemname filebeat[2425]: 2020-02-06T12:13:02.423-0500 DEBUG [publisher] memqueue/eventloop.go:535 broker ACK events: count=2, start-seq=44, end-seq=45
> Feb 06 12:13:02 systemname filebeat[2425]: 2020-02-06T12:13:02.423-0500 DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:2
> Feb 06 12:13:02 systemname filebeat[2425]: 2020-02-06T12:13:02.423-0500 DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
> Feb 06 12:13:02 systemname filebeat[2425]: 2020-02-06T12:13:02.423-0500 DEBUG [acker] beater/acker.go:69 stateless ack {"count": 2}
> Feb 06 12:13:08 systemname filebeat[2425]: 2020-02-06T12:13:08.499-0500 DEBUG [input] input/input.go:152 Run input
We ran a packet capture on the filebeat node and confirmed the Elasticsearch cluster was receiving the message and responding with an HTTP 200. From the Packet Capture's HTTP Stream:
```
POST /_bulk HTTP/1.1
Host: elasticSearchNodeHere:9200
User-Agent: Go-http-client/1.1
Content-Length: 2862
Accept: application/json
Content-Type: application/json; charset=UTF-8
Accept-Encoding: gzip
{"create":{"_index":"filebeat-7.5.2","pipeline":"filebeat-7.5.2-panw-panos-pipeline"}}
{"@timestamp":"2020-02-06T15:12:37.000Z","event":{"timezone":"-05:00","created":"2020/02/06 10:12:37","severity":6,"module":"panw","dataset":"panw.panos"},"fileset":{"name":"panos"},"input":{"type":"syslog"},"ecs":{"version":"1.1.0"},"agent":{"ephemeral_id":"8c77978b-b0b1-4df9-ac7f-ec835acaf2b6","hostname":"systemname","id":"d1a2322e-a921-4dff-8b15-7c527ed20097","version":"7.5.2","type":"filebeat"},"hostname":"dataHere","log":{"source":{"address":"someaddress:someport"}},"tags":["pan-os"],"host":{"name":"systemname","hostname":"systemname","architecture":"x86_64","os":{"platform":"rhel","version":"7.7 (Maipo)","family":"redhat","name":"Red Hat Enterprise Linux Server","kernel":"3.10.0-1062.12.1.el7.x86_64","codename":"Maipo"},"id":"78dc6a03fa634f6b883b97250d1a90fe","containerized":false},"_temp_":{"message_type":"SYSTEM","message_subtype":"general","generated_time":"2020/02/06 10:12:37"},"syslog":{"facility":1,"facility_label":"user-level","priority":14,"severity_label":"Informational"},"service":{"type":"panw"},"message":"1,2020/02/06 10:12:37,002201001018,SYSTEM,general,0,2020/02/06 10:12:37,,general,,0,0,general,informational,\"FqdnRefresh job enqueued. Enqueue time=2020/02/06 10:12:37. JobId=204605. . Type: Full\",2950462,0x0,0,0,0,0,,ifw-pa5050-fo","observer":{"serial_number":"002201001018"}}
{"create":{"_index":"filebeat-7.5.2","pipeline":"filebeat-7.5.2-panw-panos-pipeline"}}
{"@timestamp":"2020-02-06T15:12:37.000Z","syslog":{"priority":14,"severity_label":"Informational","facility":1,"facility_label":"user-level"},"event":{"created":"2020/02/06 10:12:37","severity":6,"dataset":"panw.panos","module":"panw","timezone":"-05:00"},"fileset":{"name":"panos"},"tags":["pan-os"],"agent":{"id":"d1a2322e-a921-4dff-8b15-7c527ed20097","version":"7.5.2","type":"filebeat","ephemeral_id":"8c77978b-b0b1-4df9-ac7f-ec835acaf2b6","hostname":"systemname"},"ecs":{"version":"1.1.0"},"host":{"name":"systemname","hostname":"systemname","architecture":"x86_64","os":{"codename":"Maipo","platform":"rhel","version":"7.7 (Maipo)","family":"redhat","name":"Red Hat Enterprise Linux Server","kernel":"3.10.0-1062.12.1.el7.x86_64"},"id":"78dc6a03fa634f6b883b97250d1a90fe","containerized":false},"message":"1,2020/02/06 10:12:37,002201001018,SYSTEM,general,0,2020/02/06 10:12:37,,general,,0,0,general,informational,\"FqdnRefresh job started processing. Dequeue time=2020/02/06 10:12:37. Job Id=204605. \",2950463,0x0,0,0,0,0,,ifw-pa5050-fo","hostname":"systemname","log":{"source":{"address":"someaddress:someport"}},"service":{"type":"panw"},"input":{"type":"syslog"},"observer":{"serial_number":"002201001018"},"_temp_":{"generated_time":"2020/02/06 10:12:37","message_type":"SYSTEM","message_subtype":"general"}}
```
Elastic Searches Response:
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-encoding: gzip
content-length: 188
I noticed it was using the "filebeat-7.5.2-panw-panos-pipeline" pipeline, and this seems to process without issue (I guess?). I checked to make sure the pipeline was intact:
"filebeat-7.5.2-panw-panos-pipeline" : {
"on\_failure" : [
{
"set" : {
"field" : "error.message",
"value" : "{{ \_ingest.on\_failure\_message }}"
}
},
{
"remove" : {
"ignore\_missing" : true,
"field" : [
"_temp_"
]
}
}
],
In Elasticsearch, I can see there is an index:
# curl -X GET localhost:9200/\_cat/indices
green open filebeat-7.5.2-2020.02.06-000001 foEkBgCNTe-AhdmXKWhC-w 1 1 0 0 566b 283b
green open .kibana\_task\_manager\_1 at0InUYVTzaJiqRpXoATng 1 1 2 1 32.5kb 16.2kb
green open .apm-agent-configuration GM60UyQvSGuaVmYuibxCtQ 1 1 0 0 566b 283b
green open .kibana\_1 bV0kNdxbRmuKTpGvkwBe5w 1 1 1059 47 1.2mb 598.3kb
But looking in the index there are no documents:
GET /filebeat-7.5.2-2020.02.06-000001/\_search
{
"query": {
"match\_all": {}
}
}
and this returned:
{
"took" : 2,
"timed\_out" : false,
"\_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max\_score" : null,
"hits" :
}
}
I cant see it in kibana or doing a search within Elasticsearch. The document count remains at 0, though there is clearly data coming in. I checked to make sure filebeat, kibana, and Elasticsearch are all on the same version, they are. I shut everything down, removed all data inside Elasticsearch's data directory, started Elasticsearch, kibana, filebeat, confirmed file beat loaded the pipeline, then re-added the dashboards in Kibana but the same problem persists. I am at a loss. Can you provide assistance. Thanks.
I cant see it in kibana or doing a search within Elasticsearch. The document count remains at 0, though there is clearly data coming in. I checked to make sure filebeat, kibana, and Elasticsearch are all on the same version, they are. I shut everything down, removed all data inside Elasticsearch's data directory, started Elasticsearch, kibana, filebeat, confirmed file beat loaded the pipeline, then re-added the dashboards in Kibana but the same problem persists. I am at a loss. Can you provide assistance. Thanks.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [February 6, 2020, 7:04pm UTC](https://discuss.elastic.co/t/elastic-search-is-receiving-data-from-beat-but-no-documents-appear/218230/2 "2020-02-06T19:04:26Z")
> [@dsdameron](#):
>
> Elasticsearch cluster was receiving the message and responding with an HTTP 200
>
> ...
>
> Elastic Searches Response:
>
> ```auto
> HTTP/1.1 200 OK
> content-type: application/json; charset=UTF-8
> content-encoding: gzip
> content-length: 188
>
> ```
Unfortunately a `200 OK` response might still mean there were document-level failures; HTTP does not have a status code that means "some parts of this request succeeded and other parts failed". You must look at the body of the response to see the document-by-document status.
---
### Author:  [@dsdameron](https://discuss.elastic.co/u/dsdameron)
#### Post date: [February 6, 2020, 7:12pm UTC](https://discuss.elastic.co/t/elastic-search-is-receiving-data-from-beat-but-no-documents-appear/218230/3 "2020-02-06T19:12:03Z")
Thanks David, I didn't paste the entire response from the packet capture. On a side note, is there an easier way I can get this? The response in the capture is as follows:
```
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-encoding: gzip
content-length: 183
{"took":0,"ingest_took":1,"errors":false,"items":[{"create":{"_index":"filebeat-7.5.2","_type":"_doc","_id":"auto-generated","_version":-4,"result":"noop","_shards":{"total":0,"successful":0,"failed":0},"status":200}}]}
```
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [February 6, 2020, 7:48pm UTC](https://discuss.elastic.co/t/elastic-search-is-receiving-data-from-beat-but-no-documents-appear/218230/4 "2020-02-06T19:48:17Z")
> [@dsdameron](#):
>
> `"result":"noop"`
It looks like the pipeline has decided that nothing needs doing for this document.
---
### Author:  [@dsdameron](https://discuss.elastic.co/u/dsdameron)
#### Post date: [February 6, 2020, 8:03pm UTC](https://discuss.elastic.co/t/elastic-search-is-receiving-data-from-beat-but-no-documents-appear/218230/5 "2020-02-06T20:03:57Z")
I don't understand why the pipeline would not run on this data. How can I see what the pipe line is getting, what it is processing to determine this, and at what point did it decide nothing needed to be done?
FWIW - We just installed filebeat on a separate system, enabled the system module, loaded the template, dashboards, and confirmed filebeat was publishing to elastic search (In filebeat's logs). It was, however the index is still empty. So this problem is reproduceable with multiple pipelines. Doesn't think mean the issue is related to the index itself and not the pipeline? Or something that happens before the pipelines? How can I get more data to see what is going on?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [February 6, 2020, 9:19pm UTC](https://discuss.elastic.co/t/elastic-search-is-receiving-data-from-beat-but-no-documents-appear/218230/6 "2020-02-06T21:19:10Z")
I think it's not that the pipeline isn't running, it's that it is running and deliberately dropping the data. That's a thing that pipelines can do: [https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest-conditionals.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest-conditionals.html)
I'm not familiar with this particular pipeline, however. Where has it come from? Can you share its definition here?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [February 6, 2020, 9:36pm UTC](https://discuss.elastic.co/t/elastic-search-is-receiving-data-from-beat-but-no-documents-appear/218230/7 "2020-02-06T21:36:12Z")
Ah I see, it's something that comes with `filebeat`? I think it's probably a good idea to ask this on the [Beats forum](https://discuss.elastic.co/c/beats/28), since it's more likely there will be people there who know what this pipeline's supposed to do and why it seems to be dropping this data.
---
### Author:  [@dsdameron](https://discuss.elastic.co/u/dsdameron)
#### Post date: [February 6, 2020, 9:47pm UTC](https://discuss.elastic.co/t/elastic-search-is-receiving-data-from-beat-but-no-documents-appear/218230/8 "2020-02-06T21:47:33Z")
Thanks David. This is the part that blows my mind. This is happening to everything I setup using filebeat. In the example I posted above I installed file beat on a system, enabled the panw module:
[Filebeat - panw module](https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-module-panw.html)
This one wasn't working, so I opened this thread. I then stood up a different test system to send data to elastic search, installed and configured filebeat, enabled the system module:
[Filebeat- System Module](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-system.html)
Walked through the configuration to load the templates, pipelines, dashboards, etc.... I had debug enabled on filebeat and watched the logging, there were no errors at this step. Everything should have been good to go. Yet, there are still no documents in the filebeat index. Two different sources (Filebeat's panw and system modules) from two different systems with two different pipelines are sending data but nothing is appearing. I'm wondering if there is a step before pipelines thats broken and I'm missing, but I don't know what that would be. Also, I shut the system down, deleted the contents of the data directory, brought ES back up, checked its logs, re-setup the beats, but the problem persists. I'll post both pipelines in question shortly, but I'm guessing the problem is between elastic search is getting the data and elastic search is running the pipeline. Could it be something at the index level itself I'm missing? I didn't change anything in regards to aliases etc (I think).
---
### Author:  [@dsdameron](https://discuss.elastic.co/u/dsdameron)
#### Post date: [February 6, 2020, 9:50pm UTC](https://discuss.elastic.co/t/elastic-search-is-receiving-data-from-beat-but-no-documents-appear/218230/9 "2020-02-06T21:50:27Z")
Thanks David. I'll open a thread on the Beats now. I'm still trying to figure out how I can see in elastic search's logs 1) What It got before doing anyting 2) What it did once it got it, and 3) What process it followed. Is it possible to increase logging such that elastic search grabs this and the json its producing? I had to do a packet capture to get the output I posted here. I'm wondering if there is a better way.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [February 7, 2020, 10:52am UTC](https://discuss.elastic.co/t/elastic-search-is-receiving-data-from-beat-but-no-documents-appear/218230/10 "2020-02-07T10:52:35Z")
I've looked through the source and I do not see any obvious additional logging that can be enabled to trace a single document through an ingest pipeline, sorry.
Although I'm not familiar with how Beats configures Elasticsearch, I had a quick look at the source and I see a single `drop` processor here:
>
Is this the pipeline you're running? If so, it looks like it drops messages that do not have `_temp_.message_type` set to `TRAFFIC` or `THREAT` and in the request above this field is `SYSTEM`.
---
### Author:  [@dsdameron](https://discuss.elastic.co/u/dsdameron)
#### Post date: [February 7, 2020, 2:06pm UTC](https://discuss.elastic.co/t/elastic-search-is-receiving-data-from-beat-but-no-documents-appear/218230/11 "2020-02-07T14:06:28Z")
Hey Daniel,
This looks to be the same pipeline. For reference, here is the pipeline I pulled from ES. Please note, this issue is affecting multiple filebeat modules (the system and panw module), deployed on multiple systems, and each module is using its own ingest pipline. The only commonality I see between these is the index itself (or something I'm missing). Wouldn't this imply the problem lies outside of an individual pipeline?
---
### Author:  [@dsdameron](https://discuss.elastic.co/u/dsdameron)
#### Post date: [February 7, 2020, 2:07pm UTC](https://discuss.elastic.co/t/elastic-search-is-receiving-data-from-beat-but-no-documents-appear/218230/12 "2020-02-07T14:07:21Z")
Panw pipline (Part 1 of 2):
```auto
{
"filebeat-7.5.2-panw-panos-pipeline" : {
"on_failure" : [
{
"set" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
},
{
"remove" : {
"ignore_missing" : true,
"field" : [
"_temp_"
]
}
}
],
"description" : "Pipeline for Palo Alto Networks PAN-OS Logs",
"processors" : [
{
"rename" : {
"field" : "message",
"target_field" : "log.original"
}
},
{
"date" : {
"if" : "ctx.event.timezone == null",
"field" : "_temp_.generated_time",
"formats" : [
"yyyy/MM/dd HH:mm:ss"
],
"on_failure" : [
{
"append" : {
"value" : "{{ _ingest.on_failure_message }}",
"field" : "error.message"
}
}
]
}
},
{
"date" : {
"if" : "ctx.event.timezone != null",
"field" : "_temp_.generated_time",
"formats" : [
"yyyy/MM/dd HH:mm:ss"
],
"timezone" : "{{ event.timezone }}",
"on_failure" : [
{
"append" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}
]
}
},
{
"date" : {
"field" : "event.created",
"target_field" : "event.created",
"formats" : [
"yyyy/MM/dd HH:mm:ss"
],
"on_failure" : [
{
"append" : {
"value" : "{{ _ingest.on_failure_message }}",
"field" : "error.message"
}
}
],
"if" : "ctx.event.timezone == null && ctx.event.created != null "
}
},
{
"date" : {
"on_failure" : [
{
"append" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}
],
"if" : "ctx.event.timezone != null && ctx.event.created != null ",
"field" : "event.created",
"target_field" : "event.created",
"formats" : [
"yyyy/MM/dd HH:mm:ss"
],
"timezone" : "{{ event.timezone }}"
}
},
{
"date" : {
"target_field" : "event.start",
"formats" : [
"yyyy/MM/dd HH:mm:ss"
],
"on_failure" : [
{
"append" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}
],
"if" : "ctx.event.timezone == null && ctx.event.start != null",
"field" : "event.start"
}
},
{
"date" : {
"if" : "ctx.event.timezone != null && ctx.event.start != null",
"field" : "event.start",
"target_field" : "event.start",
"timezone" : "{{ event.timezone }}",
"formats" : [
"yyyy/MM/dd HH:mm:ss"
],
"on_failure" : [
{
"append" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}
]
}
},
{
"convert" : {
"ignore_missing" : true,
"field" : "client.bytes",
"type" : "long"
}
},
{
"convert" : {
"type" : "long",
"ignore_missing" : true,
"field" : "client.packets"
}
},
{
"convert" : {
"type" : "long",
"ignore_missing" : true,
"field" : "client.port"
}
},
{
"convert" : {
"ignore_missing" : true,
"field" : "server.bytes",
"type" : "long"
}
},
{
"convert" : {
"type" : "long",
"ignore_missing" : true,
"field" : "server.packets"
}
},
{
"convert" : {
"field" : "server.port",
"type" : "long",
"ignore_missing" : true
}
},
{
"convert" : {
"type" : "long",
"ignore_missing" : true,
"field" : "source.bytes"
}
},
{
"convert" : {
"type" : "long",
"ignore_missing" : true,
"field" : "source.packets"
}
},
{
"convert" : {
"type" : "long",
"ignore_missing" : true,
"field" : "source.port"
}
},
{
"convert" : {
"type" : "long",
"ignore_missing" : true,
"field" : "destination.bytes"
}
},
{
"convert" : {
"field" : "destination.packets",
"type" : "long",
"ignore_missing" : true
}
},
{
"convert" : {
"type" : "long",
"ignore_missing" : true,
"field" : "destination.port"
}
},
{
"convert" : {
"type" : "long",
"ignore_missing" : true,
"field" : "network.bytes"
}
},
{
"convert" : {
"type" : "long",
"ignore_missing" : true,
"field" : "network.packets"
}
},
{
"convert" : {
"type" : "long",
"ignore_missing" : true,
"field" : "event.duration"
}
},
{
"convert" : {
"ignore_missing" : true,
"field" : "_temp_.labels",
"type" : "long"
}
},
{
"convert" : {
"type" : "long",
"ignore_missing" : true,
"field" : "panw.panos.sequence_number"
}
},
{
"convert" : {
"ignore_missing" : true,
"field" : "source.nat.port",
"type" : "long"
}
},
{
"convert" : {
"ignore_missing" : true,
"field" : "destination.nat.port",
"type" : "long"
}
},
{
"remove" : {
"if" : "ctx?.panw?.panos?.network?.pcap_id == \"0\"",
"field" : [
"panw.panos.network.pcap_id"
]
}
},
{
"script" : {
"lang" : "painless",
"if" : "ctx?._temp_?.labels != null && ctx._temp_.labels != 0",
"params" : {
"container_page" : 32768,
"symmetric_return" : 2048,
"ssl_decrypted" : 16777216,
"temporary_match" : 8192,
"ipv6_session" : 33554432,
"x_forwarded_for" : 524288,
"http_proxy" : 262144,
"captive_portal" : 2097152,
"nat_translated" : 4194304,
"pcap_included" : 2147483648,
"url_filter_denied" : 8388608
},
"source" : """def labels = ctx?.labels; if (labels == null) {
labels = new HashMap();
ctx['labels'] = labels;
} long value = ctx._temp_.labels; for (entry in params.entrySet()) {
if ((value & entry.getValue()) != 0) {
labels[entry.getKey()] = true;
}
}
"""
}
},
{
"script" : {
"params" : {
"NANOS_IN_A_SECOND" : 1000000000
},
"source" : """long nanos = ctx['event']['duration'] * params.NANOS_IN_A_SECOND; ctx['event']['duration'] = nanos; def start = ctx.event?.start; if (start != null) {
ctx.event['end'] = ZonedDateTime.parse(start).plusNanos(nanos);
}
""",
"lang" : "painless",
"if" : "ctx?.event?.duration != null"
}
},
{
"set" : {
"value" : "inbound",
"if" : "ctx?._temp_?.message_type == \"TRAFFIC\" && ctx?.panw?.panos?.source?.zone == \"untrust\" && ctx?.panw?.panos?.destination?.zone == \"trust\"",
"field" : "network.direction"
}
},
{
"set" : {
"value" : "outbound",
"if" : "ctx?._temp_?.message_type == \"TRAFFIC\" && ctx?.panw?.panos?.source?.zone == \"trust\" && ctx?.panw?.panos?.destination?.zone == \"untrust\"",
"field" : "network.direction"
}
},
{
"set" : {
"field" : "network.direction",
"value" : "internal",
"if" : "ctx?._temp_?.message_type == \"TRAFFIC\" && ctx?.panw?.panos?.source?.zone == \"trust\" && ctx?.panw?.panos?.destination?.zone == \"trust\""
}
},
```
---
### Author:  [@dsdameron](https://discuss.elastic.co/u/dsdameron)
#### Post date: [February 7, 2020, 2:08pm UTC](https://discuss.elastic.co/t/elastic-search-is-receiving-data-from-beat-but-no-documents-appear/218230/13 "2020-02-07T14:08:00Z")
Panw Pipline (part 2 of 2):
```auto
{
"set" : {
"value" : "external",
"if" : "ctx?._temp_?.message_type == \"TRAFFIC\" && ctx?.panw?.panos?.source?.zone == \"untrust\" && ctx?.panw?.panos?.destination?.zone == \"untrust\"",
"field" : "network.direction"
}
},
{
"set" : {
"value" : "unknown",
"if" : """ctx?._temp_?.message_type == "TRAFFIC" && ((ctx?.panw?.panos?.source?.zone != "trust" && ctx?.panw?.panos?.source?.zone != "untrust") || (ctx?.panw?.panos?.destination?.zone != "trust" && ctx?.panw?.panos?.destination?.zone != "untrust"))""",
"field" : "network.direction"
}
},
{
"set" : {
"if" : """ctx?._temp_?.message_type == "THREAT" && (ctx?._temp_?.direction == "0" || ctx?._temp_?.direction == "client-to-server")""",
"field" : "network.direction",
"value" : "inbound"
}
},
{
"set" : {
"field" : "network.direction",
"value" : "outbound",
"if" : """ctx?._temp_?.message_type == "THREAT" && (ctx?._temp_?.direction == "1" || ctx?._temp_?.direction == "server-to-client")"""
}
},
{
"set" : {
"field" : "network.direction",
"value" : "unknown",
"if" : """ctx?._temp_?.message_type == "THREAT" && ctx?.network?.direction == null"""
}
},
{
"set" : {
"field" : "network.type",
"value" : "ipv4",
"if" : """ctx?._temp_?.message_type == "TRAFFIC" && ctx?.labels?.ipv6_session == null"""
}
},
{
"set" : {
"field" : "network.type",
"value" : "ipv6",
"if" : """ctx?._temp_?.message_type == "TRAFFIC" && ctx?.labels?.ipv6_session != null"""
}
},
{
"set" : {
"field" : "event.category",
"value" : "network_traffic",
"if" : "ctx?._temp_?.message_type == \"TRAFFIC\""
}
},
{
"set" : {
"field" : "event.category",
"value" : "security_threat",
"if" : "ctx?._temp_?.message_type == \"THREAT\""
}
},
{
"drop" : {
"if" : "ctx?.event?.category == null"
}
},
{
"set" : {
"field" : "event.action",
"value" : "flow_started",
"if" : "ctx?._temp_?.message_subtype == \"start\""
}
},
{
"set" : {
"if" : "ctx?._temp_?.message_subtype == \"end\"",
"field" : "event.action",
"value" : "flow_terminated"
}
},
{
"set" : {
"field" : "event.action",
"value" : "flow_dropped",
"if" : "ctx?._temp_?.message_subtype == \"drop\""
}
},
{
"set" : {
"field" : "event.action",
"value" : "flow_denied",
"if" : "ctx?._temp_?.message_subtype == \"deny\""
}
},
{
"set" : {
"field" : "event.action",
"value" : "data_match",
"if" : "ctx?._temp_?.message_subtype == \"data\""
}
},
{
"set" : {
"value" : "file_match",
"if" : "ctx?._temp_?.message_subtype == \"file\"",
"field" : "event.action"
}
},
{
"set" : {
"field" : "event.action",
"value" : "flood_detected",
"if" : "ctx?._temp_?.message_subtype == \"flood\""
}
},
{
"set" : {
"field" : "event.action",
"value" : "packet_attack",
"if" : "ctx?._temp_?.message_subtype == \"packet\""
}
},
{
"set" : {
"field" : "event.action",
"value" : "scan_detected",
"if" : "ctx?._temp_?.message_subtype == \"scan\""
}
},
{
"set" : {
"if" : "ctx?._temp_?.message_subtype == \"spyware\"",
"field" : "event.action",
"value" : "spyware_detected"
}
},
{
"set" : {
"if" : "ctx?._temp_?.message_subtype == \"url\"",
"field" : "event.action",
"value" : "url_filtering"
}
},
{
"set" : {
"if" : "ctx?._temp_?.message_subtype == \"virus\"",
"field" : "event.action",
"value" : "virus_detected"
}
},
{
"set" : {
"field" : "event.action",
"value" : "exploit_detected",
"if" : "ctx?._temp_?.message_subtype == \"vulnerability\""
}
},
{
"set" : {
"value" : "wildfire_veredict",
"if" : "ctx?._temp_?.message_subtype == \"wildfire\"",
"field" : "event.action"
}
},
{
"set" : {
"field" : "event.action",
"value" : "wildfire_virus_detected",
"if" : "ctx?._temp_?.message_subtype == \"wildfire-virus\""
}
},
{
"set" : {
"value" : 1,
"field" : "event.severity",
"if" : "ctx.log.level == \"critical\""
}
},
{
"set" : {
"field" : "event.severity",
"if" : "ctx.log.level == \"high\"",
"value" : 2
}
},
{
"set" : {
"field" : "event.severity",
"if" : "ctx.log.level == \"medium\"",
"value" : 3
}
},
{
"set" : {
"if" : "ctx.log.level == \"low\"",
"value" : 4,
"field" : "event.severity"
}
},
{
"set" : {
"field" : "event.severity",
"if" : "ctx.log.level == \"informational\"",
"value" : 5
}
},
{
"set" : {
"field" : "event.outcome",
"value" : "drop-icmp",
"if" : "ctx?.event?.outcome == \"drop icmp\" || ctx?.event?.outcome == \"drop ICMP\""
}
},
{
"set" : {
"field" : "event.outcome",
"value" : "reset-both",
"if" : "ctx?.event?.outcome == \"reset both\""
}
},
{
"set" : {
"field" : "event.outcome",
"value" : "reset-client",
"if" : "ctx?.event?.outcome == \"reset client\""
}
},
{
"set" : {
"field" : "event.outcome",
"value" : "reset-server",
"if" : "ctx?.event?.outcome == \"reset server\""
}
},
{
"append" : {
"if" : "ctx?.source?.ip != null",
"field" : "related.ip",
"value" : [
"{{source.ip}}"
]
}
},
{
"append" : {
"value" : [
"{{destination.ip}}"
],
"if" : "ctx?.destination?.ip != null",
"field" : "related.ip"
}
},
{
"append" : {
"field" : "related.ip",
"value" : [
"{{source.nat.ip}}"
],
"if" : "ctx?.source?.nat?.ip != null"
}
},
{
"append" : {
"value" : [
"{{destination.nat.ip}}"
],
"if" : "ctx?.destination?.nat?.ip != null",
"field" : "related.ip"
}
},
{
"geoip" : {
"if" : "ctx?.source?.ip != null",
"field" : "source.ip",
"target_field" : "source.geo"
}
},
{
"geoip" : {
"target_field" : "destination.geo",
"if" : "ctx?.destination?.ip != null",
"field" : "destination.ip"
}
},
{
"geoip" : {
"ignore_missing" : true,
"database_file" : "GeoLite2-ASN.mmdb",
"field" : "source.ip",
"target_field" : "source.as",
"properties" : [
"asn",
"organization_name"
]
}
},
{
"geoip" : {
"database_file" : "GeoLite2-ASN.mmdb",
"field" : "destination.ip",
"target_field" : "destination.as",
"properties" : [
"asn",
"organization_name"
],
"ignore_missing" : true
}
},
{
"rename" : {
"field" : "source.as.asn",
"target_field" : "source.as.number",
"ignore_missing" : true
}
},
{
"rename" : {
"field" : "source.as.organization_name",
"target_field" : "source.as.organization.name",
"ignore_missing" : true
}
},
{
"rename" : {
"field" : "destination.as.asn",
"target_field" : "destination.as.number",
"ignore_missing" : true
}
},
{
"rename" : {
"field" : "destination.as.organization_name",
"target_field" : "destination.as.organization.name",
"ignore_missing" : true
}
},
{
"append" : {
"value" : [
"{{panw.panos.network.nat.community_id}}"
],
"if" : "ctx?.panw?.panos?.network?.nat?.community_id != null && ctx.panw.panos.network.nat.community_id != ctx?.network?.community_id",
"field" : "network.community_id"
}
},
{
"grok" : {
"patterns" : [
"%{GREEDYDATA:panw.panos.threat.name}\\(\\s*%{GREEDYDATA:panw.panos.threat.id}\\s*\\)"
],
"if" : "ctx?.panw?.panos?.threat?.name != null",
"field" : "panw.panos.threat.name",
"ignore_failure" : true
}
},
{
"set" : {
"field" : "panw.panos.threat.name",
"value" : "URL-filtering",
"if" : "ctx?.panw?.panos?.threat?.id == \"9999\""
}
},
{
"remove" : {
"ignore_missing" : true,
"field" : [
"_temp_"
]
}
}
]
}
}
```
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [February 7, 2020, 2:25pm UTC](https://discuss.elastic.co/t/elastic-search-is-receiving-data-from-beat-but-no-documents-appear/218230/14 "2020-02-07T14:25:49Z")
(I reformatted your posts to preserve whitespace with the `` button)
> [@dsdameron](#):
>
> ```auto
> {
> "set" : {
> "field" : "event.category",
> "value" : "network_traffic",
> "if" : "ctx?._temp_?.message_type == \"TRAFFIC\""
> }
> },
> {
> "set" : {
> "field" : "event.category",
> "value" : "security_threat",
> "if" : "ctx?._temp_?.message_type == \"THREAT\""
> }
> },
> {
> "drop" : {
> "if" : "ctx?.event?.category == null"
> }
> },
>
> ```
Yep, looks like this pipeline drops messages of type `SYSTEM`.
---
### Author:  [@dsdameron](https://discuss.elastic.co/u/dsdameron)
#### Post date: [February 10, 2020, 7:44pm UTC](https://discuss.elastic.co/t/elastic-search-is-receiving-data-from-beat-but-no-documents-appear/218230/15 "2020-02-10T19:44:29Z")
I understand. Thank you for your assistance. My takeaway is I need to learn more about pipelines and how to take apart what they are doing.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 9, 2020, 7:44pm UTC](https://discuss.elastic.co/t/elastic-search-is-receiving-data-from-beat-but-no-documents-appear/218230/16 "2020-03-09T19:44:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.