Ingest pipeline differences with Logstash and Beats

[b133 (Brian Lee)]

I want to bifurcate my logs using Logstash. When I sent logs directly from Filebeat to Elasticsearch, the log fields were all properly parsed and converted to appropriate variable types.

I have since changed the Filebeat output to Logstash and configured Logstash to send logs to Elasticsearch. I have also made sure to load the ingest pipelines manually:
filebeat setup --pipelines --modules panw,checkpoint

Even after loading the ingest pipelines, the log fields are not being fully parsed in Elasticsearch. For my panw module logs, some fields are parsed but most of the type conversion is lost. For checkpoint module logs, it fails to parse and identify most fields, leaving them lumped in the message field.

My temporary workaround is just to mirror the incoming traffic to another virtual host and run logstash over there. But that's a pretty ugly workaround until I can figure out how to get Filebeat and Logstash to cooperate. I'm guessing I just missed a step, but I'm not sure how to continue troubleshooting at this point. Any advice?

Here's a relevant bit of my Logstash configuration:

input {
     beats {
        host => "127.0.0.1"
        port => "5044"
    }
}
output {
    if "pan-os" in [tags] {
        elasticsearch {
           ilm_policy => "my-policy-01"
           ilm_rollover_alias => "panw"
           ilm_pattern => "{now/d}-01"
           hosts => ["localhost:9200"]
           user => "${es_user}"
           password => "${es_pwd}"
       }
    }
...

[Badger]

I think you need to be setting the pipeline option on the elasticsearch output so that the output will tell elasticsearch which ingest pipeline to run.

[b133 (Brian Lee)]

Thanks for your quick response! I'm now seeing parsed fields after setting the Beats pipeline within my Logstash output{elasticsearch{}} stanza:

pipeline => "%{[@metadata][pipeline]}"
manage_template => false

Cheers