ES 7.15 FileBeats Sophos XG module not separating data into variables

Elastic Stack Beats
2

Hi @Samy_Weee Welcome to the community

First I would usually try to send direct from Filebeat -> Elasticsearch and get that working?

Once you get that working .. .then I would move on to this Architecture

Filebeat -> Logstash -> Elasticsearch

And the question is... Do you actually need logstash? (it is not required)

If you do ... once you have Filebeat -> Elasticsearch working go into your filebeat.yml and then direct filebeat output to logstash output.

Then your logstash.conf should look something like this... this is a "Passthrough" configuration what is most likely happening is that you are missing this line

pipeline => "%{[@metadata][pipeline]}" which tells Elasticsearch which pipeline (AKA Parser) to use. When you go direct from Filebeat -> Elasticsearch that info is automatically passed along .. when you put logstash in the middle it is not.

Start Logstash first, and then filebeat...

################################################
# beats->logstash->es default config.
################################################
input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      user => "elastic"
      password => "secret"
    }
  }
}