Error parsing logs Filebeat - APACHE module

[marti1 (marti)]

Hi everyone,

I have configured my Apache server with Filebeat apache module to load ACCESS and ERROR logs to Elasticsearch but is not working properly, all the logs are not parsed (are raw). This is my configuration:

Apache Server Logs -> Filebeat -> Kafka -> Logstash -> Elasticsearch

When I connect my Filebeat directly to Elasticsearch that problem disappears and all the logs are parsed correctly.

Apache Server Logs -> Elasticsearch

Why? How can I configure my Filebeat to work well with Kafka?

Thanks

[marti1 (marti)]

SOLUTION:

To load pipelines to elasticsearch you must add this [pipline] option into the logstash output:

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "https://myEShost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "https://myEShost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      user => "elastic"
      password => "secret"
    }
  }
}