I've created an ingest pipeline that works perfectly when I use the simulate endpoint, POST filebeat-7.6.2-2020.04.14/_doc?pipeline=pipeline_lookup_test
, and when I reindex the data into the index. But when I try and send the data from filebeat, the data comes in, but the pipeline doesn't work. I've tried making the pipeline more simple (just setting a field) and also that doesn't work.
I've PUT filebeat-7.6.2-2020.04.14/_settings { "index" : { "default_pipeline" : "pipeline_lookup_test" } }
And also put in filebeat.yml
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["http://localhost:9200"]
pipeline: "pipeline_lookup_test"
But none of it works. Again, this works when I POST, simulate and reindex, but not when I send the data via filebeat. I'm using a module (zeek), not sure if that would interfere...
- module: zeek
# All logs
connection:
enabled: true
var.paths: ["/usr/local/zeek/logs/current/conn*.log"]
dns:
enabled: true
var.paths: ["/usr/local/zeek/logs/current/dns.log", "/usr/local/zeek/logs/current/ntp.log"]
http:
enabled: true
var.paths: ["/usr/local/zeek/logs/current/http.log"]
files:
enabled: true
var.paths: ["/usr/local/zeek/logs/current/files.log"]
ssl:
enabled: true
var.paths: ["/usr/local/zeek/logs/current/ssl.log"]
notice:
enabled: true
var.paths: ["/usr/local/zeek/logs/current/notice.log"]
weird:
enabled: true
var.paths: ["/usr/local/zeek/logs/current/weird.log"]