Everything mentioned here (ELK, filebeat etc.) is version 6.8.3. I run 1 single node of Elasticsearch.
I'm trying to use the ingest-pipelines that are included with FileBeat. As an example, I started with the one that processes nginx access logs.
The pipelines were loaded using the normal command:
filebeat setup --pipelines --modules nginx
So the flow is: logfile --> filebeat --> logstash --> elasticsearch
I created an example file and added it to Filebeat config:
- type: log
enabled: true
paths: ["/var/log/nginx_access_example"]
fields:
type: nginx_access
fields_under_root: true
I added the extra field so I can differentiate the file from others.
I followed this page Use ingest pipelines for parsing | Logstash Reference [6.8] | Elastic and so in Logstash I have something like this:
input {
beats {
port => 5044
ssl => false
}
}
output {
if [type] == "nginx_access" {
elasticsearch {
hosts => ["localhost"]
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.ww}"
pipeline => "filebeat-6.8.3-nginx-access-default"
}
} else {
elasticsearch {
hosts => ["localhost"]
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.ww}"
}
}
}
As you can see, I added the name of the pipeline to be used for my particular file.
After restarting everything, I tried to add example-access-log-lines to that file, but... nothing shows up in Elasticsearch 
1). All the connections between the components work fine, because stuff from other files appear correctly there.
2). There are no errors or warnings anywhere
3). I know that Filebeat picks up the changes, because every time I add example-log-lines to that file, I can immediately see in Filebeat logs that a harvester was started for that file.
So it doesn't work, no matter what I try.
What I did next was to assume that I did something wrong in Logstash, so I decided to bypass it entirely.
I changed Filebeat config to point directly to Elasticsearch-output and I changed the input like this:
- type: log
enabled: true
paths: ["/var/log/nginx_access_example"]
pipeline: 'filebeat-6.8.3-nginx-access-default'
fields:
type: nginx_access
fields_under_root: true
As you can see, I added the "pipeline" entry there.
Still nothing shows up in Elasticsearch.
Then I decided to do this:
1). Get the definition of the pipeline:
GET /_ingest/pipeline/filebeat-6.8.3-nginx-access-default
2). With that definition AND an example-log-line, I made use of the Ingest-Pipeline-Simulate endpoint:
POST <elasticsearch>/_ingest/pipeline/_simulate
{
"pipeline": {
............
},
"docs" : [
{
"_source": {
.............
}
}
]
}
And.... it worked exactly as I wanted. Everything got parsed correctly and split into fields etc.
So, in conclusion:
1). it works when I put it in the "simulate" endpoint
2). it does NOT work in a more "real-world" case, with Filebeat and/or Logstash involved.
3). there are NO errors or warnings anywhere
4). the pipeline does contain an "on_failure" entry:
"on_failure": [{
"set": {
"field": "error.message",
"value": "{{ _ingest.on_failure_message }}"
}
}
But of course it has no effect that I can see.
What am I missing? Thanks.
PS: I didn't post the entire definition of the pipeline because it's a bit long. But I can add if you think it can help.