Looking for help with ingest pipeline

Elastic Stack
1

Hi there, I'm having trouble with an ingest pipeline in elasticsearch, I've clearly missed something obvious.

Filebeat config

`filebeat.inputs:

  • type: log
    enabled: true
    paths:
    • D:/artifactory/logs/access.log
      pipeline: "rtf-access-log-pipeline"
      output.elasticsearch:
      hosts: ["elasticsearch:9200"]
      username: "elastic"
      password: "A PASSWORD"
      setup.kibana:
      host: "kibana:5601"
      username: "elastic"
      password: "A PASSWORD"`
Ingest pipeline

{ "rtf-access-log-pipeline" : { "description" : "Process artifactory access logs", "processors" : [ { "grok" : { "field" : "message", "patterns" : [ "%{TIMESTAMP_ISO8601:rtf.access.timestamp} \\[%{DATA:rtf.access.action_resp} %{DATA:rtf.access.action_type}\\] %{DATA:rtf.access.repo_name}\\:%{DATA:rtf.access.file_path}\\s+.* %{USERNAME:rtf.access.username}/%{IP:rtf.access.client_ip}", "%{TIMESTAMP_ISO8601:rtf.access.timestamp} \\[%{DATA:rtf.access.action_resp} %{DATA:rtf.access.action_type}\\] %{DATA:rtf.access.repo_name}\\:%{DATA:rtf.access.file_path}\\s+.* %{GREEDYDATA:rtf.access.other}", "%{TIMESTAMP_ISO8601:rtf.access.timestamp} \\\\[%{DATA:rtf.access.action_resp} %{DATA:rtf.access.action_type}\\\\]\\s+for client\\s+:\\s+%{USERNAME:rtf.access.username}\\/%{IP:rtf.access.client_ip}" ], "on_failure" : [ { "set" : { "field" : "error", "value" : "{{ _ingest.on_failure_message }}" } } ] } }, { "date" : { "field" : "rtf.access.timestamp", "target_field" : "@timestamp", "formats" : [ "YYYY-MM-dd HH:mm:ss,SSS" ], "timezone" : "Europe/Stockholm", "on_failure" : [ { "set" : { "field" : "error", "value" : "{{ _ingest.on_failure_message }}" } } ] } }, { "remove" : { "field" : "message" } } ] } }

If I comment out the pipeline in filebeat, or just mangle the grok patterns in the ingest pipeline, log entries appear fine in the filebeat indicies, however, the moment entries match the grok patterns they don't get put into the index. What is happening? What have I missed?

Please any pointers would really help, I'd be tearing my hair out if I weren't already bald