Unable to create pipeline for nginx

[yukpun]

For some reason, nginx logs stopped going to elastick stack. I checked status, and I see this:

ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(http://elastic-server:9200)): Connection marked as failed because the on Connect callback failed: Error getting pipeline for fileset nginx/access: Error YAML decoding the pipeline file: ingest/pipeline.yml: yaml: invalid map key: map[interface {}]interface {}{"clientip":interface {}(nil)}

I cannot find out what's wrong. Where I can find that map key "clientip" which is mentioned above

[shaunak (Shaunak Kashyap)]

What versions of Filebeat and Elasticsearch are you running?

Shaunak

[yukpun]

7.9.3
Here is config:

version: '3'

services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.3
    container_name: elasticsearch2
    environment:
      - discovery.type=single-node
      - ES_JAVA_OPTS=-Xms8g -Xmx8g
    volumes:
      - ./es_data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"

  kibana:
    image: docker.elastic.co/kibana/kibana:7.9.3
    container_name: kibana2
    ports:
      - "5601:5601"
    environment:
      ELASTICSEARCH_URL: http://elasticsearch:9200
    links:
      - elasticsearch

And for filebeat:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false
filebeat.modules:
- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/access.log*"]
  error:
    enabled: true
    var.paths: ["/var/log/nginx/error.log*"]
  ingress_controller:
    enabled: false

output.elasticsearch:
  hosts: "http://SERVER_WITH_DOCKER:9200"
  index: "nginx-%{+yyyy.MM.dd}"
setup.template.pattern: "nginx-*"
setup.template.name: "nginx"
setup.template.settings:
  index.number_of_replicas: 0
setup.ilm.enabled: false
setup.template.fields: "fields.yml"
setup.template.overwrite: true

[yukpun]

I found wrong parameter in

/usr/share/filebeat/module/nginx/access/ingest/pipeline.yml

There was custom modification, which stopped working for some reason