Configuring pipeline in filebeat module - nginx

Elastic Stack Beats
1

I'm trying to figure out how to configure a pipeline in my nginx filebeat module.

This link https://www.elastic.co/guide/en/beats/filebeat/master/configuring-ingest-node.html shows a general top-level configuration for all filebeats. This however is not ideal for me since it shows pre-processing errors for all other logs that fail the grok.

I have scoured the docs finding this https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html which I used to add an entry to my nginx.yml module configuration as so

- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:
    pipeline: filebeat-6.4.3-nginx-access-custom

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

I restarted filebeat but this still doesn't take effect. I know my pipeline is correct because I have tested with the _simulate endpoint and have even overridden the default nginx pipeline and it worked as expected.

2

Hi @crazywizard

I'm not sure about what error you may have. Please, ensure that the ingest node is not disabled . Also take a look in the same link that talks about the pipeline endpoint.

Just to double check, your pipeline must be PUT on ES https://www.elastic.co/guide/en/elasticsearch/reference/master/put-pipeline-api.html

I hope this helps.

3

Thanks for your response @Mario_Castro . I'm using the cloud offering, so not sure of the default ingest node setting for that.

As I mentioned previously, the pipeline was executed and created correctly and I even tested this with _simulate endpoint. For completeness, here is the request used

4

I think your configuration is incorrect. You need to add input to pass options properly.

- module: nginx
  # Access logs
  access:
    enabled: true
    input:
      pipeline: filebeat-6.4.3-nginx-access-custom