Debugging ingest pipelines

Elastic Stack Beats
1

Since I'm using custom apache log format and had issues with apache module extracting the correct source ip I wrote my own ingest pipeline for apache access logs and disabled the module. That part works fine, but now I also need to define the pipeline for error logs. Did that by copying the one generated by elastic for version 7.8.1 and used it in the input.d file (same as I did for access logs). However I see neither error logs from apache in kibana nor error with filebeat sending the logs.

How would I go about debugging this?

I've checked service logs for filebeat (journalctl -u filebeat) as well as log files at /var/log/filebeat/filebeat* but I see no mention of any errors or issues.
I've also tried using the "test pipeline" feature in management panel, but I get this error in both elastic generated pipeline as well as my 1-1 copy of it:

{
  "docs": [
    {
      "doc": {
        "_index": "index",
        "_type": "_doc",
        "_id": "id",
        "_source": {
          "error": {
            "message": "Cannot invoke \\\"Object.getClass()\\\" because \\\"receiver\\\" is null"
          }
        },
        "_ingest": {
          "timestamp": "2020-08-06T08:13:10.117977Z"
        }
      }
    }
  ]
}

Using this as a test doc (I've no idea if this is correct though, just put a sample log line in the message; I've replaced the ip in the message to 1.1.1.1):

[
  {
    "_index": "index",
    "_id": "id",
    "_source": {},
    "message": "[Thu Aug 05 07:20:57.304119 2020] [core:error] [pid 22413:tid 139902310147840] [client 1.1.1.1:44770] AH00126: Invalid URI in request 'etc/passwd' 'GET /dana-na/../dana/html5acc/guacamole/../../../../../../etc/passwd?/dana/html5acc/guacamole/ HTTP/1.1'"
  }
]

Input.d conf file for reference:

cat /etc/filebeat/inputs.d/apache_error.yml
- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/httpd/error_log*

  exclude_files: ['.gz$']

  # checked for type-os and the name is correct
  pipeline: filebeat-apache-error-log
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.