How to debug filebeat issues

Hello,
I installed filebeat, used with apache module a few month ago.
Everything was ok since I try to add the response time to my apache logs using this tutorial : https://www.partiallydisassembled.net/posts/filebeat-fields.html.
Then, the access logs are not sent to elastic search anymore, but the error logs is still working.

I spent my whole day trying to debug the issue but cannot understand where the problem really is.

I don't see any issues explaining it in the filebeat logs

Could you help me ?

Some complementaty informations :

#/etc/filebeat/filebeat.yml

#========================= Central Management =================================
# Beats is configured under central management, you can define most settings
# from the Kibana UI. You can update this file to configure the settings that
# are not supported by Kibana Beats management.
management:
  enabled: true
  period: 1m0s
  events_reporter:
    period: 30s
    max_batch_size: 1000
  access_token: ${management.accesstoken}
  kibana:
    protocol: https
    host: my-HOST
    ssl: null
    timeout: 10s
    ignoreversion: true
  blacklist:
    output: console|file
#=============================== General =====================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging
#================================ Logging =====================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: debug
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

#============================== Xpack Monitoring ===============================
# filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.
# Set to true to enable the monitoring reporter.
#monitoring.enabled: false
# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:
#============================= Elastic Cloud ==================================
# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
cloud.id: MY_ID
# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
cloud.auth: MY:CREDENTIALS
#============================= Modules ============================================
filebeat.config.modules:
  enabled: true
  path: ${path.config}/modules.d/*.yml
filebeat.overwrite_pipelines: true
#setup.template.name: "filebeat"




#/etc/filebeat/modules.d/apache.yml 
# Module: apache
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.3/filebeat-module-apache.html
- module: apache
  # Access logs
  access:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
   # var.paths: 
   #   - "/var/log/apache2/access.log*"
  # Error logs
  error:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths: 
    #  - "/var/log/apache2/error.log*"

apache log example :

127.0.0.1 - - [08/Nov/2019:06:32:51 +0000] "GET /server-status?auto= HTTP/1.1" 1199 200 593 "-" "Go-http-client/1.1" 
127.0.0.1 - - [08/Nov/2019:06:33:01 +0000] "GET /server-status?auto= HTTP/1.1" 1428 200 591 "-" "Go-http-client/1.1" 
108.162.221.146 - jeremy.gachet@gmail.com [08/Nov/2019:06:33:02 +0000] "HEAD / HTTP/1.1" 178080 200 2171 "http://spoon-elastic.com" "Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)" 
127.0.0.1 - - [08/Nov/2019:06:33:51 +0000] "GET /server-status?auto= HTTP/1.1" 1523 200 595 "-" "Go-http-client/1.1" 
172.69.94.62 - - [08/Nov/2019:06:33:57 +0000] "GET / HTTP/1.1" 293976 200 7949 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36" 
172.69.94.44 - - [08/Nov/2019:06:33:58 +0000] "GET /wp-content/uploads/2015/06/tv-dashboard-office.png HTTP/1.1" 1190 304 2100 "https://spoon-elastic.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78
.0.3904.70 Safari/537.36" 
127.0.0.1 - - [08/Nov/2019:06:34:01 +0000] "GET /server-status?auto= HTTP/1.1" 1240 200 595 "-" "Go-http-client/1.1"

My pipeline (well imported in ES) :
{
"filebeat-7.3.1-apache-access-default" : {
(...)
"processors" : [
{
"grok" : {
"field" : "message",
"patterns" : [
"""%{IPORHOST:source.address} - %{DATA:user.name} [%{HTTPDATE:apache.access.time}] "(?:%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}|-)?" %{NUMBER:http.response.time:long} %{NUMBER:http.response.status_code:long} (?:%{NUMBER:bytes}|-) ( "%{DATA:http.request.referrer}")?( "%{DATA:user_agent.original}")?""",
"""%{IPORHOST:source.address} - %{DATA:user.name} [%{HTTPDATE:apache.access.time}] "-" %{NUMBER:http.response.status_code:long}""",
"""[%{HTTPDATE:apache.access.time}] %{IPORHOST:source.address} %{DATA:apache.access.ssl.protocol} %{DATA:apache.access.ssl.cipher} "%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}" %{NUMBER:http.response.body.bytes:long}"""
],
"ignore_missing" : true

I found one error in logs :
2019-11-11T09:59:46.480Z ERROR [centralmgmt.event_reporter] api/event_reporter.go:90 could not send events, error: 1 error: Beat "36ba6f9d-cd2e-436e-b5d4-3427a305c59d" not found

I think it can be the issue, but I don't understand how to solve it

HI @Jeremy_Gachet,

I understand you overwrote the existing pipeline. Maybe something is wrong with the custom one? Either Elasticsearch logs or the documents reaching it should be showing some error. Did you try to find filebeat apache events in the discover tab?

Best regard

Hi and thank you for your answer.
Of course. I had access to error logs but not to access logs.
Finally I managed to fix my issue by totally remove and reinstall filebeat from the beginning.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.