Logstash 6.5.0 stops shipping logs to ES

Elastic Stack Logstash
5

The file input has a lot more logging at the TRACE level but before you go and set the level to TRACE for all Logstash you should rather set levels dynamically for individual components at will. Here is a quote from an internal post that I made a while back. Please read it but don't do exactly what is written in this snippet (instructions for you after)...

The logging API allows for different levels of logging for different components in LS.

First do curl -XGET 'localhost:9600/_node/logging?pretty'
You see something like this:

{
  "host" : "Elastics-MacBook-Pro.local",
  "version" : "6.4.0",
  "http_address" : "127.0.0.1:9600",
  "id" : "8789409b-7126-4034-9347-de47e6ce12a9",
  "name" : "Elastics-MacBook-Pro.local",
  "loggers" : {
    "filewatch.discoverer" : "INFO",
    "filewatch.observingtail" : "INFO",
    "filewatch.sincedbcollection" : "INFO",
    "filewatch.tailmode.handlers.createinitial" : "INFO",
    "filewatch.tailmode.processor" : "INFO",
    "logstash.agent" : "INFO",
    "logstash.api.service" : "INFO",
    "logstash.codecs.json" : "INFO",
    ...
    "logstash.filters.grok" : "INFO",
    "logstash.filters.date" : "INFO",
    "logstash.inputs.file" : "INFO",
    ...
    "logstash.outputs.stdout" : "INFO",
    "logstash.pipeline" : "INFO",
    ...
    "slowlog.logstash.codecs.json" : "INFO",
    "slowlog.logstash.codecs.rubydebug" : "INFO",
    "slowlog.logstash.filters.date" : "INFO",
    "slowlog.logstash.inputs.file" : "INFO",
    "slowlog.logstash.outputs.stdout" : "INFO"
  }
}

Using the API
Turn DEBUG on for just the date and grok filters:

curl -XPUT 'localhost:9600/_node/logging?pretty' -H 'Content-Type: application/json' -d'
{
    "logstash.filters.date" : "DEBUG"
    "logstash.filters.grok" : "DEBUG"
}
'

Turn trace off:

curl -XPUT 'localhost:9600/_node/logging?pretty' -H 'Content-Type: application/json' -d'
{
    "logstash.filters.date" : "WARN"
    "logstash.filters.grok" : "WARN"
}
'

Or

curl -XPUT 'localhost:9600/_node/logging/reset?pretty'

NOTE: it might be a good idea to start LS with logging set to WARN in the logstash.yml so other logging is less verbose.

When you do the curl -XGET 'localhost:9600/_node/logging?pretty', you will see some components that start with filewatch - you should turn on TRACE logging for those only and only when you notice the file input failing to read new lines (before you restart Logstash).

Post the log lines at TRACE level here, I should be able to see what is going wrong from these. Note: if you turn on TRACE globally there will be too much logging and that makes the fault interpretation much much harder.

1 Like