Best practices for "debugging" and problem solving logstash problems

Elastic Stack Logstash
1

Hello,
I'm new to logstash and testing
filebeat -> logstash -> elasticsearch -> kibana
for viewing log data.

I have managed to use logstash to parse and format log formats,
but I am worried that some log lines may be in a format that I did not consider and my parser will miss and not store in elasticsearch.

How can I find out if some of my log lines are not matching and not processed with the grok patterns I am using?

If the mapping of the some log data and elasticsearch doesn't match, how would I find out and view or log such errors?

I noticed the --debug command line option gives more information.
What is the --debug equivalent in logstash.yml?

Other than the --debug command line option, logs stored by logstash in /var/log/* what other useful tips for debugging and problem solving can you suggest?

Thank you.

2

How can I find out if some of my log lines are not matching and not processed with the grok patterns I am using?

They'll get tagged with _grokparsefailure.

If the mapping of the some log data and elasticsearch doesn't match, how would I find out and view or log such errors?

Check out the dead letter queue feature.

I noticed the --debug command line option gives more information.
What is the --debug equivalent in logstash.yml?

log.level: debug

Other than the --debug command line option, logs stored by logstash in /var/log/* what other useful tips for debugging and problem solving can you suggest?

I'm obviously biased but I find my testing tool Logstash Filter Verifier to be very useful when developing and debugging filters.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.