Best practices for "debugging" and problem solving logstash problems

Hello,
I'm new to logstash and testing
filebeat -> logstash -> elasticsearch -> kibana
for viewing log data.

I have managed to use logstash to parse and format log formats,
but I am worried that some log lines may be in a format that I did not consider and my parser will miss and not store in elasticsearch.

How can I find out if some of my log lines are not matching and not processed with the grok patterns I am using?

If the mapping of the some log data and elasticsearch doesn't match, how would I find out and view or log such errors?

I noticed the --debug command line option gives more information.
What is the --debug equivalent in logstash.yml?

Other than the --debug command line option, logs stored by logstash in /var/log/* what other useful tips for debugging and problem solving can you suggest?

Thank you.

How can I find out if some of my log lines are not matching and not processed with the grok patterns I am using?

They'll get tagged with _grokparsefailure.

If the mapping of the some log data and elasticsearch doesn't match, how would I find out and view or log such errors?

Check out the dead letter queue feature.

I noticed the --debug command line option gives more information.
What is the --debug equivalent in logstash.yml?

log.level: debug

Other than the --debug command line option, logs stored by logstash in /var/log/* what other useful tips for debugging and problem solving can you suggest?

I'm obviously biased but I find my testing tool Logstash Filter Verifier to be very useful when developing and debugging filters.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.