[logstash] Cannot find the logs that donot match the grok filter

Elastic Stack Logstash
1

Hello, dear all,

I updated the logstash from vesrion 2.1 to 6.4, and I find there are some differences between them which caused my problem.

  1. I cannot find any logs that don't match my grok filters in the kibana for version 6.4, but for the older logstash, I can find them with the tags: "_grokparsefailure ".

  2. I also can't see any error in the /var/log/logstash/logstash-plain.log for the new version logstash.
    And I can find it many _grokparsefailure in the /var/log/logstash/logstatsh.log for the old version logstash.

And for me, I need to save all the logs in the elasticsearch, include the parse error log.

How can I do my job for losgtash v6.4?

Any advice is very appreciated, thanks in advance.

Lyndon

2

I had found a way to resolve this issue -- add a filter that can match each log.
But it still cannot be seen in the elasticsearch.

And in the end, I found that, I need to extract a field with the type date, or it cann't be indexed to "logstash-YY-MM-DD.log", that's why the log cannot be see in the kibana.

3

And I am wondering why in the logstash of 2.1 , it can use the default field @timestamp as the elasticsearch index but logstash v6.4 cannot do it as this way.

If anybody knows why, please let me know.

Thanks very much.