Logstash not pushing logs to ElasticSearch

PAN-OS_Syslog or PAN-OS_SysLog? It does not appear to match the case you are testing for.

Sorry, my bad.

I changed the cases and tried twice (with lower case and upper case in input and filter). No luck. I even removed the tag from input and filter, doesn't seem to work.

The [message] attribute in the conditional, does it work only with syslog input?

I need a solution to search for the 4 keywords in my log file (.txt).

No, it will work with any input that generates a message field.

Working now. Thank you :slight_smile:

I've noticed that elasticsearch is ingesting files of size 1GB but generating indices of size 4GB. Is there any way to reduce it?

By default there will be a replica of everything, it is possible to disable that but then you risk data loss if a node fails. index_options can be used to control what gets indexed for each document. Indexing less reduces the volume of indexes, but also limits functionality. I have had use cases where 'index_options: docs' worked for me and saved a lot of space.

You might be better off asking a new question in the elasticsearch forum.

Logstash has stopped working again, after ingesting data for 2 days. This is the latest logs recorded.

[2019-07-30T23:59:10,202][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-07-30T23:59:10,202][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
[2019-07-30T23:59:10,202][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>52}

You ran out of disk space. Reduce disk usage on the partition(s) that elasticsearch is writing to below 95% and then follow these instructions.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.