I have set my log dir in logstash.yml as follows:
path.logs: /var/log/logstash
However, important messages like the following are going to /var/log/messages, i.e. the system log.
Mar 3 03:36:33 elklog01 logstash[6342]: [2019-03-03T03:36:33,646][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
How do I control what goes into system logs?
When a message is written to system log, the timestamp should be communicated to Syslog in its format based on the protocol and not be part of the message itself. The current way of prepending timestamp at the beginning of the message creates confusion for any subsystem consuming logs.