Hi,
I am on elastic stack 7.4.2 and using following pipeline:
log -> filebeat -> redis (TLS via stunnel) -> logstash-> elasticsearch
In logstash I use pipeline to pipeline communication, so that different inputs the same elasticsearch output.
My Logfile is consisting of a json log. json decoding takes place in logstash, not in filebeat.
I have following issue:
Sometimes the beginning of a log line is truncated. The rest of the line is not a valid json, so it is tagged as _jsonparsefailed in logstash.
What I found out so far:
- I am not able to find the beginning of the line as _jsonparsefailed. So the beginning of the line seems to be lost somewhere in the processing pipeline.
- The affected log lines are not the first line in the file (see offset).
My filebeat config looks like this:
filebeat.yml
filebeat.config:
inputs:
enabled: true
path: inputs.d/*.yml
reload.enabled: true
reload.period: 30s
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
fields:
{"log": {"sourceGroup": "staging"}}
fields_under_root: true
setup.kibana:
output.redis:
enabled: true
hosts: ["poc.redis.k8s-dev.local:16379"]
key: "%{[logType]:fallback}"
ssl.enabled: true
ssl.verification_mode: full
ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]
ssl.certificate_authorities: ["/etc/filebeat/ca.crt"]
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
The included inputs.d looks like this:
- type: log
enabled: true
paths:
- /project/jboss/*/*/log/monitoring.json.log*
encoding: windows-1252
fields:
logType: generic-json
log.format: json
fields_under_root: true
max_bytes: 90000000
Questions:
- In the past I needed to increase max_bytes because I've seen the log.flag: truncated in events.
Do I need to increase any other parameter which configures internal filebeat queues, etc? - How to prove that the error is made in filebeat and not in redis or logstash? Can I simultaneously dump the messages which filebeat sends to redis to a file too? Then I could search for incomplete messages in the file and see if they are correct there or truncated at the same position.
- Any Idea why I cannot find the beginning of the messgae? In my understanding I sould see the beginning as _jsonparsedfailed tagged message also in kibana, because this also must be an incomplete json.
- How does filebeat behave if a log line is written in two write operations, because OS or log4j buffer is full? But if that would be the problem that filebeat is splitting a message in this case into 2 or more parts, I should see the first part as incomplete message, correct?
- Any other ideas how to analyse this error?
Thanks a lot,
Andreas
