Hi people! here is the deal, I using k8s, my apps write down thier logs into a json formatted file, so I decided to send this file to the stdout (with a soft link kind of how nginx does it), so now if you check the stdout for my pods, you can see the stdout logs that the pod originally had plus the json file I am sending.
So my approach was to use filebeat as a deamonset and get all the stdout/stderr logs, and I did, I was able to see the logs and to parse the json keys as well. It works as expected, but if I check the logs for filebeat I see that it not only parse the json but also the "text" like logs.
here is a stract of my errors:
022-05-31T03:26:25.911Z ERROR [reader_json] readjson/json.go:74 Error decoding JSON: invalid character 'F' looking for beginning of value
2022-05-31T03:26:25.911Z INFO [input] log/input.go:171 Configured paths: [/var/log/containers/*-77a347fb4e603cae35e6bc5cac54957403bfe26530e61913e6fc51f875825187.log] {"input_id": "ae07a0be-ecf0-4629-abf0-f66e7b3ee88b"}
2022-05-31T03:26:25.911Z ERROR [reader_json] readjson/json.go:74 Error decoding JSON: EOF
2022-05-31T03:26:25.911Z ERROR [reader_json] readjson/json.go:74 Error decoding JSON: invalid character 'P' looking for beginning of value
2022-05-31T03:26:25.911Z ERROR [reader_json] readjson/json.go:74 Error decoding JSON: invalid character 'S' looking for beginning of value
2022-05-31T03:26:25.911Z ERROR [reader_json] readjson/json.go:74 Error decoding JSON: invalid character 'R' looking for beginning of value
2022-05-31T03:26:25.911Z ERROR [reader_json] readjson/json.go:74 Error decoding JSON: invalid character 'E' looking for beginning of value
2022-05-31T03:26:25.911Z ERROR [reader_json] readjson/json.go:74 Error decoding JSON: invalid character '-' in numeric literal
2022-05-31T03:26:25.912Z ERROR [reader_json] readjson/json.go:74 Error decoding JSON: invalid character 'a' in literal null (expecting 'u')
2022-05-31T03:26:25.912Z ERROR [reader_json] readjson/json.go:74 Error decoding JSON: invalid character 's' looking for beginning of value
2022-05-31T03:26:25.912Z ERROR [reader_json] readjson/json.go:74 Error decoding JSON: invalid character 'o' looking for beginning of value
2022-05-31T03:26:25.912Z ERROR [reader_json] readjson/json.go:74 Error decoding JSON: invalid character '-' in numeric literal
2022-05-31T03:26:25.912Z ERROR [reader_json] readjson/json.go:74 Error decoding JSON: invalid character 'j' looking for beginning of value
2022-05-31T03:26:25.912Z ERROR [reader_json] readjson/json.go:74 Error decoding JSON: invalid character 's' after array element
this is the configuration I am using:
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
hints.default_config.enabled: false
templates:
- condition:
and:
- equals:
kubernetes.namespace: filebeat
- equals:
kubernetes.labels.app: compute-group
config:
- type: container
paths:
- /var/log/containers/*-${data.kubernetes.container.id}.log
#exclude_lines: ["^\\s+[\\-`('.|_]"] # drop asciiart lines
#json.message_key: message
fields:
namespace: "${data.kubernetes.namespace}"
logzio_codec: json
type: tmapp
json.keys_under_root: true
json.add_error_key: true
json.overwrite_keys: true
fields_under_root: true
encoding: utf-8
processors:
- drop_fields:
fields: ["beat.name", "beat.hostname", "beat.version"]
- decode_json_fields:
fields: ["message"]
process_array: true
max_depth: 1
#target: ""
overwrite_keys: true
add_error_key: true
- condition:
and:
- equals:
kubernetes.namespace: ingress
- equals:
kubernetes.labels.app: sslproxy
config:
- type: container
paths:
- /var/log/containers/*-${data.kubernetes.container.id}.log
fields:
type: nginx
namespace: "${data.kubernetes.namespace}"
output.elasticsearch:
host: '${NODE_NAME}'
hosts: ['xxxxxxx:9200']
username: "xxxx"
password: "xxxxx"
Again, it works the way I want, I see the json keys indexed in kibana and also the messages for the text like stdout, but the error in the filebeat logs bothers me and I am trying to get rid of it.
Hope anyone can give me a hand. Thanks!!
. There was a fix requested for this problem, I just had to add the json.ignore_decoding_error: true and the errors are gone. I tried several ways mixing different conditions, but adding this key to ignore the error was the only solution. Hope it helps someone else!