Hi there,
I am new in participating in this board. As a long-time reader I found a lot of stuff here that helps my to keep running a few ECK instances.
Today I am facing a huge problem in migrating filebeat from 8.17.0 to 9.0.0.
I have a well running autodiscover config with conditions to splitt pod logs, which are allready migrated to json logging and those who don´t, based on pod label.
After changing the config to the now mandatory "filestream" type, the json log line did not get recognized as json, because something in filebeat adds an additional timestamp and the words "stdout F" to every line.
In Kibana i found this under the message tag:
2025-05-27T09:31:43.261529225Z stdout F {"@timestamp":"2025-05-27T09:31:43.261Z","log.level":"DEBUG","message":"Successfully requested 0 events at plant","ecs.version": "1.2.0","process.thread.name":"http-nio-8263-exec-7","log.logger":"xxxxxxx","log":{"origin":{"file":{"name":"xxxx.kt","line":35},"function":"get"}}}
The origin log line is valid json, with the old config everything worked well.
The changed autodiscover part from the config for filebeat 9.0.0:
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
hints.default_config.enabled: false
templates:
- condition.equals:
kubernetes.labels.log-format: "ecs-json"
config:
- type: filestream
id: container-${data.kubernetes.container.id}
prospector.scanner.symlinks: true
paths:
- "/var/log/containers/*-${data.kubernetes.container.id}.log"
parsers:
- ndjson:
keys_under_root: true
expand_keys: true
add_error_key: true
overwrite_keys: true
- condition.not.contains:
kubernetes.labels.log-format: "ecs-json"
config:
- type: filestream
id: container-${data.kubernetes.container.id}
prospector.scanner.symlinks: true
paths:
- "/var/log/containers/*-${data.kubernetes.container.id}.log"
parsers:
- multiline:
pattern: '^[[:space:]]+(at|\.{3})[[:space:]]+\b|^Caused by:'
negate: false
match: after
parsers:
- container:
format: docker
Any suggestions what happen there?
Regards
Dirk
Hi @DirkGerlach
You're missing the container parser in one of your Filestream configurations. Adding it as the first parser should fix the problem.
Hi @TiagoQueiroz,
Thank you for your help.
Unfortunately this changes didn´t solve the problem.
I added the container parser (Like I found it in the filestream migration doc):
paths:
- "/var/log/containers/*-${data.kubernetes.container.id}.log"
parsers:
- container: ~
parsers:
- ndjson:
keys_under_root: true
expand_keys: true
add_error_key: true
overwrite_keys: true
but nothing changed. The log lines still had the trailing timestamp and text.
After adding a third parser like:
paths:
- "/var/log/containers/*-${data.kubernetes.container.id}.log"
parsers:
- container: ~
parsers:
- ndjson:
keys_under_root: true
expand_keys: true
add_error_key: true
overwrite_keys: true
parsers:
- container: ~
I got rid of the trailing timestamp and text but I guess, only at the last line before it got transfered to elastic. The json parser still gives an error messages:
{"log.level":"error","@timestamp":"2025-05-28T11:00:01.282Z","log.logger":"parser_container","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/reader/readjson.(*DockerJSONReader).Next","file.name":"readjson/docker_json.go","file.line":231},"message":"Parse line error: decoding docker JSON: json: cannot unmarshal number into Go value of type readjson.logLine","service.name":"filebeat","ecs.version":"1.6.0"}
Switching back to the old config and Filebeat 8.17 the same log lines get parsed correctly.
So my guess is there is something different with the "ndjson" parser or Filebeat 9.
Regards
Dirk
There must be a single parsers key, it is an array of objects and the parsers will be executed in the order they're defined. You also need to indent the keys for each parser, so the correct form is:
parsers:
- container: ~
- ndjson:
keys_under_root: true
expand_keys: true
add_error_key: true
overwrite_keys: true
In our docs there is an example that uses the ndjson and multiline parser together.
Here is the example from our docs:
filebeat.inputs:
- type: filestream
...
parsers:
- ndjson:
target: ""
message_key: msg
- multiline:
type: count
count_lines: 3
I looked briefly to the Filestream migration guide and did not find any instance where multiple parsers key is defined for the same input 
. So the docs look correct, however they do not have an example of multiple parsers on the same input.
Hi @TiagoQueiroz,
thank you for your answer. Sorry for my late response, vacation leave 
With referring to the migration guide I was pointing to the syntax:
- container: ~
The probably short version with the ~
I found no examples in the guides with more than one parser.
Unfortunately the config with double parser still not work.
The json log is still not recognized by the "ndjson" part.
My time for this type of trial and error is limited.
I will try this again occasionally between the next projects.
Thank you for your help.
I will stay with 8.17 so far.
I found the error.
During C&P from Windows to Linux there were a few "\n" at the end of the lines in the [ndjson] part of my config.
I write it new and now the json logs get parsed correctly.
Thanks for the support