Ndjson.expand_keys doesn't seem to work

[andreycha (Andrei Chasovskikh)]

Hi,

My application produces logs in ECS format where loggers write log level as a flat field "log.level". I use Filebeat 7.17.5 to ship logs into Elasticsearch. As stated in the documentation I set ndjson.expand_keys to true:

type: filestream
parsers:
  - ndjson:
      target: ''
      expand_keys: true
      overwrite_keys: true
      add_error_key: true

However, when log entry arrives to the Elasticsearch I still see that "log.level" is not expanded:

If I add dot_expander processor to the ingest pipeline:

- dot_expander:
    if: ctx['log.level'] != null
    field: 'log.level'
    override: true

Then "log.level" is properly expanded:

So expand_keys doesn't seem to work properly. Or am I understanding/doing something wrong? Thank you.

[TiagoQueiroz]

Hi @andreycha,

Your configuration looks correct, I've tried to reproduce it but did not manage to. So my guess is that there might be some small issue on your configuration, probably some indentation. Try to double check it.

Here is the configuration I used to test (and worked fine):

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - /tmp/log.log
  parsers:
    - ndjson:
        target: ''
        expand_keys: true
        overwrite_keys: true
        add_error_key: true

You can also try some online YAML validator like Best YAML Validator Online.

If you still don't find the cause of the issue, then, please post here your whole input configuration, including the filebeat.inputs bit (just redact the sensitive information, if any).

[andreycha (Andrei Chasovskikh)]

Hi @TiagoQueiroz ,

Thank you for the quick reply. I've checked my configuration using validator -- it's correct. Also Filebeat logs don't have any complains. What I forgot to mention is that I ship logs using application-specific module. Here is a redacted version of fileset input config:

type: filestream
id: my-application-logs
paths:
{{ range $i, $path := .paths }}
  - {{$path}}
{{ end }}
parsers:
  - ndjson:
      target: ''
      expand_keys: true
      overwrite_keys: true
      add_error_key: true
fields_under_root: true
fields:
  ecs.version: 1.12.0
  event.kind: event
  event.category: process
  event.module: my-application
  service.type: invoicetrack
  service.name: my-application

[TiagoQueiroz]

I can't seem to find anything wrong with your config, I even copied and pasted it into a module and it worked for me.

Try looking errors on Filebeat's logs, maybe there it can give us some insights.

Is there other processor/pipelines configured on your module or globally that could also be interacting with those events/fields?

Another thing you can try is to run Filebeat with log debug enabled and look for the Publish event: messages, the whole event sent to Elasticsearch will be there, that way we can see exactly what Filebeat is outputting.

[andreycha (Andrei Chasovskikh)]

While I was adding/testing other stuff, I needed to reinstall/restart Filebeat and now when I'm looking at the logs, they look good, i.e. keys are expanded. Might be that Filebeat somehow didn't pickup module changes (although we have reload enabled). Anyhow, the issue is not actual anymore. Thanks for you support, @TiagoQueiroz !