Filebeat overwriting log.level and log.logger

Elastic Stack Beats
1

I have my application logger below:

{"@timestamp":"2019-11-12T16:05:05.407636Z","log":{"level":"INFO","logger":"audit"}

However when i run Filebeat in debug mode i notice that my keys are gone and replace with the file pointers:

2019-11-12T16:05:15.305+0100	DEBUG	[processors]	processing/processors.go:183	Publish event: {
  "@timestamp": "2019-11-12T15:05:15.305Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.4.2"
  },
  "log": {
    "offset": 32778,
    "file": {
      "path": "/app/data/log/audit.log"
    }
  },

My filebeat config overall is pretty basic:

#=========================== Filebeat inputs =============================

filebeat.inputs:

- type: log
  enable: true
  paths:
    - /app/data/log/**/*.log
  json.keys_under_root: true
  json.add_error_key: true
  fields:
    application: myapp

#================================ Processors =====================================
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - timestamp:
      field: '@timestamp'
      timezone: CET
      layouts:
        - '2006-01-02T15:04:05Z'
        - '2006-01-02T15:04:05.999Z'
      test:
        - '2019-06-22T16:33:51Z'
        - '2019-11-18T04:59:51.123Z'
2

Hi @Heatzone87,

What's happening is that log input is overwriting your fields with log related info. Perhaps you can try to decode fields under json, and then rename them using the rename processor.

Best regards

3

Yeah i am flexible though so i could push it to a different key however according the Elastic Common Schema this is the way to go?

4

Yes, and you can definitely do that. Is just that in order to get it right you need to do it with the rename processor, after the input has written it's own fields. That should ensure you don't get yours overwritten.

Best regards

5

@exekias i dont understand what you mean i think. The problem is that the filebeat processor is overwriting the key instead of merging it. How will a rename processor help here where actually the values are in log.level and should be there as well?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.