Rename field with filebeat

Elastic Stack Beats
1

I am trying to rename non json field with filebeat but json field also getting renamed. Not sure what i am missing. Can some budy help me?

Here is my config.

setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1
  index.mapping.ignore_malformed: true
output.elasticsearch:
  hosts: ["localhost:9200"]
  indices:
    - index: "%{[fields.env]}-%{[fields.app]}-%{[fields.name]}-%{[fields.type]}-%{+yyyy.MM.dd}"

filebeat.inputs:
  ### API Trace
  - type: log
    fields:
      env: hello
      app: hi
      name: a
      type: trace
    paths:
      - /var/log/json/*.json
    json.ignore_decoding_error: true
    json.keys_under_root: true
    json.add_error_keys: true
    processors:
      - rename:
          fields:
           - from: "context"
             to: "context_str"
          ignore_missing: false
          fail_on_error: true
          when:
             not:
               equals:
                 regexp:
                   context: '^\{\}$'
      - decode_json_fields:
          fields: ["message", "context", "input", "body", "response", "content", "request"]
          process_array: true
          max_depth: 15
2

Do you have some sample input event?

3

context field with Json: { "@timestamp":"2019-04-23T16:33:16.045Z", "context":{"test1":"test1", "test":"test"} }
context field with nonjson: { "@timestamp":"2019-04-23T16:33:16.045Z", "context":["test|test|test|test|test"] }

4

there will be a rename processor in Beats. See https://www.elastic.co/guide/en/beats/filebeat/master/rename-fields.html .

But until that is released you need to use either Logstash or Ingest Node to rename tellpopeyes fields.

5

@Raju_Gupta, Kindly provide the output you are getting during event publish. Which version of filebeat are you using?

6

@Phillip7631, From filebeat version 6.3 "rename" processor is already available.

7

The context field is a valid JSON object. You can not capture it as a string and rename, as it's already parsed. JSON support in filebeat parses the complete document as is. Reading your config I guess you assume all top-level fields still to be strings. This is not the case.

The regexp.context filter can not succeed, cause regexp condition looks for a string or an array of strings, yet you present it either an object or and array.

Your options are (one of):

  • Drop the context field, so indexing works
  • Create an Ingest Node pipeline with painless script to process the events. JSON can be parsed either in Beats or Ingest Node
  • Enforce some stronger types/schema for the context fields
1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.