Not able to convert json string to json object with filebeat TCP input

Hi All,

I am sending json string(For example {"key1": "value1", "key2":"value2"}) to filebeat->elasticsearch->kibana using TCP input in filebeat. However I am getting json string under message field on kibana/elasticsearch (for example "message": {"key1": "value1", "key2":"value2"}.

Note I am sending 1 json string(example given above) at a time with TCP stream

I have tried various option(and combination) in filebeat but nothing worked so far.

    • type: tcp
      host: "XXXXXX"
      enabled: true
      processors:
    • decode_json_fields:
      fields: ["message"]
      process_array: true
      max_depth: 1
      target: ""
      overwrite_keys: false
      add_error_key: true
  1. json.keys_under_root: true
    json.add_error_key: true

  2. In elasticsearch output in filebeat.yml
    codec.json:
    pretty: true
    escape_html: false

Could anyone please help me to solve this issue.

Could you please format your configuration using </>? Also, share the debug logs of Filebeat (./filebeat -e -d "*").

I was able to resolve this by moving processor in global section

processors:

  • add_host_metadata: ~
  • add_cloud_metadata: ~
  • decode_json_fields:
    fields: ["message"]
    process_array: false
    max_depth: 1
    target: ""
    overwrite_keys: true
    add_error_key: true
  • drop_fields:
    fields: ["message","ecs","log","input","host","os","agent"]

However, the filebeat output file did not set the 777 permission.

below is the configuration:

output.file:
path: "/home/opc/log"
filename: filebeat
permissions: 0777
rotate_every_kb: 13312
number_of_files: 2

Note I was able to set other permission like 0600, etc(but maximum permission i was able to set is 755).
Below is the snapshot of files:
-rwxr-xr-x. 1 root root 4210956 Nov 26 06:39 filebeat
-rwxr-xr-x. 1 root root 13631350 Nov 26 06:39 filebeat.1
-rwxr-xr-x. 1 root root 13631350 Nov 26 06:39 filebeat.2

Do you have any idea/inputs whether the filebeat behavior is correct or not? Because, in this case logstash is not able to delete file after reading the same as it was created by root and persmission for other and group user is R+X