Not able to convert json string to json object with filebeat TCP input

[vaseem (vaseem akram)]

Hi All,

I am sending json string(For example {"key1": "value1", "key2":"value2"}) to filebeat->elasticsearch->kibana using TCP input in filebeat. However I am getting json string under message field on kibana/elasticsearch (for example "message": {"key1": "value1", "key2":"value2"}.

Note I am sending 1 json string(example given above) at a time with TCP stream

I have tried various option(and combination) in filebeat but nothing worked so far.

1. • type: tcp
     host: "XXXXXX"
     enabled: true
     processors:
   • decode_json_fields:
     fields: ["message"]
     process_array: true
     max_depth: 1
     target: ""
     overwrite_keys: false
     add_error_key: true

2. json.keys_under_root: true
   json.add_error_key: true

3. In elasticsearch output in filebeat.yml
   codec.json:
   pretty: true
   escape_html: false

Could anyone please help me to solve this issue.

[kvch (Noémi Ványi)]

Could you please format your configuration using </>? Also, share the debug logs of Filebeat (./filebeat -e -d "*").

[vaseem (vaseem akram)]

I was able to resolve this by moving processor in global section

processors:

• add_host_metadata: ~
• add_cloud_metadata: ~
• decode_json_fields:
  fields: ["message"]
  process_array: false
  max_depth: 1
  target: ""
  overwrite_keys: true
  add_error_key: true
• drop_fields:
  fields: ["message","ecs","log","input","host","os","agent"]

[vaseem (vaseem akram)]

However, the filebeat output file did not set the 777 permission.

below is the configuration:

output.file:
path: "/home/opc/log"
filename: filebeat
permissions: 0777
rotate_every_kb: 13312
number_of_files: 2

Note I was able to set other permission like 0600, etc(but maximum permission i was able to set is 755).
Below is the snapshot of files:
-rwxr-xr-x. 1 root root 4210956 Nov 26 06:39 filebeat
-rwxr-xr-x. 1 root root 13631350 Nov 26 06:39 filebeat.1
-rwxr-xr-x. 1 root root 13631350 Nov 26 06:39 filebeat.2

Do you have any idea/inputs whether the filebeat behavior is correct or not? Because, in this case logstash is not able to delete file after reading the same as it was created by root and persmission for other and group user is R+X