Filebeat unable to parse JSON log(contains array) output to logstash

Elastic Stack Beats
1

My current usage Filebeat => Logstash => Elasticsearch.
Below is the error log

Oct 14 14:07:21 vault-audit filebeat[11314]: 2020-10-13T14:07:21.055Z        INFO        [publisher]        pipeline/retry.go:217          done
Oct 14 14:07:21 vault-audit filebeat[11314]: 2020-10-13T14:07:21.055Z        INFO        [publisher]        pipeline/retry.go:213        retryer: send wait signal to consumer
Oct 14 14:07:21 vault-audit filebeat[11314]: 2020-10-13T14:07:21.055Z        INFO        [publisher]        pipeline/retry.go:217          done
Oct 14 14:07:21 vault-audit filebeat[11314]: 2020-10-13T14:07:21.095Z        ERROR        [logstash]        logstash/async.go:280        Failed to publish events caused by: EOF
Oct 14 14:07:21 vault-audit filebeat[11314]: 2020-10-13T14:07:21.095Z        ERROR        [logstash]        logstash/async.go:280        Failed to publish events caused by: EOF
Oct 14 14:07:21 vault-audit filebeat[11314]: 2020-10-13T14:07:21.096Z        ERROR        [logstash]        logstash/async.go:280        Failed to publish events caused by: EOF
Oct 14 14:07:21 vault-audit filebeat[11314]: 2020-10-13T14:07:21.096Z        INFO        [publisher]        pipeline/retry.go:213        retryer: send wait signal to consumer
Oct 14 14:07:21 vault-audit filebeat[11314]: 2020-10-13T14:07:21.096Z        INFO        [publisher]        pipeline/retry.go:217          done
Oct 14 14:07:21 vault-audit filebeat[11314]: 2020-10-13T14:07:21.096Z        ERROR        [logstash]        logstash/async.go:280        Failed to publish events caused by: client is not connected

Filebeat configuration File:

filebeat.inputs:
- type: log
  fields:
    index_env: staging
    index_name: vault
  json.keys_under_root: true
  json.overwrite_keys: false
  json.add_error_key: true
  paths:
    - /var/log/vault_audit.log
encoding: utf-8
ignore_older: 3h
#logging.level: debug
#output.file:
#  path: "/tmp/filebeat"
#  filename: filebeat.json
output.logstash:
  hosts: ["logstash.stagin.com:5044"]
  ssl.enabled: true

Logstash Configuration:

  input {
  beats {
    port => 5044
    host => "<PRIVATE_IP>"
    client_inactivity_timeout => 3000
  }
}

Below is the vault audit logs that are loading w/ Filebeat

{"time":"2020-10-14T17:01:09.100112447Z","type":"response","auth":{"token_type":"default"},"request":{"id":"5f9b4d89-d4a6-5692-b161-71356090e8bb","replication_cluster":"tvx-ch","operation":"update","mount_type":"approle","client_token":"hmac-sha256:07be5ba8f9125f92403254de7060a3dd3a21d1466bf2d64860d12be66f7fd4fc","client_token_accessor":"hmac-sha256:4a55985e0d0f8c4be17aa1129e631d02cede0cd747b0d86ebccb4f882b3684c4","namespace":{"id":"root"},"path":"auth/approle/role/modesto_xmobile/secret-id","remote_address":"172.27.155.125"},"response":{"mount_type":"approld","data":{"secret_id":"hmac-sha256:1b5fa0984871b998d3e13c9397eeaa5e493e55c3964da05a1212d3611077ba1f","secret_id_accessor":"hmac-sha256:655e120622f49a069fbd969f993d038935225f09c293822acedb3893a619a0f2"}}}

{"time":"2020-10-14T17:58:05.120449646Z","type":"request","auth":{"token_type":"default"},"request":{"id":"c87af493-d126-7057-2e96-6abf4a9caefc","replication_cluster":"tvg-ab","operation":"update","mount_type":"generic","client_token":"hmac-sha256:eff8db6b9bfabd2a4df87938c5313c81620e5dc7df5fe8ea45f5844ad89g0c88","client_token_accessor":"hmac-sha256:1ff9fd760aad2fee503cb98c0bf892dfc5eebd5b300553643ec2704c792d4fbc","namespace":{"id":"root"},"path":"secret/directoryservices/inc-residential-service/bbc-security-authserver-config-esp-prd-r3","data":{"xps.security.authServer.configs[0].endpointPatterns":"hmac-sha256:a5c1f6402142e7df0e3d1c5940f981906e6cf9eb8a996113bd3c92c3b9dc6e59","xps.security.authServer.configs[0].scopes":"hmac-sha256:e7326a67dcf53b1f744a856be4a73f380b32c3897e786af3379eda2f541debd5","xps.security.authServer.configs[0].secret":"hmac-sha256:ab5c4b32c03dd877dae0a7b0bda89d9979b6d30bd8d9a95e5b194f342c7c0a2c","xps.security.authServer.configs[0].username":"hmac-sha256:c86dddd51966d4a77967724de1c43274a65d0b7354c72d8f2b2b19f42663c324","xps.security.authServer.configs[1].endpointPatterns":"hmac-sha256:bc599e6b16e57890e4247068c1cfdc7b9d1aed76313a76b828c158e3faf105b4","xps.security.authServer.configs[1].scopes":"hmac-sha256:fcc4acf36658b90fec62f9dddae83042a4ae9b430674a541a131a5d522771236","xps.security.authServer.configs[1].secret":"hmac-sha256:ab5c4b32c03dd877dae0a7b0bda89d9979b6d30bd8d9a95e5b194f342c7c0a2c","xps.security.authServer.configs[1].username":"hmac-sha256:c86dddd51966d4a77967724de1c43274a65d0b7354c72d8f2b2b19f42663c324","xps.security.authServer.configs[2].endpointPatterns":"hmac-sha256:8efc977777036af1e63cd931b6ec1b5a9db32a5af8cd86fd944d9a5833919ed1","xps.security.authServer.configs[2].scopes":"hmac-sha256:e9e34d58a7d7611fb56c7b3958265f512098efeb33f6282601110121359bfc92","xps.security.authServer.configs[2].secret":"hmac-sha256:ab5c4b32c03dd877dae0a7b0bda89d9979b6d30bd8d9a95e5b194f342c7c0a2c","xps.security.authServer.configs[2].username":"hmac-sha256:c86dddd51966d4a77967724de1c43274a65d0b7354c72d8f2b2b19f42663c324"},"remote_address":"127.0.0.1"}}

from the above both log lines, the filebeat was unable to load the 2nd log on to logstash but when I output to logging.files: or elasticsearch then I don't see any errors on the filebeat.

Please I need some light on solving this issue ASAP.
Thanks.
@adrisr @Badger (tagging since you have solved similar issue: Filebeat parse json)

2

The errors you are seeing have nothing to do with JSON parsing. Filebeat is not able to connect to Logstash. Based on your configuration you have enabled SSL but failed to configure it correctly. You need to configure ssl.* settings if you would like to secure the connection between LS and FB: https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ssl-logstash.html

3

@kvch Thanks for your response. I'm using AWS Loadbalancer(NLB). am seeing a successful connection test when I run tcptraceroute or telnet commands. so after removing host => "PRIVATE_IP" from the Logstash configuration, I started seeing the below errors in logstash.

Oct 21 10:41:14 ip-96-103-60-114.ec2.internal logstash[3260]: org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `xsp.security.authServer.configs[0].secret`
Oct 21 10:41:16 ip-96-103-60-114.ec2.internal logstash[3260]: org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `xsp.security.authServer.configs[0].username`

I might need help in getting the filters. It looks like a pretty similar issue opened on github: https://github.com/elastic/logstash/issues/11608

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.