# Filebeat unable to parse JSON log(contains array) output to logstash
**URL:** https://discuss.elastic.co/t/filebeat-unable-to-parse-json-log-contains-array-output-to-logstash/252107
**Category:** Beats
**Tags:** filebeat
**Created:** [October 14, 2020, 9:31pm UTC](https://discuss.elastic.co/t/filebeat-unable-to-parse-json-log-contains-array-output-to-logstash/252107 "2020-10-14T21:31:09Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@polaaditya](https://discuss.elastic.co/u/polaaditya)
#### Post date: [October 14, 2020, 9:31pm UTC](https://discuss.elastic.co/t/filebeat-unable-to-parse-json-log-contains-array-output-to-logstash/252107/1 "2020-10-14T21:31:09Z")
My current usage Filebeat =\> Logstash =\> Elasticsearch.
Below is the error log
```auto
Oct 14 14:07:21 vault-audit filebeat[11314]: 2020-10-13T14:07:21.055Z INFO [publisher] pipeline/retry.go:217 done
Oct 14 14:07:21 vault-audit filebeat[11314]: 2020-10-13T14:07:21.055Z INFO [publisher] pipeline/retry.go:213 retryer: send wait signal to consumer
Oct 14 14:07:21 vault-audit filebeat[11314]: 2020-10-13T14:07:21.055Z INFO [publisher] pipeline/retry.go:217 done
Oct 14 14:07:21 vault-audit filebeat[11314]: 2020-10-13T14:07:21.095Z ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: EOF
Oct 14 14:07:21 vault-audit filebeat[11314]: 2020-10-13T14:07:21.095Z ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: EOF
Oct 14 14:07:21 vault-audit filebeat[11314]: 2020-10-13T14:07:21.096Z ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: EOF
Oct 14 14:07:21 vault-audit filebeat[11314]: 2020-10-13T14:07:21.096Z INFO [publisher] pipeline/retry.go:213 retryer: send wait signal to consumer
Oct 14 14:07:21 vault-audit filebeat[11314]: 2020-10-13T14:07:21.096Z INFO [publisher] pipeline/retry.go:217 done
Oct 14 14:07:21 vault-audit filebeat[11314]: 2020-10-13T14:07:21.096Z ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: client is not connected
```
Filebeat configuration File:
```auto
filebeat.inputs:
- type: log
fields:
index_env: staging
index_name: vault
json.keys_under_root: true
json.overwrite_keys: false
json.add_error_key: true
paths:
- /var/log/vault_audit.log
encoding: utf-8
ignore_older: 3h
#logging.level: debug
#output.file:
# path: "/tmp/filebeat"
# filename: filebeat.json
output.logstash:
hosts: ["logstash.stagin.com:5044"]
ssl.enabled: true
```
Logstash Configuration:
```auto
input {
beats {
port => 5044
host => ""
client_inactivity_timeout => 3000
}
}
```
Below is the vault audit logs that are loading w/ Filebeat
```auto
{"time":"2020-10-14T17:01:09.100112447Z","type":"response","auth":{"token_type":"default"},"request":{"id":"5f9b4d89-d4a6-5692-b161-71356090e8bb","replication_cluster":"tvx-ch","operation":"update","mount_type":"approle","client_token":"hmac-sha256:07be5ba8f9125f92403254de7060a3dd3a21d1466bf2d64860d12be66f7fd4fc","client_token_accessor":"hmac-sha256:4a55985e0d0f8c4be17aa1129e631d02cede0cd747b0d86ebccb4f882b3684c4","namespace":{"id":"root"},"path":"auth/approle/role/modesto_xmobile/secret-id","remote_address":"172.27.155.125"},"response":{"mount_type":"approld","data":{"secret_id":"hmac-sha256:1b5fa0984871b998d3e13c9397eeaa5e493e55c3964da05a1212d3611077ba1f","secret_id_accessor":"hmac-sha256:655e120622f49a069fbd969f993d038935225f09c293822acedb3893a619a0f2"}}}
```
```auto
{"time":"2020-10-14T17:58:05.120449646Z","type":"request","auth":{"token_type":"default"},"request":{"id":"c87af493-d126-7057-2e96-6abf4a9caefc","replication_cluster":"tvg-ab","operation":"update","mount_type":"generic","client_token":"hmac-sha256:eff8db6b9bfabd2a4df87938c5313c81620e5dc7df5fe8ea45f5844ad89g0c88","client_token_accessor":"hmac-sha256:1ff9fd760aad2fee503cb98c0bf892dfc5eebd5b300553643ec2704c792d4fbc","namespace":{"id":"root"},"path":"secret/directoryservices/inc-residential-service/bbc-security-authserver-config-esp-prd-r3","data":{"xps.security.authServer.configs[0].endpointPatterns":"hmac-sha256:a5c1f6402142e7df0e3d1c5940f981906e6cf9eb8a996113bd3c92c3b9dc6e59","xps.security.authServer.configs[0].scopes":"hmac-sha256:e7326a67dcf53b1f744a856be4a73f380b32c3897e786af3379eda2f541debd5","xps.security.authServer.configs[0].secret":"hmac-sha256:ab5c4b32c03dd877dae0a7b0bda89d9979b6d30bd8d9a95e5b194f342c7c0a2c","xps.security.authServer.configs[0].username":"hmac-sha256:c86dddd51966d4a77967724de1c43274a65d0b7354c72d8f2b2b19f42663c324","xps.security.authServer.configs[1].endpointPatterns":"hmac-sha256:bc599e6b16e57890e4247068c1cfdc7b9d1aed76313a76b828c158e3faf105b4","xps.security.authServer.configs[1].scopes":"hmac-sha256:fcc4acf36658b90fec62f9dddae83042a4ae9b430674a541a131a5d522771236","xps.security.authServer.configs[1].secret":"hmac-sha256:ab5c4b32c03dd877dae0a7b0bda89d9979b6d30bd8d9a95e5b194f342c7c0a2c","xps.security.authServer.configs[1].username":"hmac-sha256:c86dddd51966d4a77967724de1c43274a65d0b7354c72d8f2b2b19f42663c324","xps.security.authServer.configs[2].endpointPatterns":"hmac-sha256:8efc977777036af1e63cd931b6ec1b5a9db32a5af8cd86fd944d9a5833919ed1","xps.security.authServer.configs[2].scopes":"hmac-sha256:e9e34d58a7d7611fb56c7b3958265f512098efeb33f6282601110121359bfc92","xps.security.authServer.configs[2].secret":"hmac-sha256:ab5c4b32c03dd877dae0a7b0bda89d9979b6d30bd8d9a95e5b194f342c7c0a2c","xps.security.authServer.configs[2].username":"hmac-sha256:c86dddd51966d4a77967724de1c43274a65d0b7354c72d8f2b2b19f42663c324"},"remote_address":"127.0.0.1"}}
```
from the above both log lines, the filebeat was unable to load the 2nd log on to logstash but when I output to `logging.files:` or `elasticsearch` then I don't see any errors on the filebeat.
Please I need some light on solving this issue ASAP.
Thanks.
@adrisr @Badger (tagging since you have solved similar issue: [Filebeat parse json](https://discuss.elastic.co/t/filebeat-parse-json/130834))
---
### Author:  [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [October 21, 2020, 1:13pm UTC](https://discuss.elastic.co/t/filebeat-unable-to-parse-json-log-contains-array-output-to-logstash/252107/2 "2020-10-21T13:13:58Z")
The errors you are seeing have nothing to do with JSON parsing. Filebeat is not able to connect to Logstash. Based on your configuration you have enabled SSL but failed to configure it correctly. You need to configure `ssl.*` settings if you would like to secure the connection between LS and FB: [https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ssl-logstash.html](https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ssl-logstash.html)
---
### Author:  [@polaaditya](https://discuss.elastic.co/u/polaaditya)
#### Post date: [October 21, 2020, 2:45pm UTC](https://discuss.elastic.co/t/filebeat-unable-to-parse-json-log-contains-array-output-to-logstash/252107/3 "2020-10-21T14:45:28Z")
@kvch Thanks for your response. I'm using AWS Loadbalancer(NLB). am seeing a successful connection test when I run `tcptraceroute` or `telnet` commands. so after removing `host => "PRIVATE_IP"` from the Logstash configuration, I started seeing the below errors in logstash.
```auto
Oct 21 10:41:14 ip-96-103-60-114.ec2.internal logstash[3260]: org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `xsp.security.authServer.configs[0].secret`
Oct 21 10:41:16 ip-96-103-60-114.ec2.internal logstash[3260]: org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `xsp.security.authServer.configs[0].username`
```
I might need help in getting the filters. It looks like a pretty similar issue opened on github: [https://github.com/elastic/logstash/issues/11608](https://github.com/elastic/logstash/issues/11608)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 18, 2020, 4:45pm UTC](https://discuss.elastic.co/t/filebeat-unable-to-parse-json-log-contains-array-output-to-logstash/252107/4 "2020-11-18T16:45:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.