I am facing issues while ingesting logs to logstash using filebeat. both of filebeat and logstash versions are 7.5.2 and please find the details below
filebeat config
filebeat.inputs:
- type: log
enabled: true
encoding: utf-8
ignore_older: 3h
json.keys_under_root: true
json.add_error_key: true
fields:
index_env: prod
index_name: vault
index_dc: ho
paths:
- /var/log/vault_audit.log
output.logstash:
enabled: true
hosts: ["<IP ADDRESS>:5045"]
actual log from /var/log/vault_audit.log: which is causing the error, below is the log with new lines added for convenience
{
"time": "2020-05-12T13:21:27.468042577Z",
"type": "response",
"auth": {
"client_token": "hmac-sha256:21549fef0d44b7b880b021e4d80b966bc59383393a9cf1695a8d849602e7e19d",
"accessor": "hmac-sha256:6f5cab54378cb18d10f067763dca36215d150e5198939b24ac6c20349a5f0c82",
"display_name": "github_abc-def-provisioning",
"policies": ["abc", "default", "admin-configuration_vault"],
"token_policies": ["abc", "default", "admin-configuration_vault"],
"metadata": {
"org": "ABC-Provisioning-Configuration-DEF",
"username": "admin123"
},
"entity_id": "21549fe-fef0-e7e19d-9ec9-f1695a8d8496",
"token_type": "service"
},
"request": {
"id": "7d50810b-e7dd-5430-1c85-ba496e98c1ed",
"operation": "read",
"client_token": "hmac-sha256:2cab76cb0ba20f1c2ecc60241c02d4cc5f7593b36bd0b120b7a153f2695a1258",
"client_token_accessor": "hmac-sha256:2a45f24b436b93f5a5a85ad2f0fff0c1500f6d18c6dd7e67ff303d8b4d89d76a",
"namespace": {
"id": "root"
},
"path": "secret/abc-def-provisioning-configuration/my-own-service/prd-r2",
"remote_address": "127.0.0.1"
},
"response": {
"secret": {},
"data": {
"vault.npp.authServer.publicKey": "hmac-sha256:15a36c8272ca984212df5f3fed206ac53cefdc63d359fbb8d3d66b16d965351f",
"vault.npp.soapclient.def.customerServiceClient.wss4j.password": "hmac-sha256:707d59c031e29e29e8cc8fa93946b828989f4d56ce031056fbad5249aaa862e7",
"vault.npp.soapclient.def.customerServiceClient.wss4j.username": "hmac-sha256:1bb69b75bc1667d2809261c41e481bedf8b566fe41404ed3c3c8cc194ee39699",
"vault.npp.xsp.security.authServer.configs[0].secret": "hmac-sha256:1bb69b75bc1667d2809261c41e481bedf8b566fe41404ed3c3c8cc194ee39699",
"vault.npp.xsp.security.authServer.configs[0].username": "hmac-sha256:707d59c031e29e29e8cc8fa93946b828989f4d56ce031056fbad5249aaa862e7",
"vault.npp.xsp.security.authServer.configs[1].secret": "hmac-sha256:707d59c031e29e29e8cc8fa93946b828989f4d56ce031056fbad5249aaa862e7",
"vault.npp.xsp.security.authServer.configs[1].username": "hmac-sha256:707d59c031e29e29e8cc8fa93946b828989f4d56ce031056fbad5249aaa862e7",
"vault.npp.xsp.security.authServer.configs[2].secret": "hmac-sha256:1bb69b75bc1667d2809261c41e481bedf8b566fe41404ed3c3c8cc194ee39699",
"vault.npp.xsp.security.authServer.configs[2].username": "hmac-sha256:707d59c031e29e29e8cc8fa93946b828989f4d56ce031056fbad5249aaa862e7"
}
}
}
and error from filebeat is below
|2020-05-19T15:25:56.543Z|ERROR|[logstash]|logstash/async.go:279|Failed to publish events caused by: read tcp <IP>:34870-><IP>:5045: read: connection reset by peer|
|---|---|---|---|---|
|2020-05-19T15:25:56.543Z|ERROR|[logstash]|logstash/async.go:279|Failed to publish events caused by: read tcp <IP>:34870-><IP>: read: connection reset by peer|
|2020-05-19T15:25:56.543Z|ERROR|[logstash]|logstash/async.go:279|Failed to publish events caused by: read tcp <IP>:34870-><IP>: read: connection reset by peer|
error from the logstash is below
[INFO ] 2020-05-19 15:25:56.523 [defaultEventExecutorGroup-4-3] BeatsHandler - [local: 96.112.245.149:5045, remote: <IP>:34870] Handling exception: Invalid FieldReference: `vault.npp.xsp.security.authServer.configs[0].username`
[WARN ] 2020-05-19 15:25:56.531 [nioEventLoopGroup-2-4] DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `vault.npp.xsp.security.authServer.configs[0].username`
at org.logstash.FieldReference$StrictTokenizer.tokenize(FieldReference.java:303) ~[logstash-core.jar:?]
at org.logstash.FieldReference.parse(FieldReference.java:204) ~[logstash-core.jar:?]
at org.logstash.FieldReference.parseToCache(FieldReference.java:195) ~[logstash-core.jar:?]
at org.logstash.FieldReference.from(FieldReference.java:127) ~[logstash-core.jar:?]
at org.logstash.ConvertedMap.put(ConvertedMap.java:95) ~[logstash-core.jar:?]
at org.logstash.ConvertedMap.newFromMap(ConvertedMap.java:75) ~[logstash-core.jar:?]
at org.logstash.Valuefier.lambda$initConverters$13(Valuefier.java:172) ~[logstash-core.jar:?]
at org.logstash.Valuefier.convert(Valuefier.java:94) ~[logstash-core.jar:?]
at org.logstash.ConvertedMap.newFromMap(ConvertedMap.java:75) ~[logstash-core.jar:?]
at org.logstash.Valuefier.lambda$initConverters$13(Valuefier.java:172) ~[logstash-core.jar:?]
at org.logstash.Valuefier.convert(Valuefier.java:94) ~[logstash-core.jar:?]
at org.logstash.ConvertedMap.newFromMap(ConvertedMap.java:75) ~[logstash-core.jar:?]
at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.initializeFallback(JrubyEventExtLibrary.java:323) ~[logstash-core.jar:?]
at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_initialize(JrubyEventExtLibrary.java:97) ~[logstash-core.jar:?]
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_beats_minus_6_dot_0_dot_9_minus_java.lib.logstash.inputs.beats.message_listener.RUBY$method$onNewMessage$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.9-java/lib/logstash/inputs/beats/message_listener.rb:40) ~[?:?]