Events Received Rate Suddenly Dropped

Elastic Stack Logstash
1

Hi all,

I was looking at my stack monitoring and I was emitting just under 10,000 e/s and then all of a sudden it dropped to around 5,000 e/s and has stayed there. There were no changes made, so there shouldn't be a ~50% drop in packets.

Screen Shot 2020-02-20 at 9.10.48 AM

These are my logstash logs:

Feb 20 09:07:37 hrc-mgnt-logstash01 logstash[24064]: [2020-02-20T09:07:36,032][WARN ][org.logstash.dissect.Dissector][Infra_Syslog] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"<%{syslog_pri}>date=%{date} ", "event"=>{"tags"=>["udp", ":514", "_grokparsefailure", "_dissectfailure"], "message"=>"<143>2020-02-20 04:13:57,576 | DEBUG    | pc-activity-pool-53934    | PropertyMapDao                 | [vcId = 1224940e-ff87-4945-a15b-82a19f095acb] Updating resgroup-1061 in inventory DB, blob size 1,467 bytes. | \n", "type"=>"syslog", "@version"=>"1", "host"=>"$HOST_IP", "@timestamp"=>2020-02-20T00:13:48.980Z}}
Feb 20 09:07:37 hrc-mgnt-logstash01 logstash[24064]: [2020-02-20T09:07:36,032][WARN ][org.logstash.dissect.Dissector][Infra_Syslog] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"<%{syslog_pri}>date=%{date} ", "event"=>{"tags"=>["udp", ":514", "_grokparsefailure", "_dissectfailure"], "message"=>"<143>2020-02-20 04:13:57,576 | DEBUG    | pc-activity-pool-53934    | ObjectUpdateProcessor          | vc-id=1224940e-ff87-4945-a15b-82a19f095acb, Handling update \tObjectUpdate of kind => 'modify', type => 'ResourcePool' for object => 'resgroup-9642 | \n", "type"=>"syslog", "@version"=>"1", "host"=>"$HOST_IP", "@timestamp"=>2020-02-20T00:13:48.980Z}}
Feb 20 09:07:37 hrc-mgnt-logstash01 logstash[24064]: [2020-02-20T09:07:36,032][WARN ][org.logstash.dissect.Dissector][Infra_Syslog] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"<%{syslog_pri}>date=%{date} ", "event"=>{"tags"=>["udp", ":514", "_grokparsefailure", "_dissectfailure"], "message"=>"<182>1 2020-02-20T04:13:57.569+04:00 hrc-1c-vam.local l NSXV 6881 - [nsxv@6876 comp=\"nsx-manager\" level=\"INFO\" subcomp=\"manager\"] Invoking PropertyCollector.waitForUpdatesEx on session[52273717-e60e-9f18-57b7-ebf78f3759d2]52ff47cd-26e5-8d53-a8ab-3ef47cf8015e, operationID=opId-4c8f8-18809806\n", "type"=>"syslog", "@version"=>"1", "host"=>"$HOST_IP", "@timestamp"=>2020-02-20T00:13:48.982Z}}
Feb 20 09:07:37 hrc-mgnt-logstash01 logstash[24064]: [2020-02-20T09:07:36,032][WARN ][org.logstash.dissect.Dissector][Infra_Syslog] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"<%{syslog_pri}>date=%{date} ", "event"=>{"tags"=>["udp", ":514", "_grokparsefailure", "_dissectfailure"], "message"=>"<143>2020-02-20 04:13:57,579 | DEBUG    | pc-activity-pool-53934    | PropertyMapDao                 | [vcId = 1224940e-ff87-4945-a15b-82a19f095acb] Updating resgroup-9642 in inventory DB, blob size 1,423 bytes. | \n", "type"=>"syslog", "@version"=>"1", "host"=>"$HOST_IP", "@timestamp"=>2020-02-20T00:13:48.983Z}}
Feb 20 09:07:37 hrc-mgnt-logstash01 logstash[24064]: [2020-02-20T09:07:36,032][WARN ][org.logstash.dissect.Dissector][Infra_Syslog] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"<%{syslog_pri}>date=%{date} ", "event"=>{"tags"=>["udp", ":514", "_grokparsefailure", "_dissectfailure"], "message"=>"<143>2020-02-20 04:13:57,581 | DEBUG    | pc-activity-pool-53934    | PropertyCollectorReader        | [Activity Execution] PC version=65628, synced=true, vc-id=1224940e-ff87-4945-a15b-82a19f095acb, profile=IAAS_OTHER listener-state=CONNECTED - Handle: urn:uuid:e211737b-c6cc-3553-8bf0-9cd01d2d7376, Current Phase: PropertyCollectorReader$ReadPhase | \n", "type"=>"syslog", "@version"=>"1", "host"=>"$HOST_IP", "@timestamp"=>2020-02-20T00:13:48.986Z}}

Any help is greatly appreciated.
Thanks,
Nhung

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.