I'm looking at using Logstash to poll several hosts for SNMP data migrating from an old python
script in favor of ELK.
The pipeline is really incredibly simple. read data, convert to message format (ruby), send to google pubsub.
The message processing time is great but the boot up time before logstash is able to process data is around 20+ minutes. I was wondering if that's expected or if I'm doing something wrong. I have about 200 hosts i'm monitoring, with the config broken down into a config file for each host.
Example below:
input {
snmp {
# query interval
interval => 30
tables => [
{ "name" => "oid.TIMETRA-SAP-MIB::sapBaseStatsEgressQchipDroppedInProfOctets" "columns" => [ "1.3.6.1.4.1.6527.3.1.2.4.3.6.1.16" ] } ,
{ "name" => "oid.TIMETRA-SAP-MIB::sapBaseStatsEgressQchipDroppedInProfPackets" "columns" => [ "1.3.6.1.4.1.6527.3.1.2.4.3.6.1.15" ] } ,
{ "name" => "oid.TIMETRA-SAP-MIB::sapBaseStatsEgressQchipDroppedOutProfOctets" "columns" => [ "1.3.6.1.4.1.6527.3.1.2.4.3.6.1.18" ] } ,
{ "name" => "oid.TIMETRA-SAP-MIB::sapBaseStatsEgressQchipDroppedOutProfPackets" "columns" => [ "1.3.6.1.4.1.6527.3.1.2.4.3.6.1.17" ] } ,
{ "name" => "oid.TIMETRA-SAP-MIB::sapBaseStatsEgressQchipForwardedInProfOctets" "columns" => [ "1.3.6.1.4.1.6527.3.1.2.4.3.6.1.20" ] } ,
{ "name" => "oid.TIMETRA-SAP-MIB::sapBaseStatsEgressQchipForwardedInProfPackets" "columns" => [ "1.3.6.1.4.1.6527.3.1.2.4.3.6.1.19" ] } ,
{ "name" => "oid.TIMETRA-SAP-MIB::sapBaseStatsEgressQchipForwardedOutProfOctets" "columns" => [ "1.3.6.1.4.1.6527.3.1.2.4.3.6.1.22" ] } ,
{ "name" => "oid.TIMETRA-SAP-MIB::sapBaseStatsEgressQchipForwardedOutProfPackets" "columns" => [ "1.3.6.1.4.1.6527.3.1.2.4.3.6.1.21" ] } ,
{ "name" => "oid.TIMETRA-SAP-MIB::sapBaseStatsIngressQchipDroppedHiPrioOctets" "columns" => [ "1.3.6.1.4.1.6527.3.1.2.4.3.6.1.8" ] } ,
{ "name" => "oid.TIMETRA-SAP-MIB::sapBaseStatsIngressQchipDroppedHiPrioPackets" "columns" => [ "1.3.6.1.4.1.6527.3.1.2.4.3.6.1.7" ] } ,
{ "name" => "oid.TIMETRA-SAP-MIB::sapBaseStatsIngressQchipDroppedLoPrioOctets" "columns" => [ "1.3.6.1.4.1.6527.3.1.2.4.3.6.1.10" ] } ,
{ "name" => "oid.TIMETRA-SAP-MIB::sapBaseStatsIngressQchipDroppedLoPrioPackets" "columns" => [ "1.3.6.1.4.1.6527.3.1.2.4.3.6.1.9" ] } ,
{ "name" => "oid.TIMETRA-SAP-MIB::sapBaseStatsIngressQchipForwardedInProfOctets" "columns" => [ "1.3.6.1.4.1.6527.3.1.2.4.3.6.1.12" ] } ,
{ "name" => "oid.TIMETRA-SAP-MIB::sapBaseStatsIngressQchipForwardedInProfPackets" "columns" => [ "1.3.6.1.4.1.6527.3.1.2.4.3.6.1.11" ] } ,
{ "name" => "oid.TIMETRA-SAP-MIB::sapBaseStatsIngressQchipForwardedOutProfOctets" "columns" => [ "1.3.6.1.4.1.6527.3.1.2.4.3.6.1.14" ] } ,
{ "name" => "oid.TIMETRA-SAP-MIB::sapBaseStatsIngressQchipForwardedOutProfPackets" "columns" => [ "1.3.6.1.4.1.6527.3.1.2.4.3.6.1.13" ] } ,
{ "name" => "oid.IF-MIB::ifHCInOctets" "columns" => [ "1.3.6.1.2.1.31.1.1.1.6" ] } ,
{ "name" => "oid.IF-MIB::ifHCOutOctets" "columns" => [ "1.3.6.1.2.1.31.1.1.1.10" ] }
]
# List of hosts to poll
hosts => [{host => "udp:example-host/161" community => "public" version => "2c" retries => 2 timeout => 1000 } ]
add_field => {
"[@metadata][projectId]" => 1
"[@metadata][collectionId]" => 1
}
}
## the snmp{} section would repeat for additional intervals. This particular host has 30 s interval, 300s, 600s and 3600s
I can group the configs by grouping of oIds and send an array of hosts instead. Is there an optimal way of configuring logstash where the bootup time isn't so long?
Also any known limits beyond physical memory/cpu ?