Right now I am using ES to store and search NSM logs produced with Bro. I am monitoring a 4.4g network which as you can imagine produces an extreme amount of log data.
My current setup:
Sensor using logstash to read logs -> Kafka on sensor (works great)
Kafka cluster reading logs from sensor kafka (works great and ingests the logs at what appears to be minimal lag)
Kafka cluster stats:
3 Zookeepers:
Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz Stepping: 2 CPU MHz: 2141.706 BogoMIPS: 4283.41 Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 46080K NUMA node0 CPU(s): 0-3
RAM = 16GB
9 Broker Nodes with the same Stats as Zookeepers
I then have 4 nodes running logstash only,
Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 1 Core(s) per socket: 8 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 45 Model name: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz Stepping: 7 CPU MHz: 1966.048 BogoMIPS: 3932.09 Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 20480K NUMA node0 CPU(s): 0-7
RAM = 32GB with 24 set for JAVA_HEAP
These 4 nodes are reading the kafka cluster topics, filtering data, and outputting to ES.
below is my output section:
output {
########## BRO Outputs -> ES Cluster ##########
if [type] == "BRO" {
if [sensor1] == "host1" {
elasticsearch {
hosts => [ "10.1.55.8:9200","10.1.55.9:9200","10.1.55.10:9200","10.1.55.11:9200","10.1.55.12:9200","10.1.55.13:9200","10.1.55.14:9200","10.1.55.15:9200","10.1.55.16:9200","10.1.55.17:9200","10.1.55.18:9200","10.1.55.19:9200" ]
manage_template => false
flush_size => 5000
idle_flush_time => 5
workers => 8
index => "sensor1-bro-%{+YYYY.MM.dd}"
}
}
if [sensor2] == "host2" {
elasticsearch {
hosts => [ "10.1.55.8:9200","10.1.55.9:9200","10.1.55.10:9200","10.1.55.11:9200","10.1.55.12:9200","10.1.55.13:9200","10.1.55.14:9200","10.1.55.15:9200","10.1.55.16:9200","10.1.55.17:9200","10.1.55.18:9200","10.1.55.19:9200" ]
manage_template => false
flush_size => 5000
idle_flush_time => 5
workers => 8
index => "host2-bro-%{+YYYY.MM.dd}"
}
}
}
}
My ES cluster is currently 3 Masters and 12 Data Nodes, all stats are the same:
Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Thread(s) per core: 1 Core(s) per socket: 16 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 45 Model name: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz Stepping: 7 CPU MHz: 1941.527 BogoMIPS: 3883.05 Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 20480K NUMA node0 CPU(s): 0-15
RAM = 32GB with 24 set for JAVA_HEAP
continued on second post
