Hi,
I recently tried the Filebeat Cisco module.
I tested the module with a 3 Node cluster where all nodes are: dilmrt
There is no other data ingested in the Cluster except Filebeat Cisco Asa log syslogs.
Every node have 32GB Memory and 16GB Heap, 4 vcpu.
I have tested the ingest pipeline from the module with bulk request over ESrally, and over Filebeat loading the logs from a file.
After doing this i had max throughput of ~2,5k events per second.
I was not satisfied with this throughput and started investigating if there's a bottleneck.
I'm aware that using a dedicated ingest node can give me more throughput, but i wanted to get the max out of my 3 Nodes.
During the ingest time of the Bulk request my CPU is about 90%+ all the time.
After reading the hot threads from elastic, i saw that CompoundProcessor/ConditionalProcessor did the most CPU load(file in attachment).
Investigating further the Ingest pipeline i realized that there a lot of conditionals that are tested on every incoming log (255).
I never heard that conditionals could load that much a CPU, but after using only one log type(302014) and deleting the other conditional processor especially dissect processors and now have only 95 conditionals. I get a higher throughput...(~4,2k ep/s).
Have someone experienced the same, or can say me if i did a mistake here?
Maybe we can do the Ingest processor performing better in divide the log types in different ingest processor so there not so many conditionals in the future...
hot_thread_elasticsearch
::: {xxxxxxxxxxx}{xxxxxxxxxxxxxxx}{xxxxxx-xxxx}{xxxxxx}{xxxxxxxxx:9300}{dilmrt}{ml.machine_memory=33733103616, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}
Hot threads at 2020-08-24T07:38:38.420Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
99.9% (499.5ms out of 500ms) cpu usage by thread 'elasticsearch[siem-elastic-host-0][write][T#4]'
10/10 snapshots sharing following 437 elements
app//org.elasticsearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:143)
app//org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:129)
app//org.elasticsearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:143)
app//org.elasticsearch.ingest.CompoundProcessor.lambda$innerExecute$1(CompoundProcessor.java:162)
app//org.elasticsearch.ingest.CompoundProcessor$$Lambda$6557/0x0000000801d88440.accept(Unknown Source)
app//org.elasticsearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:134)
app//org.elasticsearch.ingest.CompoundProcessor.lambda$innerExecute$1(CompoundProcessor.java:150)
app//org.elasticsearch.ingest.CompoundProcessor$$Lambda$6557/0x0000000801d88440.accept(Unknown Source)
app//org.elasticsearch.ingest.Processor.execute(Processor.java:54)
app//org.elasticsearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:143)
app//org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:129)
app//org.elasticsearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:143)
app//org.elasticsearch.ingest.CompoundProcessor.lambda$innerExecute$1(CompoundProcessor.java:162)
app//org.elasticsearch.ingest.CompoundProcessor$$Lambda$6557/0x0000000801d88440.accept(Unknown Source)
app//org.elasticsearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:134)
app//org.elasticsearch.ingest.CompoundProcessor.lambda$innerExecute$1(CompoundProcessor.java:150)
app//org.elasticsearch.ingest.CompoundProcessor$$Lambda$6557/0x0000000801d88440.accept(Unknown Source)
app//org.elasticsearch.ingest.Processor.execute(Processor.java:54)
app//org.elasticsearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:143)
app//org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:129)
app//org.elasticsearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:143)
app//org.elasticsearch.ingest.CompoundProcessor.lambda$innerExecute$1(CompoundProcessor.java:162)
app//org.elasticsearch.ingest.CompoundProcessor$$Lambda$6557/0x0000000801d88440.accept(Unknown Source)
app//org.elasticsearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:134)
app//org.elasticsearch.ingest.CompoundProcessor.lambda$innerExecute$1(CompoundProcessor.java:162)
app//org.elasticsearch.ingest.CompoundProcessor$$Lambda$6557/0x0000000801d88440.accept(Unknown Source)
app//org.elasticsearch.ingest.Processor.execute(Processor.java:57)
app//org.elasticsearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:143)
app//org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:129)
app//org.elasticsearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:143)
app//org.elasticsearch.ingest.CompoundProcessor.lambda$innerExecute$1(CompoundProcessor.java:162)
app//org.elasticsearch.ingest.CompoundProcessor$$Lambda$6557/0x0000000801d88440.accept(Unknown Source)
app//org.elasticsearch.ingest.ConditionalProcessor.lambda$execute$1(ConditionalProcessor.java:114)
app//org.elasticsearch.ingest.ConditionalProcessor$$Lambda$6559/0x0000000801d89840.accept(Unknown Source)
app//org.elasticsearch.ingest.Processor.execute(Processor.java:57)
app//org.elasticsearch.ingest.ConditionalProcessor.execute(ConditionalProcessor.java:107)