i am currently testing the elastic stack for observerability use-cases in my company. For testing purposes we build a small elasticsearch cluster (3 nodes) and ingesting http-logs with filebeat.
Our PoC-setup looks like the following:
3 ES-Nodes: 8 Cores, 8 GB RAM (4GB ES Heap), 100GB HDD
Filebeat: 4 Cores, 4 GB RAM, 50GB HDD
During testing we noticed very slow performance ingesting http-logs from filebeat to elasticsearch. We used about 5 GB of old apache logs and ingesting them in a three primary shard index (no replication) took about 6 hours on average. We tested this multiple times.
Using only a single elasticsearch instance and a single primary shard, ingesting this data took even longer with an average duration of 8-9 hours.
For comparison: Using Splunk and their universalforwarder indexing only takes about 15min (on the same system).
Every configuration is kept to the most default. We aren't using an x-pack extension and only configured minimum elasticsearch and filebeat settings. (We do not want to tweak performance to a maximum, but rather get the "default" performance of elastic)
Here are also some elasticsearch-system stats, which were collected during ingesting.
CPU Usage: About 50% (htop shows load average of ~4 of 8 cores)
RAM Usage: About 5-6/8 GB per Node (4 GB is fully assigned as ES Heap with mem_lock: true)
|Node|%user|%iowait|kB_read/s|kB_write/s|~ uptime| |------|-------|-------|------|-------|-----------| |ES-1|4.20|13.00|108|1423|160min| |ES-2|4.15|12.37|113|1425|160min| |ES-3|4.07|12.63|96 |1389|160min| |FB |1.31|1.27 |559|19 | |
Is this kind of performance excepted? If not, has anyone an idea where the problem might be? I am free to provide any additionally needed infos and can also offer to take a live view on our poc-systems.