# Ingestion performance issues - where to start? **URL:** \ **Category:** Elasticsearch\ **Created:** [August 21, 2020, 4:11am UTC](https://discuss.elastic.co/t/ingestion-performance-issues-where-to-start/245849 "2020-08-21T04:11:18Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![wardallen](https://avatars.discourse-cdn.com/v4/letter/w/ed655f/32.png) [@wardallen](https://discuss.elastic.co/u/wardallen)\ **Post date:** [August 21, 2020, 4:11am UTC](https://discuss.elastic.co/t/ingestion-performance-issues-where-to-start/245849/1 "2020-08-21T04:11:18Z")
Hi all, I'm having ingestion performance issues that I haven't gotten to the bottom of, and I'm quite new to the elastic stack, so I thought I'd seek advice here. I have a cluster of 3 VMs (4 CPU/64GB RAM/500GB disk). RHEL 7. Elasticsearch 7.8.0 is installed on all of them, and configured in a cluster (transport encrypted, http not encrypted). 26GB heap size, usually around 50% utilised The index is in 3 shards with 2 copies (high availability was a priority) Logstash 7.8.0 is also installed on every box, output pointing at elastic on the same box. 4GB heap size, 8 workers, 125 batch size. Logs are round-robined through a VIP to each of the boxes. Filter configured in logstash to use csv to pull out 66 fields Data is on average ~400 bytes/event, the sources send it through at approx. 160Mb/s. I’m finding that this system cannot keep up, logs buffers are building up on the data source devices. When logstash is turned on, there are TCP window resize requests (to a few hundred bytes) arriving at the data source. However, I find nothing saying anything about having to throttle incoming data in the logs. Logstash is ingesting at approx 40Mb/s. When I turn logstash off and listen with ncat dumping straight to /dev/null, this backlog disappears because throughput skyrockets. . The CPU usage hangs around 70%,. I have tried reducing the 66 fields to 8, which reduced ingest time by about a third in my lab. Is there anywhere in this that jumps out that should be done differently? Do I have enough hardware? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 21, 2020, 4:17am UTC](https://discuss.elastic.co/t/ingestion-performance-issues-where-to-start/245849/2 "2020-08-21T04:17:29Z")
The first thing to do is to try and determine whether Elasticsearch is the bottleneck. I often start looking at storage as indexing can be very I/O intensive and it sounds like CPU and heap usage are OK. Are you using local SSDs for storage? If not, what does `iostat -x` give on the nodes during indexing? ---
**Author:** ![wardallen](https://avatars.discourse-cdn.com/v4/letter/w/ed655f/32.png) [@wardallen](https://discuss.elastic.co/u/wardallen)\ **Post date:** [August 21, 2020, 4:33am UTC](https://discuss.elastic.co/t/ingestion-performance-issues-where-to-start/245849/3 "2020-08-21T04:33:51Z")
Thanks, Christian. I'm not using SSDs. ```auto # iostat -x Linux 3.10.0-957.el7.x86_64 () 08/21/2020 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 27.14 6.88 2.71 9.78 0.00 53.49 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 0.00 460.67 0.00 311.51 0.00 12809.37 82.24 0.84 2.78 2.96 2.78 1.23 38.34 ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [August 21, 2020, 5:42am UTC](https://discuss.elastic.co/t/ingestion-performance-issues-where-to-start/245849/4 "2020-08-21T05:42:52Z")
Have you perhaps considered running logstash on separate different VMs from elasticsearch, not that you can't but they will be competing for resources. Maybe I am not understanding your architecture. And as Christian asked, what kind of storage are you using? Also HW wise even though CPU says 70% 4 CPU for 64GB RAM plus Logstash feels light too me. You are also writing 3 copies of the data. ---
**Author:** ![Steve\_Mushero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_mushero/32/22441_2.png) [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)\ **Post date:** [August 21, 2020, 8:00am UTC](https://discuss.elastic.co/t/ingestion-performance-issues-where-to-start/245849/5 "2020-08-21T08:00:01Z")
Agreed, as I bet it's Logstash as that high CPU might mean you are maxing out on a thread or two that's doing the work, likely in LS, though possible in ES - ES queues may show if it's the bottleneck; IO looks very good - I bet the CSV processing and 66 fields is tying up LS too much. GREAT up front description, by the way, listing all the nodes, RAM heaps, data rates & sizes, and so on. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [August 21, 2020, 1:35pm UTC](https://discuss.elastic.co/t/ingestion-performance-issues-where-to-start/245849/6 "2020-08-21T13:35:04Z")
One minor additional comment dissect filter is much more performant than csv filter in logstash. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 18, 2020, 1:35pm UTC](https://discuss.elastic.co/t/ingestion-performance-issues-where-to-start/245849/7 "2020-09-18T13:35:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.