Increased disk writes on elasticsearch

Elastic Stack Elasticsearch
1

Hi all, kinda new to ES, so please bear with me.
I've noticed a clear increse in disk WRITES more or less (but not quite) coincident with the upgrade to ES 7.
I have only one node with around 150 shards ingesting data from logstash to filter and store server log information.
Until a month ago, the disk write was "basically" zero. Then suddenly, around April 25th, it increased to around 5MB/s continuasly. Iotop shows java as the main responsible for this writes. CPU usage remains low as always.
This in itself is not really a problem. The server can sustain the increased I/O, but I would really like to know what happened.
What I did:

  1. stopping elasticsearch, disk writes go down
  2. stopping logstash, no changes
  3. removing swap, no changes
  4. increasing java heap, no changes
  5. browsing the logs didn't show anything clear to me

Any clue on how to "troubleshoot" this would be appreciated. Browsing through the forum did show some similar issues but with clear causes that don't really apply to my case.

Thanks for any help you can provide,
Best regards,
Carlos

2

It is possible that this is the issue fixed by #42299, so perhaps the simplest thing to do is upgrade to 7.1.1 when that becomes available, and see if this resolves your issue. If not then it'd make sense to dig further.

How are you measuring this? How suddenly did the increase occur? How does this relate to your upgrade to 7.x in terms of timing?

We recommend running without swap as a matter of course.

1 Like
3

Hey DavidTurner, thanks for your reply.
You seem to have hit something. As for your comments:

  1. I visited #42299. Waaay too cryptic for me.

  2. I'm monitoring all my servers using Netdata [(https://github.com/netdata/netdata)] and using elasticsearch for long term storage of data.
    The upgrade to 7.0.0 happened the 21st of April. After some unrelated glitches, on the 24th, the old 6.7 indexes where reindexed to 7.0. This is where things went awry. The reindexing took a few (around 4) hours where CPU and disk I/O spiked. After that , cpu went to it's normal values, but disk I/O never returned to baseline. See attached image; on the upper right graph, the green line is the server's disk writes.
    image
    All indexes (old and new) are green

  3. Swap is a remnant from the past. I'll remove it eventually.

best regards!
Carlos

4

Ok, that could well be consistent with the issue fixed by #42299. I wouldn't worry about the details, the key point is that this change should be included in 7.1.1, so let's wait for the upgrade before doing much more investigation.

5

Very well.
Like I said, it's really not a critical issue for me. Just...anoying.
Thanks again for your help. looking forward to 7.1.1 then
Best regards,
Carlos

6

It looks like 7.1.1 is now available for download: https://www.elastic.co/downloads/elasticsearch

7

Hi David,
indeed it has and indeed it worked!
Disk I/O went back to expected levels.
Once again, thank you very much for your assistance.

Best regards,
Carlos

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.