Problem of ElasticSearch on ZFS

Elastic Stack Elasticsearch
1

Dear All

Could I check whether anyone deploy ElasticSearch on ZFS?

We are facing some problems with our ES instance on ZFS. When indexing (and
idle mode) there seems to be high volume of reads on the drives using ZFS,
thus resulting in high iowait, low CPU utilization. However, there is very
little writes on the drives.

We also experimented putting the data directory in a non-ZFS based drive.
The performance seems much better, good CPU utilization, much higher
volumes of writes compared to reads.

Anyone have any hints of what could be wrong.

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a73ed552-068b-4695-a975-526d50b919d9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Do you run ZFS in sync mode? Are you on Solaris?

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE9K97GTyLhi-7B--ycnNawAaj%3D64LASJMugo4vUrnSOw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hi

we are running on Linux (ubuntu).

The ZFS is set to standard (default).

On Wednesday, January 1, 2014 12:43:35 AM UTC+8, Jörg Prante wrote:

Do you run ZFS in sync mode? Are you on Solaris?

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5c32e5c6-1188-4551-b96c-2e1a3bd1e21e%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Which Ubuntu version? What ZFS product/version?

What hardware controller do you use for ZFS disks?

You have to carefully prepare ZFS for apps like ES, to avoid RAM
overallocation, block misalignments, and double caching.

For example:

  • only assign a fragment of RAM to ZFS adaptive replacement cache
    (zfs_arc_max) - ES is doing all the caching

  • use the noop scheduler (ZFS reorders IO requests)

  • use a read ahead block size of ZFS block size (drive ops must match ZFS
    block size)

  • turn on write through of the disks (turn of write cache)

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGFyqzpRg275X%3DHxYSVVfpZV5z6auyN9qA-LdD9cVyGVw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.