Thanks for your insight about primarycache=metadata, which is considered a suitable setting for database workload with high write throughput. For Elasticsearch, however, query workload will typically overweigh other workload patterns. So if ZFS ARC can be adapted to coexist with mmap() cached files of most recent Lucene 6, it is reasonable to use primarycache=all.
Another ZFS-specific issue on Linux is that Linux reads 128k at each random read from a file system, while ZFS block access will be 4k for each call, which will lead to a 32x higher IO random read rate when caching is effectively disabled by primarycache=metadata, resulting in poor performance.
ZFS has quite a learning curve and it took me months to configure ZFS for an Oracle DB on Solaris (which I migrated finally to XFS under Linux).