Performance with SSD

Elastic Stack Elasticsearch
1

Hi folks,

Has anyone tried ES with SSDs in general, or FusionIO in particular? Some
folks are reporting drastic performance increase when used with databases,
etc. Curious to find out whether ES usage patterns are conducive to take
advantage of this technology.

Regards,
Berkay Mollamustafaoglu
Ph: +1 (571) 766-6292
mberkay on yahoo, google and skype

2

Hey,
We were using SSDs on an old search system because we were disk
bound. With ES, our load testing showed us that we were actually CPU
bound, mostly because we have enough RAM for a large amount of disk
caching.

So, we saved some money and got larger non-SSDs. Our servers have the
following h/w:

  • 24 core CPU
  • 48 GB ram

This is with a total index size of ~180GB, 30 million documents.

You won't really know until you give it a shot, though. My
recommendation is to do load testing and figure out what your
bottleneck is without SSDs and if you do appear disk bound, give them
a shot.

Regards,
Paul

On Dec 21, 1:25 pm, Berkay Mollamustafaoglu mber...@gmail.com wrote:

Hi folks,

Has anyone tried ES with SSDs in general, or FusionIO in particular? Some
folks are reporting drastic performance increase when used with databases,
etc. Curious to find out whether ES usage patterns are conducive to take
advantage of this technology.

Regards,
Berkay Mollamustafaoglu
Ph: +1 (571) 766-6292
mberkay on yahoo, google and skype

3

Hi Paul,

Thanks for sharing the info; it is very useful. You're right the best way
will be to do the tests. Our application is write heavy so results will
likely be different than most.

We'll start with virtual environment, check to see where the bottlenecks are
and go from there. It's just that FusionIO numbers talked about were very
impressive so I'm curious about how much of an impact it would have. Of
course, if the app turns out to be CPU constrained, then it is not that
relevant.

Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype

On Tue, Dec 21, 2010 at 3:50 PM, Paul ppearcy@gmail.com wrote:

Hey,
We were using SSDs on an old search system because we were disk
bound. With ES, our load testing showed us that we were actually CPU
bound, mostly because we have enough RAM for a large amount of disk
caching.

So, we saved some money and got larger non-SSDs. Our servers have the
following h/w:

  • 24 core CPU
  • 48 GB ram

This is with a total index size of ~180GB, 30 million documents.

You won't really know until you give it a shot, though. My
recommendation is to do load testing and figure out what your
bottleneck is without SSDs and if you do appear disk bound, give them
a shot.

Regards,
Paul

On Dec 21, 1:25 pm, Berkay Mollamustafaoglu mber...@gmail.com wrote:

Hi folks,

Has anyone tried ES with SSDs in general, or FusionIO in particular? Some
folks are reporting drastic performance increase when used with
databases,
etc. Curious to find out whether ES usage patterns are conducive to take
advantage of this technology.

Regards,
Berkay Mollamustafaoglu
Ph: +1 (571) 766-6292
mberkay on yahoo, google and skype

4

Paul and I work together and I wanted to add a few additional notes.

With adequate memory, search performance will be limited by CPU. We
allocate 24G ram (1/2 of total) to ES. The rest is for our search and
indexing apps, but mostly for Linux disk cache.

Index performance will be more disk bound. When indexing full text
documents (we have a lot of 250-500k PDFs) we can see the difference
between a system with RAID5 and one with RAID10. In our systems we
have 4x146G 15k rpm drives. In our steady state case of indexing new
documents, indexing time and load are minimal. Only when building the
full index is where we can see disk performance becoming a limiting
factor.

As a general rule, I'd spend the money to ensure adequate RAM before
jumping from high performance HDs to SSDs.
David

On Dec 22, 8:53 am, Berkay Mollamustafaoglu mber...@gmail.com wrote:

Hi Paul,

Thanks for sharing the info; it is very useful. You're right the best way
will be to do the tests. Our application is write heavy so results will
likely be different than most.

We'll start with virtual environment, check to see where the bottlenecks are
and go from there. It's just that FusionIO numbers talked about were very
impressive so I'm curious about how much of an impact it would have. Of
course, if the app turns out to be CPU constrained, then it is not that
relevant.

Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype

On Tue, Dec 21, 2010 at 3:50 PM, Paul ppea...@gmail.com wrote:

Hey,
We were using SSDs on an old search system because we were disk
bound. With ES, our load testing showed us that we were actually CPU
bound, mostly because we have enough RAM for a large amount of disk
caching.

So, we saved some money and got larger non-SSDs. Our servers have the
following h/w:

  • 24 core CPU
  • 48 GB ram

This is with a total index size of ~180GB, 30 million documents.

You won't really know until you give it a shot, though. My
recommendation is to do load testing and figure out what your
bottleneck is without SSDs and if you do appear disk bound, give them
a shot.

Regards,
Paul

On Dec 21, 1:25 pm, Berkay Mollamustafaoglu mber...@gmail.com wrote:

Hi folks,

Has anyone tried ES with SSDs in general, or FusionIO in particular? Some
folks are reporting drastic performance increase when used with
databases,
etc. Curious to find out whether ES usage patterns are conducive to take
advantage of this technology.

Regards,
Berkay Mollamustafaoglu
Ph: +1 (571) 766-6292
mberkay on yahoo, google and skype