Using ES in production? Are you indexing huge quantities of data per second or do you have a steady but critical trickle of data coming in? Are you running complex queries with tonnes of aggregations or just browsing through documents with Kibana? If you have a stable ES cluster that you've built up and tuned for your applications and are happy with it in production, chances are that your configuration and 'secret sauce' in the tuning and resources of your cluster are very different than someone else's with a similar application but a different indexing load, different query types, or different resources to dedicate to ES. Would you be willing to share some insight into your cluster configuration for the benefit of other users of ES, both new and experienced?
I am a relatively new user of ES. I have found that the number of knobs, levers, bells, and whistles available in ES to be somewhat daunting, and I have often bumped into problems with my cluster that I am sure are due to a poor choice for one or two of those levers... I am hoping that enough people will respond to the questions below that we can have a good overview of working production configurations, and their use-cases, so that new users will have a good sample of configurations to build their own cluster with. If you spent hours, and buckets of blood and sweat, tuning your ES cluster, wouldn't you like to share your masterpiece with the community? This might also help the ES people in improving their documentation (not saying the documentation is bad, but documentation can always stand some improving 
).
If this has already been done, please let me know and point me to the results! If you have any additions or changes to suggest to the questions below, please provide some feedback!
Questions:
- If there was an over-riding reason -- such as improved performance or reliability or just a hard lesson learned -- for making a specific configuration decision below, please provide some details.
General usage:
- What version of ES are you using?
- How would you describe the use you have put ES to? Logging of services or applications? Application-specific document store? User activity analytics? System performance metrics? Others?
- What sort of indexing rates do you have? Number of documents per second on average?
- Are there better metrics describing the rate of data ingestion of your cluster? If so, what are they?
- Is the bulk of the indexing primarily on one index, or several?
- What sort of queries run on the cluster? Are queries primarily against the most recent data indexed (e.g. to trigger responses to certain events or event rates)? Do searches hit new and old data alike, hitting lots of documents and indexes? Do you use lots of complex aggregations?
- Any other details you'd like to provide?
Data volume:
- How many primary shards are there in your cluster?
- How many indexes?
- How many documents are there in your largest index? Largest shard? How big are these in terms of gigabytes?
- What is the total volume of data in your cluster (documents, gigabytes)?
- Any other metrics you'd like to share?
Physical Configuration:
- How many data nodes are there in your cluster? What sort of resources do they have (RAM, cpu cores, disk space dedicated to ES data)? What sort of disks and RAID do you use?
- How many master nodes do you have in your cluster? What sort of resources do they have (RAM, cpu cores)?
- Are your data and master nodes dedicated to that specific purpose, or do you combine data and master nodes? Do those nodes provide any other function, other than being members of your ES cluster?
- Do you use query-only (client-mode) ES nodes?
- What sort of network do you have among these nodes?
- Any other details you think are relevant or interesting or novel?
ES Configuration:
- How many shards does each index have?
- How many replicas does each index have?
- Was there a performance or reliability reason for choosing the number of shards per index? If so, what lead you to this number?
- Are there any ES configuration parameters that you have had to modify from the defaults in order to improve indexing, querying, or recovery performance? Please list those parameters, along with a few words on why you modified those parameters and arrived at the specific settings you are using.
- What options do you run the ES java instances with (either on the command-line or sysconfig file, not the elasticsearch.yml file)?
- Do you snapshot your cluster or back it up in some way? If so, any lessons learned there?
General Wisdom:
- Are there any details of your cluster that were not captured in the above questions that you'd like to share?
- Have you had any experiences with ES that taught you some valuable lessons which you'd like to share?
- Any other words of advice about ES, or pitfalls to avoid?
Thanks!
John Ouellette