# Trying to find the reason of performance hiccup **URL:** \ **Category:** Elasticsearch\ **Created:** [September 25, 2012, 9:02pm UTC](https://discuss.elastic.co/t/trying-to-find-the-reason-of-performance-hiccup/9142 "2012-09-25T21:02:12Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Jae](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jae/32/791_2.png) [@Jae](https://discuss.elastic.co/u/Jae)\ **Post date:** [September 25, 2012, 9:02pm UTC](https://discuss.elastic.co/t/trying-to-find-the-reason-of-performance-hiccup/9142/1 "2012-09-25T21:02:12Z")
Hi I am running 36 instances of elasticsearch-0.19.9. Around 11AM, one instance was terminated because of AWS system check and it was replaced immediately. After then, I can see 4 times performance hiccups. Please look at the attached graph. There is a bounded queue from logger and elasticsearch client. If the elasticsearch client is getting slower to be responded from the servers, the queue will be filled up. Red line means dropped messages and green line plummeted means the server was not handling any traffic. How can I trace it? Also, what should I do to prevent the performance hiccup? I was trying to find out what happened after 11AM, but I couldn't find anything except connection time out to the dead instances. In the client side, there were no failure, it was just a performance degradation. The following is logging.yml. Thank you Best, Jae * * * rootLogger: INFO, console, file logger: # log action execution errors for easier debugging action: DEBUG # reduce the logging for aws, too much is logged under the default INFO com.amazonaws: WARN # gateway gateway: DEBUG #index.gateway: DEBUG # peer shard recovery #indices.recovery: DEBUG # discovery discovery: TRACE index.search.slowlog: TRACE, index\_search\_slow\_log\_file org.apache: WARN additivity: index.search.slowlog: false appender: console: type: console layout: type: consolePattern conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n" file: type: dailyRollingFile file: ${path.logs}/${cluster.name}.log datePattern: "'.'yyyy-MM-dd" layout: type: pattern conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n" index\_search\_slow\_log\_file: type: dailyRollingFile file: ${path.logs}/${cluster.name}\_index\_search\_slowlog.log datePattern: "'.'yyyy-MM-dd" layout: type: pattern conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n" -- ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [September 26, 2012, 3:31am UTC](https://discuss.elastic.co/t/trying-to-find-the-reason-of-performance-hiccup/9142/2 "2012-09-26T03:31:36Z")
Hi, What about other ES, JVM, or OS metrics - they may reveal the source. Maybe shards were being moves around the cluster during those 4 times? You can get this info from ES. We graph that in SPM and we found this very informative when troubleshooting performance issues. ## Otis Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Tuesday, September 25, 2012 5:02:13 PM UTC-4, Jae wrote: > Hi > > I am running 36 instances of elasticsearch-0.19.9. Around 11AM, one > instance was terminated because of AWS system check and it was replaced > immediately. After then, I can see 4 times performance hiccups. Please look > at the attached graph. There is a bounded queue from logger and > elasticsearch client. If the elasticsearch client is getting slower to be > responded from the servers, the queue will be filled up. Red line means > dropped messages and green line plummeted means the server was not handling > any traffic. > > How can I trace it? Also, what should I do to prevent the performance > hiccup? > > I was trying to find out what happened after 11AM, but I couldn't find > anything except connection time out to the dead instances. In the client > side, there were no failure, it was just a performance degradation. The > following is logging.yml. > > Thank you > Best, Jae > > * * * > > rootLogger: INFO, console, file > logger: > > # log action execution errors for easier debugging > > action: DEBUG > > # reduce the logging for aws, too much is logged under the default INFO > > com.amazonaws: WARN > > # gateway > > gateway: DEBUG > #index.gateway: DEBUG > > # peer shard recovery #indices.recovery: DEBUG > > # discovery > > discovery: TRACE > index.search.slowlog: TRACE, index\_search\_slow\_log\_file > > org.apache: WARN > > additivity: > index.search.slowlog: false > > appender: > console: > type: console > layout: type: consolePattern > conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n" > file: type: dailyRollingFile > file: ${path.logs}/${cluster.name}.log > datePattern: "'.'yyyy-MM-dd" > layout: > type: pattern > conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n" > > index\_search\_slow\_log\_file: > type: dailyRollingFile > file: ${path.logs}/${cluster.name}\_index\_search\_slowlog.log > datePattern: "'.'yyyy-MM-dd" > layout: > type: pattern > conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n" -- ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [September 27, 2012, 1:59am UTC](https://discuss.elastic.co/t/trying-to-find-the-reason-of-performance-hiccup/9142/3 "2012-09-27T01:59:24Z")
If it coincided with a node leaving and joining the cluster it was most likely due to the cold shards that came online. If you're doing sorts or facets, they pull the entire dataset for that field into memory. I believe this feature will mitigate: > > > Index warming allows to run registered search requests to warm up the index befo…re it is available for search. With the near real time aspect of search, cold data (segments) will be warmed up before they become available for search. > > Warmup searches typically include requests that require heavy loading of data, such as faceting or sorting on specific fields. > > The warmup APIs allows to register warmup (search) under specific names, remove them, and get them. > > Index warmup can be disabled by setting \`index.warmer.enabled\` to \`false\`. It is supported as a realtime setting using update settings API. This can be handy when doing initial bulk indexing, disabling pre registered warmers to make indexing faster and less expensive and then enable it. > \## Put Warmer > > Allows to put a warmup search request on a specific index (or indices), with the body composing of a regular search request. Types can be provided as part of the URI if the search request is designed to be run only against the specific types. > > Here is an example that registers a warmup called \`warmer\_1\` against index \`test\` (can be alias or several indices), for a search request that runs against all types: > > \`\`\` > curl -XPUT localhost:9200/test/\_warmer/warmer\_1 -d '{ > "query" : { > "match\_all" : {} > }, > "facets" : { > "facet\_1" : { > "terms" : { > "field" : "field" > } > } > } > }' > \`\`\` > > And an example that registers a warmup against specific types: > > \`\`\` > curl -XPUT localhost:9200/test/type1/\_warmer/warmer\_1 -d '{ > "query" : { > "match\_all" : {} > }, > "facets" : { > "facet\_1" : { > "terms" : { > "field" : "field" > } > } > } > }' > \`\`\` > \## Delete Warmer > > Removing a warmer can be done against an index (or alias / indices) based on its name. The provided name can be a simple wildcard expression or omitted to remove all warmers. Some samples: > > \`\`\` > \# delete warmer named warmer\_1 on test index > curl -XDELETE localhost:9200/test/\_warmer/warmer\_1 > > \# delete all warmers that start with warm on test index > curl -XDELETE localhost:9200/test/\_warmer/warm\* > > \# delete all warmers for test index > curl -XDELETE localhost:9200/test/\_warmer/ > \`\`\` > \## GETting Warmer > > Getting a warmer for specific index (or alias, or several indices) based on its name. The provided name can be a simple wildcard expression or omitted to get all warmers. Some examples: > > \`\`\` > \# get warmer named warmer\_1 on test index > curl -XGET localhost:9200/test/\_warmer/warmer\_1 > > \# get all warmers that start with warm on test index > curl -XGET localhost:9200/test/\_warmer/warm\* > > \# get all warmers for test index > curl -XGET localhost:9200/test/\_warmer/ > \`\`\` Best Regards, Paul On Tuesday, September 25, 2012 9:31:37 PM UTC-6, Otis Gospodnetic wrote: > Hi, > > What about other ES, JVM, or OS metrics - they may reveal the source. > Maybe shards were being moves around the cluster during those 4 times? > You can get this info from ES. We graph that in SPM and we found this very > informative when troubleshooting performance issues. > > ## Otis > > Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) > Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) > > On Tuesday, September 25, 2012 5:02:13 PM UTC-4, Jae wrote: > > > Hi > > > > I am running 36 instances of elasticsearch-0.19.9. Around 11AM, one > > instance was terminated because of AWS system check and it was replaced > > immediately. After then, I can see 4 times performance hiccups. Please look > > at the attached graph. There is a bounded queue from logger and > > elasticsearch client. If the elasticsearch client is getting slower to be > > responded from the servers, the queue will be filled up. Red line means > > dropped messages and green line plummeted means the server was not handling > > any traffic. > > > > How can I trace it? Also, what should I do to prevent the performance > > hiccup? > > > > I was trying to find out what happened after 11AM, but I couldn't find > > anything except connection time out to the dead instances. In the client > > side, there were no failure, it was just a performance degradation. The > > following is logging.yml. > > > > Thank you > > Best, Jae > > > > * * * > > > > rootLogger: INFO, console, file > > logger: > > > > # log action execution errors for easier debugging > > > > action: DEBUG > > > > # reduce the logging for aws, too much is logged under the default INFO > > > > com.amazonaws: WARN > > > > # gateway > > > > gateway: DEBUG > > #index.gateway: DEBUG > > > > # peer shard recovery #indices.recovery: DEBUG > > > > # discovery > > > > discovery: TRACE > > index.search.slowlog: TRACE, index\_search\_slow\_log\_file > > > > org.apache: WARN > > > > additivity: > > index.search.slowlog: false > > > > appender: > > console: > > type: console > > layout: type: consolePattern > > conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n" > > file: type: dailyRollingFile > > file: ${path.logs}/${cluster.name}.log > > datePattern: "'.'yyyy-MM-dd" > > layout: > > type: pattern > > conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n" > > > > index\_search\_slow\_log\_file: > > type: dailyRollingFile > > file: ${path.logs}/${cluster.name}\_index\_search\_slowlog.log > > datePattern: "'.'yyyy-MM-dd" > > layout: > > type: pattern > > conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n" -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:11am UTC](https://discuss.elastic.co/t/trying-to-find-the-reason-of-performance-hiccup/9142/4 "2017-07-06T03:11:15Z")
Closed on July 6, 2017, 3:11am UTC