Is heapCurrent reflective of performance

Elastic Stack Elasticsearch
1

Im trying to dig a bit into the performance of my ES cluster and I'm trying to understand _cat/nodes.

My response to
GET /_cat/nodes?v&h=name,ip,heapCurrent,heapMax,ramCurrent,ramMax,cpu,getTime,getTotal,getMissingTotal,indexingIndexTotal,mergesTotalTime

Screen Shot 2017-05-18 at 10.37.20 AM.png1820×198 40.4 KB

My query time is very slow and I get a timeout error often (even for small searches).

The heapCurrent isn't even close to to heapMax. So is my system even using the heap that I allotted to it? My machines have 8gb ram (other than the elastic1 machine which has 4 which I'm upgrading soon). Does this view give anyone insight as so why my query speed from kibana is slow?

2

Please don't post pictures of text, they are difficult to read and some people may not be even able to see them.

Currently, ES is only as fast as the slowest node's response time.

That said, have you tried looking at the slow log?

3

Yes, I've taken a look at the slow log now (although I'm not super familiar with it) and they are empty on all nodes. Any other places to look? I've upped that last machine and all of the machines to 12GB RAM 2 cores on each machine but the searches in Discover are still very unresponsive. I've set the m_lock but that is about it for the optimization efforts.

Would a lot of historical data slow things down? I have about half of the total size of my cluster's indices with data from 3 months ago that I don't necessarily use but could be useful to have; especially because I have the space. My impression is that only the shards that are relevant to the timeframe you define are searched and the other indices are irrelevant.

Interestingly, on some dashboards I have with about 6 visuals, for a search across 24 hours (through about 60k logs only) I have around a 69 ms query duration and a 15 s request duration. Does that point to kibana going slow? How do I improve Kibana's performance?

Oh and sorry about the screenshot but pasting the results looked absolutely awful, but I'll try not to do that if I can avoid it :slight_smile:

4

It'll be ES as KB is really a front end to the APIs.

Are you monitoring ES at all?

5

My goal is monitoring elastic which i was trying to do with my first post:
understand the ram usage. Are there other ways to monitor elastic? I've
been watching the normal logs and trying to watch the ram usage as i send
different requests and seeing how ES responds. Is that the best method?

6

The best method is https://www.elastic.co/guide/en/x-pack/current/xpack-monitoring.html

7

Are there other options or commands I can use? I've already use my trial a
few months back and can't afford x-pack. Way above my price range...

8

Monitoring is free :slight_smile:

9

So to confirm, if my x-pack trial is expired on my cluster. Monitoring will still work, just Watcher, Shield, etc.. won't? That would be great news!

10

That is correct, it's part of our basic license - https://www.elastic.co/subscriptions

11

But you will have to ask for a basic license. Which is free.

12

Just to follow up and give some feedback. I have my cluster in 5.2.2.

The installation of the license is not intuitive and I had a very hard time getting it to work (I don't think I ever did.) I tried posting it from Kibana itself and tried to do it using the instructions from the docs of

curl -XPUT -u elastic 'http://host:9200/_xpack/license' -H "Content-Type: application/json" -d @license.json

Variations were to remove the -u and my license.json is located in the root "/" directory where I was calling the command. I dl'ed from the link I got, and copied the contents into that file.

and kept getting errors:

{"error":{"root_cause":[{"type":"parse_exception","reason":"Failed to derive xcontent"}],"type":"parse_exception","reason":"Failed to derive xcontent"},"status":400}curl: (6) Could not resolve host: license.json; Name or service not known

or

{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [/_xpack/license]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"missing authentication token for REST request [/_xpack/license]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}curl: (6) Could not resolve host: license.json; Name or service not known

or

curl: (6) Could not resolve host: elastic; Unknown error

I looked online but this is really overwhelming when trying to do something as simple as install a license.

I then thought to move on and see if I can just get things to run and focused on my kibana instance. When I tried to disable xpack.security, I got this error

{"type":"log","@timestamp":"2017-05-30T05:48:49Z","tags":["fatal"],"pid":4587,"level":"fatal","message":"EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'","error":{"message":"EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'","name":"Error","stack":"Error: EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'\n at Error (native)","code":"EACCES"}}
FATAL { Error: EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'
at Error (native)
cause:
{ Error: EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'
at Error (native)
errno: -13,
code: 'EACCES',
syscall: 'open',
path: '/usr/share/kibana/optimize/bundles/monitoring.entry.js' },
isOperational: true,
errno: -13,
code: 'EACCES',

I know there are tickets for this error, but I'm kind of stuck because I can't change versions currently and I'm not that confident things will actually work if I do.

Then I set in kibana.yml
xpack.security.enabled: false
xpack.graph.enabled: false
xpack.monitoring.enabled: false
and I got:

{"type":"error","@timestamp":"2017-05-30T06:05:43Z","tags":,"pid":8322,"level":"error","message":"Cannot read property 'toJSON' of undefined","error":{"message":"Cannot read property 'toJSON' of undefined","name":"TypeError","stack":"TypeError: Cannot read property 'toJSON' of undefined\n at withXpackInfo (/usr/share/kibana/plugins/x-pack/plugins/xpack_main/server/lib/replace_injected_vars.js:5:32)\n at replaceInjectedVars$ (/usr/share/kibana/plugins/x-pack/plugins/xpack_main/server/lib/replace_injected_vars.js:10:12)\n at tryCatch (/usr/share/kibana/node_modules/regenerator/runtime.js:61:40)\n at GeneratorFunctionPrototype.invoke [as _invoke] (/usr/share/kibana/node_modules/regenerator/runtime.js:328:22)\n at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (/usr/share/kibana/node_modules/regenerator/runtime.js:94:21)\n at invoke (/usr/share/kibana/node_modules/regenerator/runtime.js:136:37)\n at callInvokeWithMethodAndArg (/usr/share/kibana/node_modules/regenerator/runtime.js:172:16)\n at previousPromise (/usr/share/kibana/node_modules/regenerator/runtime.js:194:19)\n at AsyncIterator.enqueue (/usr/share/kibana/node_modules/regenerator/runtime.js:193:13)\n at AsyncIterator.prototype.(anonymous function) [as next] (/usr/share/kibana/node_modules/regenerator/runtime.js:94:21)\n at Object.runtime.async (/usr/share/kibana/node_modules/regenerator/runtime.js:215:14)\n at replaceInjectedVars (/usr/share/kibana/plugins/x-pack/plugins/xpack_main/server/lib/replace_injected_vars.js:3:22)\n at /usr/share/kibana/src/ui/index.js:67:22\n at undefined.next (native)\n at step (/usr/share/kibana/src/ui/index.js:9:273)\n at /usr/share/kibana/src/ui/index.js:9:443\n at /usr/share/kibana/src/ui/index.js:9:99\n at tryCatcher (/usr/share/kibana/node_modules/bluebird/js/main/util.js:26:23)\n at ReductionPromiseArray._promiseFulfilled (/usr/share/kibana/node_modules/bluebird/js/main/reduce.js:109:18)\n at ReductionPromiseArray.init (/usr/share/kibana/node_modules/bluebird/js/main/promise_array.js:92:18)\n at ReductionPromiseArray.init (/usr/share/kibana/node_modules/bluebird/js/main/reduce.js:42:10)\n at Async._drainQueue (/usr/share/kibana/node_modules/bluebird/js/main/async.js:128:12)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"","query":{},"pathname":"/app/kibana","path":"/app/kibana","href":"/app/kibana"}}
May 30 02:05:43 KOFI03ES-Kibana kibana: {"type":"response","@timestamp":"2017-05-30T06:05:43Z","tags":,"pid":8322,"method":"get","statusCode":500,"req":{"url":"/app/kibana","method":"get","headers":{"host":"172.28.97.58:5601","connection":"keep-alive","cache-control":"max-age=0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8","referer":"http://172.28.97.58:5601/","accept-encoding":"gzip, deflate, sdch","accept-language":"en-US,en;q=0.8"},"remoteAddress":"172.29.10.87","userAgent":"172.29.10.87","referer":"http://172.28.97.58:5601/"},"res":{"statusCode":500,"responseTime":29,"contentLength":9},"message":"GET /app/kibana 500 29ms - 9.0B"}

so as a last ditch method, I allowed security, graph and monitoring to run and put everything in kibana.yml as default and things finally seemed to bundle, then when I tried to login to kibana I got

{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}

This is all very exhausting. I'm just reporting what I've gotten so you guys can take a look to see if these are old news or not. I think I'll just remove x-pack and not be able to do any sort of monitoring which is a shame...

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.