fnork
April 3, 2018, 2:03pm
1
We have an 8 node cluster with about 100 indexes and 1000 shards.
The node with the primary marvel index shard shows a running >80% cpu usage (in marvel :), while indexing about 60 docs/sec.
hot threads shows the marvel-exporter thread using about 30%. I suppose the other cpu usage is indexing?
Is this level of cpu usage normal for marvel or is there something in our cluster that causes excessive cpu use?
-andré
spinscale
(Alexander Reelsen)
April 4, 2018, 10:40am
2
can you share the hot threads output in a gist?
That said, you may also want to reduce your number of shards, and especially you may want to think about upgrading, if possible. As there have been a lot of changes in the two major releases in between and APIs that can help you like the shrink index API.
fnork
April 4, 2018, 10:52am
3
Thanks for your reply. Here's the hot threads output for the node.
gistfile1.txt
Hot threads at 2018-04-04T10:45:01.599Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
54.1% (270.4ms out of 500ms) cpu usage by thread 'elasticsearch[node-p8][marvel.exporters]'
2/10 snapshots sharing following 27 elements
java.lang.String.intern(Native Method)
com.fasterxml.jackson.core.util.InternCache.intern(InternCache.java:59)
com.fasterxml.jackson.core.sym.ByteQuadsCanonicalizer.addName(ByteQuadsCanonicalizer.java:816)
com.fasterxml.jackson.core.json.UTF8StreamJsonParser.addName(UTF8StreamJsonParser.java:2390)
com.fasterxml.jackson.core.json.UTF8StreamJsonParser.findName(UTF8StreamJsonParser.java:2274)
com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseLongName(UTF8StreamJsonParser.java:1854)
This file has been truncated. show original
I realize there are a lot of shards and that we let marvel index back into the production cluster, but the cpu load still seems a bit high to me.
Can the free Marvel export to a separate cluster or is a full license needed for that?
We need the Marvel logging for debugging right now, but we could possible increase the metric collection interval when we are done with that.
-andré
spinscale
(Alexander Reelsen)
April 4, 2018, 11:06am
4
Is it possible that you have a big cluster state?
fnork
April 4, 2018, 11:07am
5
Yes we have enormous mappings for some of the indexes.
There is an unfortunate dynamic data scheme in this application.
_mapping returns 30MB of json. By a rough grep one index has about 40k fields and there are probably 10-20 with 5k to 15k fields.
-andré
spinscale
(Alexander Reelsen)
April 4, 2018, 11:48am
6
then just serializing this cluster state seems to need a lot of CPU according to hot_threads - which makes sense to me.
fnork
April 4, 2018, 12:19pm
7
ok
is this independent of marvel-agent or is it something marvel-agent does?
-andré
spinscale
(Alexander Reelsen)
April 4, 2018, 3:12pm
8
The marvel agent reads the cluster state and serializes parts of it (using filters) - I havent looked up how the filters are working, but more efficient filtering using jackson has been added much later than 2.4.
--Alex
fnork
April 5, 2018, 11:01am
9
ok thanks for taking the time to look into this
-andré
system
(system)
Closed
May 3, 2018, 11:01am
10
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.