High cpu usage with marvel-agent (2.4)

fnork · April 3, 2018, 2:03pm

We have an 8 node cluster with about 100 indexes and 1000 shards.

The node with the primary marvel index shard shows a running >80% cpu usage (in marvel :), while indexing about 60 docs/sec.

hot threads shows the marvel-exporter thread using about 30%. I suppose the other cpu usage is indexing?

Is this level of cpu usage normal for marvel or is there something in our cluster that causes excessive cpu use?

-andré

spinscale · April 4, 2018, 10:40am

can you share the hot threads output in a gist?

That said, you may also want to reduce your number of shards, and especially you may want to think about upgrading, if possible. As there have been a lot of changes in the two major releases in between and APIs that can help you like the shrink index API.

fnork · April 4, 2018, 10:52am

Thanks for your reply. Here's the hot threads output for the node.

gist.github.com

https://gist.github.com/andrely/7d2368f8ad70bc916909bffb0f356671

gistfile1.txt

   Hot threads at 2018-04-04T10:45:01.599Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

   54.1% (270.4ms out of 500ms) cpu usage by thread 'elasticsearch[node-p8][marvel.exporters]'
     2/10 snapshots sharing following 27 elements
       java.lang.String.intern(Native Method)
       com.fasterxml.jackson.core.util.InternCache.intern(InternCache.java:59)
       com.fasterxml.jackson.core.sym.ByteQuadsCanonicalizer.addName(ByteQuadsCanonicalizer.java:816)
       com.fasterxml.jackson.core.json.UTF8StreamJsonParser.addName(UTF8StreamJsonParser.java:2390)
       com.fasterxml.jackson.core.json.UTF8StreamJsonParser.findName(UTF8StreamJsonParser.java:2274)
       com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseLongName(UTF8StreamJsonParser.java:1854)

This file has been truncated. show original

I realize there are a lot of shards and that we let marvel index back into the production cluster, but the cpu load still seems a bit high to me.

Can the free Marvel export to a separate cluster or is a full license needed for that?

We need the Marvel logging for debugging right now, but we could possible increase the metric collection interval when we are done with that.

-andré

spinscale · April 4, 2018, 11:06am

Is it possible that you have a big cluster state?

fnork · April 4, 2018, 11:07am

Yes we have enormous mappings for some of the indexes.

There is an unfortunate dynamic data scheme in this application.

_mapping returns 30MB of json. By a rough grep one index has about 40k fields and there are probably 10-20 with 5k to 15k fields.

-andré

spinscale · April 4, 2018, 11:48am

then just serializing this cluster state seems to need a lot of CPU according to hot_threads - which makes sense to me.

fnork · April 4, 2018, 12:19pm

ok

is this independent of marvel-agent or is it something marvel-agent does?

-andré

spinscale · April 4, 2018, 3:12pm

The marvel agent reads the cluster state and serializes parts of it (using filters) - I havent looked up how the filters are working, but more efficient filtering using jackson has been added much later than 2.4.

--Alex

fnork · April 5, 2018, 11:01am

ok thanks for taking the time to look into this

-andré

system · May 3, 2018, 11:01am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.