High cpu usage with marvel-agent (2.4)

Elastic Stack Elasticsearch
1

We have an 8 node cluster with about 100 indexes and 1000 shards.

The node with the primary marvel index shard shows a running >80% cpu usage (in marvel :), while indexing about 60 docs/sec.

hot threads shows the marvel-exporter thread using about 30%. I suppose the other cpu usage is indexing?

Is this level of cpu usage normal for marvel or is there something in our cluster that causes excessive cpu use?

-andré

2

can you share the hot threads output in a gist?

That said, you may also want to reduce your number of shards, and especially you may want to think about upgrading, if possible. As there have been a lot of changes in the two major releases in between and APIs that can help you like the shrink index API.

3

Thanks for your reply. Here's the hot threads output for the node.

I realize there are a lot of shards and that we let marvel index back into the production cluster, but the cpu load still seems a bit high to me.

Can the free Marvel export to a separate cluster or is a full license needed for that?

We need the Marvel logging for debugging right now, but we could possible increase the metric collection interval when we are done with that.

-andré

4

Is it possible that you have a big cluster state?

5

Yes we have enormous mappings for some of the indexes.

There is an unfortunate dynamic data scheme in this application.

_mapping returns 30MB of json. By a rough grep one index has about 40k fields and there are probably 10-20 with 5k to 15k fields.

-andré

6

then just serializing this cluster state seems to need a lot of CPU according to hot_threads - which makes sense to me.

7

ok

is this independent of marvel-agent or is it something marvel-agent does?

-andré

8

The marvel agent reads the cluster state and serializes parts of it (using filters) - I havent looked up how the filters are working, but more efficient filtering using jackson has been added much later than 2.4.

--Alex

9

ok thanks for taking the time to look into this :slight_smile:

-andré

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.