High cpu usage with marvel-agent (2.4)

We have an 8 node cluster with about 100 indexes and 1000 shards.

The node with the primary marvel index shard shows a running >80% cpu usage (in marvel :), while indexing about 60 docs/sec.

hot threads shows the marvel-exporter thread using about 30%. I suppose the other cpu usage is indexing?

Is this level of cpu usage normal for marvel or is there something in our cluster that causes excessive cpu use?

-andré

can you share the hot threads output in a gist?

That said, you may also want to reduce your number of shards, and especially you may want to think about upgrading, if possible. As there have been a lot of changes in the two major releases in between and APIs that can help you like the shrink index API.

Thanks for your reply. Here's the hot threads output for the node.

I realize there are a lot of shards and that we let marvel index back into the production cluster, but the cpu load still seems a bit high to me.

Can the free Marvel export to a separate cluster or is a full license needed for that?

We need the Marvel logging for debugging right now, but we could possible increase the metric collection interval when we are done with that.

-andré

Is it possible that you have a big cluster state?

Yes we have enormous mappings for some of the indexes.

There is an unfortunate dynamic data scheme in this application.

_mapping returns 30MB of json. By a rough grep one index has about 40k fields and there are probably 10-20 with 5k to 15k fields.

-andré

then just serializing this cluster state seems to need a lot of CPU according to hot_threads - which makes sense to me.

ok

is this independent of marvel-agent or is it something marvel-agent does?

-andré

The marvel agent reads the cluster state and serializes parts of it (using filters) - I havent looked up how the filters are working, but more efficient filtering using jackson has been added much later than 2.4.

--Alex

ok thanks for taking the time to look into this :slight_smile:

-andré

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.