Randomly performance issue

Elastic Stack Elasticsearch
1

Hello All,

We are randomly facing performance issue in elasticsearch read/write operations. The elasticsearch in installed on Azure VM (Linux). This is not constant. Generally a common method to insert/update in elasticsearch takes 1 to 5 secs, but when we face the issue the general operation time reach to 2 to 4 mins.

We are using hang-fire to update the elasticsearch indexes.

Here are the configuration on our VM ->
CPU : 4
RAM : 16GB
Data Disks - 8
Max IOPS : 6400
Storage : 32 GB

Please guide what can be the issue and how to monitor what was the IOPS, Memory usages etc.

Thanks in advance.

2

Hi @nmathur,

I would advise to first take a look at the Elasticsearch log file and GC log files to see if the JVM is memory overloaded or something stands out.

Additionally, looking into the _nodes/hot_threads can reveal what Elasticsearch is spending time on.

You can see current heap usage and many other metrics in _nodes/stats.

A single request that normally takes 1 to 5 seconds sounds like a big request. Maybe splitting it into smaller requests is better?

3

Thanks for reply.

I checked elasticsearch log file and there was X-Pack license expiry exception. We are not using X-Pack so we disabled it. As you suggested, I will check other area also and share what I get.

The 1 to 5 sec call also includes database query. Each elasticsearch type have combined data of 2 or 3 tables. When our call taking minutes, we majored our database and it was working fine. So the issue can be in Elasticsearch call.

I will share log details.

Again, thanks for the reply.

4

Hello,

As I shared earlier we checked elasticsearch log and disabled the X-Pack. I am sharing yesterday's log here:

[2020-01-16T00:00:02,639][INFO ][o.e.c.m.MetaDataCreateIndexService] [elasticindex-0] [.monitoring-es-6-2020.01.16] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[1], mappings [doc]
[2020-01-16T00:00:17,738][WARN ][o.e.m.j.JvmGcMonitorService] [elasticindex-0] [gc][43399] overhead, spent [4.7s] collecting in the last [5.6s]
[2020-01-16T00:00:47,078][WARN ][o.e.m.j.JvmGcMonitorService] [elasticindex-0] [gc][43425] overhead, spent [3.7s] collecting in the last [4.2s]
[2020-01-16T00:01:06,939][WARN ][o.e.l.LicenseService ] [elasticindex-0]
[2020-01-16T00:01:22,667][WARN ][o.e.m.j.JvmGcMonitorService] [elasticindex-0] [gc][43456] overhead, spent [4.8s] collecting in the last [5.5s]

You said that if single request takes 1 to 5 sec then request is big, from last few days a call of a fresh index which barely has 10 record is also taking 4-5 secs (we monitored this on application insight).

In my post I forgot to share my elasticsearch version, it is 5.5. I am using Nest for queries.

"_nodes/hot_threads" does not returning much information:
::: {elasticindex-0}{*****}{}{...}{...:9300}{ml.enabled=true}
Hot threads at 2020-01-17T09:32:17.429Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

I think I have configuration related issues, initially it was working fine .. now when our users are increasing day by day then we are facing this issue.

Can you please guide me what I should share here to show my configuration details and can you suggest any change.

One point I need to share here: RAM on our VM was RAM : 16GB but our heap size was default
-Xms4g
-Xmx4g

we increased it to
-Xms8g
-Xmx8g

Still our query is taking time.

Thanks in advance.

5

Hi @nmathur,

it is normally advisable to index monitoring data into a separate cluster. Whether that is part of it or not is hard to say, but if you can use a separate monitoring cluster, we can at least rule that out.

Also, you wrote:

When you say "fresh index", do you mean an index that did not exist and was autocreated by the bulk request? That makes sense, since such an operation takes considerably more time to both create the index and the dynamic mappings (if not using a template including mappings of all fields). Those operations update the cluster state, which is a more expensive operation than normal indexing of docs.

It might be worth looking at the pending cluster tasks when experiencing this problem.

6

Sorry, I used the incorrect word, it is not fresh index .. it is a new index which I created recently and it is related to my new module and has few records only. Still, the search request is taking 4 to 5 secs.

As I previously said, It can be configuration or setup related issue .. so I am sharing the create index code:

      var descriptor = new CreateIndexDescriptor("asset_index")
              .Settings(s => s

                   .Analysis(a => a
                       .Analyzers(an => an
                           .Custom("case_insensitive", ca => ca
                               .Tokenizer("keyword")
                               .Filters("lowercase")
                           )
                       )
                   ))
           .Mappings(ms => ms.Map<Asset>("asset", m => m.AutoMap().Properties(p => p
                    .Text(t => t.Name(n => n.Name).Analyzer("case_insensitive").Fielddata(true))
                    .Text(t => t.Name(n => n.RegistrationNumber).Analyzer("case_insensitive").Fielddata(true))
                    .Text(t => t.Name(n => n.Code).Analyzer("case_insensitive").Fielddata(true))
                    .Text(t => t.Name(n => n.Description).Fielddata(true))
                    .Text(t => t.Name(n => n.CreateByName).Analyzer("case_insensitive").Fielddata(true))
                    .Text(t => t.Name(n => n.ModifyByName).Analyzer("case_insensitive").Fielddata(true))
            )));

I am using Nest 5.0 .. I always create the index manually.

index monitoring data into a separate cluster

I am not using any monitoring, the X-Pack was installed by default, I never used it .. can you guide us more?

Thanks.

7

Hi @nmathur ,

to double check that the request did arrive at Elasticsearch, you can use the tasks API to list the tasks that Elasticsearch is working on.

Simply doing GET _tasks should display this.

About monitoring, I believe you can disable it on your 5.5 release by setting:
xpack.monitoring.enabled=false
See documentation here.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.