[esaggs] > too many requests is showing on dashboard

[Rohit_Kumbhar]

Hi,

I am getting [esaggs] > too many requests on my Dashboard.
Cannot find any solution for that.
Please help

Thanks .
Regards Rohit.

[warkolm (Mark Walkom)]

Unfortunately you haven't really given us much to work with here.

What version of the stack are you using?
When do you see this error?
Can you share a screenshot?
Is there anything in the Kibana or Elasticsearch logs that corresponds with the time you see this?
Is there anything in your browser's console?

[Rohit_Kumbhar]

Yes Sure

I am using ELK Stack 7.9.2
I see this error when i increase my Timerange in Kibana.
Attaching Screenshot.
When i hover over it says [esaggs] > too many requests

Thanks
Regards Rohit

[warkolm (Mark Walkom)]

@warkolm

Is there anything in the Kibana or Elasticsearch logs that corresponds with the time you see this?
Is there anything in your browser's console?

These are important please.

[Rohit_Kumbhar]

On elasticsearch

    [master-1] unexpected error while indexing monitoring document
    org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[index [.monitoring-kibana-7-2021.01.13] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

On Kibana

\"reason\\\":\\\"[parent] Data too large, data for [indices:data/read/search[phase/query]] would be [6030980140/5.6gb], which is larger than the limit of [6012954214/5.5gb], real usage: [6030971392/5.6gb], new bytes reserved: [8748/8.5kb]

Thanks for quick reply
Regards Rohit.

[warkolm (Mark Walkom)]

@Rohit_Kumbhar

disk usage exceeded flood-stage watermark, index has read-only-allow-delete block

Looks like your Elasticsearch node is out of disk.

[Rohit_Kumbhar]

Yes,
Now i want to Delete some of the Index data which is like 400gb in 1 index
and want to delete data of 4-5 months .

I used /delete_by_query API

POST index_name/_delete_by_query?wait_for_completion=true&conflicts=proceed&scroll=10m
{
"query": {
"range" : {
"@timestamp" : {
"gte" : "2020-04-15T00:00:00.000Z",
"lte" : "2020-08-17T00:00:00.000Z"
}
}
}
}

But its Version Conflict because i upgraded from 7.6.2 to 7.9.2.

Any solution??

Thanks

[warkolm (Mark Walkom)]

If your data is time based, you should be really using ILM.
Delete by query is very inefficient.

@Rohit_Kumbhar

But its Version Conflict because i upgraded from 7.6.2 to 7.9.2.

What do you mean by that? Are you getting an error? If so please post the response you are seeing.

[Rohit_Kumbhar]

{
  "nodes" : {
"QXaUm3G3T9qitukM7Z3SRw" : {
  "name" : "master-1",
  "transport_address" : "192.IP:9300",
  "host" : "192.IP",
  "ip" : "192.IP:9300",
  "roles" : [
    "data",
    "ingest",
    "master",
    "ml",
    "remote_cluster_client",
    "transform"
  ],
  "attributes" : {
    "ml.machine_memory" : "16657059840",
    "xpack.installed" : "true",
    "transform.node" : "true",
    "ml.max_open_jobs" : "20"
  },
  "tasks" : {
    "QXaUm3G3T9qitukM7Z3SRw:1954905" : {
      "node" : "QXaUm3G3T9qitukM7Z3SRw",
      "id" : 1954905,
      "type" : "transport",
      "action" : "indices:data/write/delete/byquery",
      "status" : {
        "total" : 23114241,
        "updated" : 0,
        "created" : 0,
        "deleted" : 466,
        "batches" : 4,
        "version_conflicts" : 2534,
        "noops" : 0,
        "retries" : {
          "bulk" : 0,
          "search" : 0
        },
        "throttled_millis" : 0,
        "requests_per_second" : -1.0,
        "throttled_until_millis" : 0
      },
      "description" : "delete-by-query [index]",
      "start_time_in_millis" : 1610513022750,
      "running_time_in_nanos" : 83108269262,
      "cancellable" : true,
      "headers" : { }
    },

Here output of /delete_by_query API

"version_conflicts" : 2534,