Delete_by_query? , gateway timeout

Elastic Stack Elasticsearch
1

Hi , I have a problem , I have some huge indexes , and I want to delete some documents but only if they have some value , I have tried it:

POST docker-2018.07.05/_delete_by_query?conflicts=proceed&timeout=10m
{
"query": {
"match": {
"TIPO": "INFO"
}
}
}

but it fails for timeout , how can I fix it?

thanks in advance.

2

Hi,

You are having the fail because you defined the timeout in the url. You can do like this to have the timeout set to 20 minutes:

POST docker-2018.07.05/_delete_by_query?conflicts=proceed&timeout=20m
{
  "query": {
    "match": {
     "TIPO": "INFO"
     }
   }
}

Change the timeout=20m to any value that fits your case, the default value is 5 minutes.

If you need more information you can find at:
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html

3

What is the output of:

POST docker-2018.07.05/_search?size=1
{
  "query": {
    "match": {
      "TIPO": "INFO"
    }
  }
}
4

the output is:

{
"took": 19,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 3827639,
"max_score": 0.000890049,
"hits": [
{
"_index": "docker-2018.07.06",
"_type": "doc",
"_id": "8hfkbGQBCKk4lo83XTkW",
"_score": 0.000890049,
"_source": {
"zabbix_host": "elk01",
"TIPO": "INFO",
"message": "[2018-07-05T21:00:59] | [PROD] | [WSRESTTRAMITES] | [INFO] | [c.w.f.TokenFilter] | [tokenHeader: 0ec078efabf7d9694f19e2dda7f3b18967e40e7508097a89e9f2e412384f81e3]",
"@version": "1",
"offset": 260054091,
"tags": [
"beats_input_codec_plain_applied"
],
"MENSAJE": "tokenHeader: 0ec078efabf7d9694f19e2dda7f3b18967e40e7508097a89e9f2e412384f81e3",
"zkey": "dataelk",
"zabbix_value": "PROD - WSRESTTRAMITES - tokenHeader: 0ec078efabf7d9694f19e2dda7f3b18967e40e7508097a89e9f2e412384f81e3",
"FECHA": "2018-07-05T21:00:59",
"CLASE": "c.w.f.TokenFilter",
"input": {
"type": "log"
},
"APPLICATION": "WSRESTTRAMITES",
"source": "/var/log/dockerContainers/containers/j8tomcat8_wsresttramites01/catalina.log",
"prospector": {
"type": "log"
},
"beat": {
"name": "kdsflpdock003.kolektor.com.ar",
"hostname": "kdsflpdock003.kolektor.com.ar",
"version": "6.3.0"
},
"@timestamp": "2018-07-06T00:01:01.659Z",
"ENV": "PROD"
}
}
]
}
}

5

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

image
image.jpg764×92 16.4 KB

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.

What is the output of:

POST docker-2018.07.05/_search?size=0
6

sorry , this is the output for size=0

  {
  "took": 16,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 3827639,
    "max_score": 0,
    "hits": []
  }
}
7

So you have 3827639 documents and you want to remove 3827639 documents?

Just drop the index then!

8

I just want to remove the documents where the fields "TIPO" has the value "INFO" , the index has a lot of information that is necessary to keep , for that I can't remove the index.

9

DBQ Will remove the documents not only one field.

10

sorry my ignorance , what is DBQ?

11

Sorry. Delete By Query

12

yes but we are in the same step , it fails by timeout , is there any way that I can move some documents to a new index? if do i can remove the index

13

Yes. Use reindex API.

But as I said if you intend to remove 100% documents, just drop the index.

14

thanks man ! I just move the information that I need to a new index and then remove it.

POST /_reindex
    {
      "source": {
        "index": "docker-2018.08.01",
        "query": {
          "match": {
            "TIPO": "ERROR"
          }
        }
      },
      "dest": {
        "index": "docker-errors-2018.08.01"
      }
    }

Best Regards!

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.