Delete_by_query? , gateway timeout


(Roque Moyano) #1

Hi , I have a problem , I have some huge indexes , and I want to delete some documents but only if they have some value , I have tried it:

POST docker-2018.07.05/_delete_by_query?conflicts=proceed&timeout=10m
{
"query": {
"match": {
"TIPO": "INFO"
}
}
}

but it fails for timeout , how can I fix it?

thanks in advance.


(Tamara Rosini) #2

Hi,

You are having the fail because you defined the timeout in the url. You can do like this to have the timeout set to 20 minutes:

POST docker-2018.07.05/_delete_by_query?conflicts=proceed&timeout=20m
{
  "query": {
    "match": {
     "TIPO": "INFO"
     }
   }
}

Change the timeout=20m to any value that fits your case, the default value is 5 minutes.

If you need more information you can find at:
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html


(David Pilato) #3

What is the output of:

POST docker-2018.07.05/_search?size=1
{
  "query": {
    "match": {
      "TIPO": "INFO"
    }
  }
}

(Roque Moyano) #4

the output is:

{
"took": 19,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 3827639,
"max_score": 0.000890049,
"hits": [
{
"_index": "docker-2018.07.06",
"_type": "doc",
"_id": "8hfkbGQBCKk4lo83XTkW",
"_score": 0.000890049,
"_source": {
"zabbix_host": "elk01",
"TIPO": "INFO",
"message": "[2018-07-05T21:00:59] | [PROD] | [WSRESTTRAMITES] | [INFO] | [c.w.f.TokenFilter] | [tokenHeader: 0ec078efabf7d9694f19e2dda7f3b18967e40e7508097a89e9f2e412384f81e3]",
"@version": "1",
"offset": 260054091,
"tags": [
"beats_input_codec_plain_applied"
],
"MENSAJE": "tokenHeader: 0ec078efabf7d9694f19e2dda7f3b18967e40e7508097a89e9f2e412384f81e3",
"zkey": "dataelk",
"zabbix_value": "PROD - WSRESTTRAMITES - tokenHeader: 0ec078efabf7d9694f19e2dda7f3b18967e40e7508097a89e9f2e412384f81e3",
"FECHA": "2018-07-05T21:00:59",
"CLASE": "c.w.f.TokenFilter",
"input": {
"type": "log"
},
"APPLICATION": "WSRESTTRAMITES",
"source": "/var/log/dockerContainers/containers/j8tomcat8_wsresttramites01/catalina.log",
"prospector": {
"type": "log"
},
"beat": {
"name": "kdsflpdock003.kolektor.com.ar",
"hostname": "kdsflpdock003.kolektor.com.ar",
"version": "6.3.0"
},
"@timestamp": "2018-07-06T00:01:01.659Z",
"ENV": "PROD"
}
}
]
}
}


(David Pilato) #5

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.

What is the output of:

POST docker-2018.07.05/_search?size=0

(Roque Moyano) #6

sorry , this is the output for size=0

  {
  "took": 16,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 3827639,
    "max_score": 0,
    "hits": []
  }
}

(David Pilato) #7

So you have 3827639 documents and you want to remove 3827639 documents?

Just drop the index then!


(Roque Moyano) #8

I just want to remove the documents where the fields "TIPO" has the value "INFO" , the index has a lot of information that is necessary to keep , for that I can't remove the index.


(David Pilato) #9

DBQ Will remove the documents not only one field.


(Roque Moyano) #10

sorry my ignorance , what is DBQ?


(David Pilato) #11

Sorry. Delete By Query


(Roque Moyano) #12

yes but we are in the same step , it fails by timeout , is there any way that I can move some documents to a new index? if do i can remove the index


(David Pilato) #13

Yes. Use reindex API.

But as I said if you intend to remove 100% documents, just drop the index.


(Roque Moyano) #14

thanks man ! I just move the information that I need to a new index and then remove it.

POST /_reindex
    {
      "source": {
        "index": "docker-2018.08.01",
        "query": {
          "match": {
            "TIPO": "ERROR"
          }
        }
      },
      "dest": {
        "index": "docker-errors-2018.08.01"
      }
    }

Best Regards!


(system) #15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.