[resolved] Delete_by_query using elasticsearch-py

Elastic Stack Elasticsearch
1

Hello Guys,
I tried delete_by_query using using Python , because I need automation some task in my ELK cluster.
Now I'm using a simple script in bash with the following code:

#!/bin/bash

/usr/bin/curl -XDELETE http://localhost:9200/logstash-2016.06.14/Nxlogs/_query?pretty -d '{ "query":{"term":{"hostname.raw":{"value":"'"${SERVER-01}"'"}}}}'

So, I tried in some ways copy the same query in Python, like a documentation:
perform_request(method, url, params=None, body=None)

import elasticsearch
es = elasticsearch.Elasticsearch(
    ['node-01:9200'],
    # sniff before doing anything
    sniff_on_start=True,
    # refresh nodes after a node fails to respond
    sniff_on_connection_fail=True,
    # and also every 60 seconds
    sniffer_timeout=60
)

#First Try
    data = es.transport.perform_request('DELETE','logstash-2016.04.05/Nxlogs/_query',body={"query":{"term":{"hostname.raw":{"value":"SERVER-01"}}}}
    )

#Second Try:

data = es.transport.perform_request('DELETE','http://localhost:9200/logstash-2016.04.05/Nxlogs/_query',body={"query":{"term":{"hostname.raw":{"value":"SERVER-01"}}}}
)

The error is the same for both cases:

HostChangedError                          Traceback (most recent call last)
C:\Anaconda3\lib\site-packages\elasticsearch\connection\http_urllib3.py in perform_request(self, method, url, params, body, timeout, ignore)
     93 
---> 94             response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
     95             duration = time.time() - start

C:\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, **response_kw)
    543         if assert_same_host and not self.is_same_host(url):
--> 544             raise HostChangedError(self, url, retries)
    545 

HostChangedError: HTTPConnectionPool(host='10.101.78.53', port=9200): Tried to open a foreign host with url: http://localhost:9200/logstash-2016.04.05/Nxlogs/_query

During handling of the above exception, another exception occurred:

ConnectionError                           Traceback (most recent call last)
<ipython-input-1-1827e5f7db91> in <module>()
     10 )
     11 
---> 12 data = es.transport.perform_request('DELETE','http://localhost:9200/logstash-2016.04.05/Nxlogs/_query',body={"query":{"term":{"hostname.raw":{"value":"SERVER-01"}}}}
     13 )

C:\Anaconda3\lib\site-packages\elasticsearch\transport.py in perform_request(self, method, url, params, body)
    327 
    328             try:
--> 329                 status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
    330 
    331             except TransportError as e:

C:\Anaconda3\lib\site-packages\elasticsearch\connection\http_urllib3.py in perform_request(self, method, url, params, body, timeout, ignore)
    103         except Exception as e:
    104             self.log_request_fail(method, full_url, body, time.time() - start, exception=e)
--> 105             raise ConnectionError('N/A', str(e), e)
    106 
    107         if not (200 <= response.status < 300) and response.status not in ignore:

ConnectionError: ConnectionError(HTTPConnectionPool(host='10.101.78.53', port=9200): Tried to open a foreign host with url: http://localhost:9200/logstash-2016.04.05/Nxlogs/_query) caused by: HostChangedError(HTTPConnectionPool(host='10.101.78.53', port=9200): Tried to open a foreign host with url: http://localhost:9200/logstash-2016.04.05/Nxlogs/_query)

Anyone has tried delete by query plugin?.

Thank you.

2

Disable sniffing and see how it behaves.

3

I did , my code is the following now:

from elasticsearch import Elasticsearch 

es = elasticsearch.Elasticsearch(['localhost:9200'])

data = es.transport.perform_request('DELETE','logstash-2016.06.05/Nxlogs/',body={"query":{"term":{"hostname.raw":{"value":"SERVER-01"}}}}
)

The error is the same:
ConnectionError: ConnectionError(HTTPConnectionPool(host='localhost', port=9200): Tried to open a foreign host with url: logstash-2016.06.05/Nxlogs/) caused by: HostChangedError(HTTPConnectionPool(host='localhost', port=9200): Tried to open a foreign host with url: logstash-2016.06.05/Nxlogs/)

4

It certainly appears that you are passing a URL without the host, as documented, but it's being interpreted as the host.

Investigating.

5

It was resolved:

data = es.transport.perform_request('DELETE','/logstash-2016.06.05/Nxlogs/_query',body={"query":{"term":{"hostname.raw":{"value":"SERVER-01"}}}}
)

I tried find the server in the index by Kibana and no one is there.

Thank you for you help Glen.

6

Had the same problem.

Just for clarity, the fix is to start the URL with a / (as seen in Juan Andres's latest example)

good:
es.transport.perform_request('DELETE','/logsta...)

bad:
es.transport.perform_request('DELETE','logsta...)