Connecting an ES client with a kibana on Python

Elastic Security
1

This is the problem: I need Kibana and the ES client to work on python at the same time.
At the same time, all security settings are enabled:

xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
 enabled: true
xpack.security.transport.ssl:
 enabled: true

1. Standard Authentication

I tried to do it the usual way, through authentication:

aioes = AsyncElasticsearch(
    hosts='http://localhost:9200',
    basic_auth=("user", "password")
)

But the client cannot reach the host. Disconnect is happening.
The 2nd security parameter is responsible for this. When it is disabled, the ES client works, but the kibana crashes. And vice versa

2. Generate Api Key

I tried to solve this problem through the api key.

In kibana, I made such a request:

POST _security/api_key
{
  "name": "ES_client",
  "role_descriptors": {
    "es_python_client": {
      "cluster": ["all"],
      "index": [
        {
          "names": ["index_timestamp", "dish_info_index", "test-index"],
          "privileges": ["all"]
        }
      ]
    }
  }
}

After receiving the api, I went to add it to the elastic. I typed in this command:

root: elasticsearch-8.15.4
$ ./bin/elasticsearch-keystore add es_client.python.api_key

The key has been added

After receiving the api, I went to add it to the elastic. I typed in this command:

The key has been added

3. Delete api key from keystore

But after that, my elasticsearch didn't start at all.
Therefore, I was forced to delete the api key. I typed in this command:

./bin/elasticsearch-keystore remove es_client.python.api_key

The Elatsik is working

I do not know what I did wrong. I will be very glad if someone shows me a way to solve my problem :cry:

The main thing is that:

  1. Аuthorization on kibanа remains
  2. At the same time, it was possible to work with both Kibana and the ES python client

Thanks!

2

Hi @ALXIReinar Welcome to the community

since you enabled

xpack.security.http.ssl:
 enabled: true

the connection hosts needs to be https

You should refer to:

You will need the CA as well as shown in that example

# Create the client instance
client = Elasticsearch(
    "https://localhost:9200",
    ca_certs="/path/to/http_ca.crt",
    api_key="api_key",
)

Please take a close look at the documentation.

https://elasticsearch-py.readthedocs.io/en/v8.15.0/api/elasticsearch.html

I am not sure why you are doing this... this is not needed... the Python client does not read from the elastic keystore... you do not need to put the API key you generated in there.

The basic steps are

Generate the API Key from Kibana

Connect using the https endpoint + API Key + the CA Cert

You could also test with this to not validate the cert, but that is not recommended for production.

# Create the client instance
client = Elasticsearch(
    "https://localhost:9200",
    verify_certs=false,
    api_key="api_key",
)
1 Like
3

Thank you very much, Stephen! I had assumptions about https for this case, but I wasn't completely sure about it.
Anyway, thanks a lot again!

4

Did you get it to connect?

5

I haven't tried it yet. I decided that tomorrow I would sort out this headache with a fresh head. I have only recently started studying this wonderful search engine, so there are some difficulties with using it. I will be happy to report the result:)

1 Like
6

Alas, I was unable to connect :smiling_face_with_tear:

Code

aioes = AsyncElasticsearch(
    hosts='https://localhost:9200',
    api_key='API',
    ca_certs='/config/certs/http_ca.crt'
)

Output
ValueError: ca_certs parameter is not a path

It did not work either through the Api key or with basic authorization
I assumed it was about asynchrony, but even here it's past:

Code

es = Elasticsearch(
    hosts='https://localhost:9200',
    basic_auth=('elastic', 'password'),
    ca_certs='/config/certs/http_ca.crt'
)

res = es.search(index='index_index')
print(res)

Output
elastic_transport.TlsError: TLS error caused by: TlsError(TLS error caused by: SSLError([Errno 2] No such file or directory))

However, in this case, it already writes that the path to the file is incorrect... Here I stopped understanding even more what my mistake was. :sweat_smile:

elasticsearch-8.15.4\config\certs - the PATH to the CA

image
image820×289 16.2 KB

7 
es = AsyncElasticsearch(
    hosts='https://localhost:9200',
    api_key='API',
    ca_certs='/config/certs/http_ca.crt'
)

You need to put the full path to the certs that were created from the root directory.

Like

ca_certs=/Users/stephen/elasticsearch-8.15.4/config/certs/http_ca.crt

If it's Windows you need the full path as well!

1 Like
8

Many thanks

Oh, Stephen... Everything worked out!!! Finally!
I've been in agony since these 6 days. I have constant problems with ensuring that the middleware for my projects is working properly.
The funny thing about this story is that yesterday I tried to set up security (yes, it was already set up automatically without me). Everything broke down for me, so I had to put elatsik and kibana on a new one. I lost 4.5 hours on this. This is terrible

I am grateful for your help

The final version

With safety turned on in the elasticsearch.yml:

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

In my case, this setting worked:

es = AsyncElasticsearch(
    hosts='https://localhost:9200',
    basic_auth=('elastic', 'password'),
    ca_certs='C:/Users/User/Desktop/elasticsearch-8.15.4/config/certs/http_ca.crt'
)

The same is true for the synchronous version of the ES client