Issue Connecting Python to Elasticsearch in Docker Environment

Elastic Stack Elasticsearch
, ,
2

You have a few issues.

First, you should be consistent with the versions of Elasticsearch server and client that you use. For your server you are using 8.7.1, and for your Python client you are using 7.13.3. My recommendation is that use 8.14.0 (latest release as of today) for both.

Second, I'm not familiar with the retry package that you are using on the Python code, but it does not appear to work. The Python client has retry logic, so I would just remove the @retry decorator to keep things simple.

Third, you have some issues in the connection call in main.py:

  • the connection host and port is best given as a URL
  • your Elasticsearch instance uses a self-signed SSL certificate. To be able to connect from Python you will need to either disable certification verification or add the certificate authority so that the validation can be performed.
  • you have not provided the credentials to connect to the Elasticsearch instance.

The changes that I made to make your code work are below.

This is the app section of the docker-compose.yml file. Here I just added the Elasticsearch password as an environment variable:

 app:
    build:
      context: ./app
      dockerfile: Dockerfile
    container_name: python_app
    volumes:
      - ./app:/app
    depends_on:
      es01:
        condition: service_healthy
    environment:
     - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
    command: python /app/main.py

In the .env file, I configured the 8.14.0 stack version:

# Version of Elastic products
STACK_VERSION=8.14.0

In the requirements.txt file I configured the 8.14.0 Python client and removed unused dependencies:

elasticsearch==8.14.0

Finally, this is the main.py file with the connection fixes:

import os
from elasticsearch import Elasticsearch

def indexer():
    es = Elasticsearch(
        hosts=["https://host.docker.internal:9200"],
        basic_auth=('elastic', os.getenv("ELASTIC_PASSWORD")),
        verify_certs=False,
        max_retries=30,
        retry_on_timeout=True,
        request_timeout=30,
    )

    print("Connection with Elasticsearch: ", es.ping())

    es.index(index='test-index', id=1, document={'name': 'Teste', 'value': 123})

    result = es.get(index='test-index', id=1)
    print(result)

indexer()

Hope this helps!