Issue Connecting Python to Elasticsearch in Docker Environment

Elastic Stack Elasticsearch
, ,
1

Hi, I need help.

I'm working on a college project where I need to use Elasticsearch and Kibana, and access them through Docker.

Here is my problem: I created the Docker setup (using docker-compose) following the steps from the links below, but I need to send data to my Elasticsearch Docker container using Python, and I'm having trouble connecting Python to the localhost.

Links I used to configure my Docker setup:

  • https://www.elastic.co/pt/blog/getting-started-with-the-elastic-stack-and-docker-compose

  • https://shandou.medium.com/docker-compose-example-for-importing-csv-into-elasticsearch-via-python-client-a754bd4d7aa8

Here is the structure of my project:

docker/
├── docker-compose.yml
├── .env
├── app/
│   ├── Dockerfile
│   ├── main.py
│   └── requirements.txt

docker-compose.yml

volumes:
 certs:
   driver: local
 esdata01:
   driver: local
 kibanadata:
   driver: local
 metricbeatdata01:
   driver: local
 filebeatdata01:
   driver: local
 logstashdata01:
   driver: local

networks:
 default:
   name: elastic
   external: false

services:
 setup:
   image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
   volumes:
     - certs:/usr/share/elasticsearch/config/certs
   user: "0"
   command: >
     bash -c '
       if [ x${ELASTIC_PASSWORD} == x ]; then
         echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
         exit 1;
       elif [ x${KIBANA_PASSWORD} == x ]; then
         echo "Set the KIBANA_PASSWORD environment variable in the .env file";
         exit 1;
       fi;
       if [ ! -f config/certs/ca.zip ]; then
         echo "Creating CA";
         bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
         unzip config/certs/ca.zip -d config/certs;
       fi;
       if [ ! -f config/certs/certs.zip ]; then
         echo "Creating certs";
         echo -ne \
         "instances:\n"\
         "  - name: es01\n"\
         "    dns:\n"\
         "      - es01\n"\
         "      - localhost\n"\
         "    ip:\n"\
         "      - 127.0.0.1\n"\
         "  - name: kibana\n"\
         "    dns:\n"\
         "      - kibana\n"\
         "      - localhost\n"\
         "    ip:\n"\
         "      - 127.0.0.1\n"\
         > config/certs/instances.yml;
         bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
         unzip config/certs/certs.zip -d config/certs;
       fi;
       echo "Setting file permissions"
       chown -R root:root config/certs;
       find . -type d -exec chmod 750 \{\} \;;
       find . -type f -exec chmod 640 \{\} \;;
       echo "Waiting for Elasticsearch availability";
       until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
       echo "Setting kibana_system password";
       until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
       echo "All done!";
     '
   healthcheck:
     test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
     interval: 1s
     timeout: 5s
     retries: 120
    
 es01:
   container_name: example_es
   depends_on:
     setup:
       condition: service_healthy
   image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
   labels:
     co.elastic.logs/module: elasticsearch
   volumes:
     - certs:/usr/share/elasticsearch/config/certs
     - esdata01:/usr/share/elasticsearch/data
   ports:
     - ${ES_PORT}:9200
   environment:
     - node.name=es01
     - cluster.name=${CLUSTER_NAME}
     - discovery.type=single-node
     - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
     - bootstrap.memory_lock=true
     - xpack.security.enabled=true
     - xpack.security.http.ssl.enabled=true
     - xpack.security.http.ssl.key=certs/es01/es01.key
     - xpack.security.http.ssl.certificate=certs/es01/es01.crt
     - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
     - xpack.security.transport.ssl.enabled=true
     - xpack.security.transport.ssl.key=certs/es01/es01.key
     - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
     - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
     - xpack.security.transport.ssl.verification_mode=certificate
     - xpack.license.self_generated.type=${LICENSE}
   mem_limit: ${ES_MEM_LIMIT}
   ulimits:
     memlock:
       soft: -1
       hard: -1
   healthcheck:
     test:
       [
         "CMD-SHELL",
         "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
       ]
     interval: 10s
     timeout: 10s
     retries: 120

 **kibana:**
   depends_on:
     es01:
       condition: service_healthy
   image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
   labels:
     co.elastic.logs/module: kibana
   volumes:
     - certs:/usr/share/kibana/config/certs
     - kibanadata:/usr/share/kibana/data
   ports:
     - ${KIBANA_PORT}:5601
   environment:
     - SERVERNAME=kibana
     - ELASTICSEARCH_HOSTS=https://es01:9200
     - ELASTICSEARCH_USERNAME=kibana_system
     - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
     - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
     - XPACK_SECURITY_ENCRYPTIONKEY=${ENCRYPTION_KEY}
     - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=${ENCRYPTION_KEY}
     - XPACK_REPORTING_ENCRYPTIONKEY=${ENCRYPTION_KEY}
   mem_limit: ${KB_MEM_LIMIT}
   healthcheck:
     test:
       [
         "CMD-SHELL",
         "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
       ]
     interval: 10s
     timeout: 10s
     retries: 120

 app:
    build:
      context: ./app
      dockerfile: Dockerfile
    container_name: python_app
    volumes:
      - ./app:/app
    depends_on:
      es01:
        condition: service_healthy
    command: python /app/main.py

.env

# Project namespace (defaults to the current folder name if not set)
#COMPOSE_PROJECT_NAME=myproject


# Password for the 'elastic' user (at least 6 characters)
ELASTIC_PASSWORD=dundler


# Password for the 'kibana_system' user (at least 6 characters)
KIBANA_PASSWORD=mifflin


# Version of Elastic products
STACK_VERSION=8.7.1


# Set the cluster name
CLUSTER_NAME=docker-cluster


# Set to 'basic' or 'trial' to automatically start the 30-day trial
LICENSE=basic
#LICENSE=trial


# Port to expose Elasticsearch HTTP API to the host
ES_PORT=9200


# Port to expose Kibana to the host
KIBANA_PORT=5601


# Increase or decrease based on the available host memory (in bytes)
ES_MEM_LIMIT=1073741824
KB_MEM_LIMIT=1073741824
LS_MEM_LIMIT=1073741824


# SAMPLE Predefined Key only to be used in POC environments
ENCRYPTION_KEY=c34d38b3a14956121ff2170e5030b471551370178f43e5626eec58b04a30fae2

Dockerfile

FROM python:3.9-slim

WORKDIR /app

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["python", "main.py"]

main.py

from retry import retry
import pandas as pd
from elasticsearch import Elasticsearch
import elasticsearch.helpers

@retry(elasticsearch.ConnectionError, max_delay=300, delay=5)
def indexer():
    es = Elasticsearch(hosts=[{"host": "host.docker.internal", "port": 9200}], max_retries=30,
                       retry_on_timeout=True, request_timeout=30)

    print("Connection with Elasticsearch: ", es.ping())

    es.indices.create(index='test-index', ignore=400)
    es.index(index='test-index', id=1, document={'name': 'Teste', 'value': 123})

    result = es.get(index='test-index', id=1)
    print(result)
    
indexer()

requeriments.txt

elasticsearch==7.13.3
pandas==1.3.3
retry==0.9.2
2

You have a few issues.

First, you should be consistent with the versions of Elasticsearch server and client that you use. For your server you are using 8.7.1, and for your Python client you are using 7.13.3. My recommendation is that use 8.14.0 (latest release as of today) for both.

Second, I'm not familiar with the retry package that you are using on the Python code, but it does not appear to work. The Python client has retry logic, so I would just remove the @retry decorator to keep things simple.

Third, you have some issues in the connection call in main.py:

  • the connection host and port is best given as a URL
  • your Elasticsearch instance uses a self-signed SSL certificate. To be able to connect from Python you will need to either disable certification verification or add the certificate authority so that the validation can be performed.
  • you have not provided the credentials to connect to the Elasticsearch instance.

The changes that I made to make your code work are below.

This is the app section of the docker-compose.yml file. Here I just added the Elasticsearch password as an environment variable:

 app:
    build:
      context: ./app
      dockerfile: Dockerfile
    container_name: python_app
    volumes:
      - ./app:/app
    depends_on:
      es01:
        condition: service_healthy
    environment:
     - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
    command: python /app/main.py

In the .env file, I configured the 8.14.0 stack version:

# Version of Elastic products
STACK_VERSION=8.14.0

In the requirements.txt file I configured the 8.14.0 Python client and removed unused dependencies:

elasticsearch==8.14.0

Finally, this is the main.py file with the connection fixes:

import os
from elasticsearch import Elasticsearch

def indexer():
    es = Elasticsearch(
        hosts=["https://host.docker.internal:9200"],
        basic_auth=('elastic', os.getenv("ELASTIC_PASSWORD")),
        verify_certs=False,
        max_retries=30,
        retry_on_timeout=True,
        request_timeout=30,
    )

    print("Connection with Elasticsearch: ", es.ping())

    es.index(index='test-index', id=1, document={'name': 'Teste', 'value': 123})

    result = es.get(index='test-index', id=1)
    print(result)

indexer()

Hope this helps!

3

It worked, thank you!!

1 Like