Logstash dockerfile issue while syncing elasticsearch with mysql using python

Elastic Stack Logstash
1

Below is the logstash dockerfile taken from GitHub - r13i/sync-elasticsearch-mysql: Using Logstash to synchronize an Elasticsearch index with MySQL data but note that I am using Python and JDBC works with Java.

How can I setup logstash with Python. Note that I am using mysql as my database and api's are built using django-ninja. So, kind of I want to connect logstash with MySQL using Python.

Please reply if any other detail is required.

FROM docker.elastic.co/logstash/logstash:7.9.3

# Download JDBC connector for Logstash
RUN curl -L --output "mysql-connector-java-8.0.22.tar.gz" "https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.22.tar.gz" \
    && tar -xf "mysql-connector-java-8.0.22.tar.gz" "mysql-connector-java-8.0.22/mysql-connector-java-8.0.22.jar" \
    && mv "mysql-connector-java-8.0.22/mysql-connector-java-8.0.22.jar" "mysql-connector-java-8.0.22.jar" \
    && rm -r "mysql-connector-java-8.0.22" "mysql-connector-java-8.0.22.tar.gz"

ENTRYPOINT ["/usr/local/bin/docker-entrypoint"]
2

Hello, it is not clear what you mean with this.

Logstash is a java application, it will use the JDBC to connect to relation databases.

Logstash will connect to your database and send the data to the configured output, which normally is Elasticsearch, your python code/api would have nothing to do in this process.

You have something like this:

[Database] -> Logstash] -> [Elasticsearch]

3

Then, will the code shared in the logstash dockerfile work for connecting with the MySQL database ?

4

@leandrojmp ??

5

Please be patient in waiting for responses to your question and refrain from pinging multiple times asking for a response or opening multiple topics for the same question. This is a community forum, it may take time for someone to reply to your question. For more information please refer to the Community Code of Conduct specifically the section "Be patient". Also, please refrain from pinging folks directly, this is a forum and anyone that participates might be able to assist you.

If you are in need of a service with an SLA that covers response times for questions then you may want to consider talking to us about a subscription.

It's fine to answer on your own thread after 2 or 3 days (not including weekends) if you don't have an answer.

6

Okay, any idea about the problem I shared above ?

7

Then, will the code shared in the logstash dockerfile work for connecting with the MySQL database ?

I don't think so but you did not clarify what you are trying to do.

8

In the first post, I shared a GitHub repository which I am using to sync MySQL with Elasticsearch.

Note that I am using docker and in my compose file I configured Elasticsearch and kibana and upto this point everything is running.

Issue comes with the logstash, I am not getting how to setup logstash.

My docker-compose.yml is as below:

version: "1"

services:
    elasticsearch:
        image: docker.elastic.co/elasticsearch/elasticsearch:8.12.2
        container_name: elasticsearch
        environment:
        - node.name=elasticsearch
        - cluster.name=es-docker-cluster
        - discovery.seed_hosts=elasticsearch
        - cluster.initial_master_nodes=elasticsearch
        - bootstrap.memory_lock=true
        - xpack.security.enabled=false
        - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
        ulimits:
            memlock:
                soft: -1
                hard: -1
        volumes:
            - ./volumes/elasticsearch:/usr/share/elasticsearch/data
        logging:
            driver: "json-file"
            options:
                max-size: "10k"
                max-file: "10"
        ports:
        - 9200:9200
        networks:
        - elastic

    kibana:
        image: docker.elastic.co/kibana/kibana:8.12.2
        container_name: kibana
        environment:
            - node.name=kibana
            - cluster.name=es-docker-cluster
            - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
        ports:
            - 5601:5601
        networks:
            - elastic
        depends_on:
            - elasticsearch

volumes:
  data:
    driver: local

networks:
    elastic:
        driver: bridge

If you look at the docker-compose in the repo, it's defining service for logstash as:

  logstash:
    build:
      context: .
      dockerfile: Dockerfile-logstash
    container_name: logstash
    # restart: on-failure
    depends_on:
      - mysql
      - elasticsearch
    volumes:
      - ./volumes/logstash/pipeline/:/usr/share/logstash/pipeline/
      - ./volumes/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
      - ./volumes/logstash/config/pipelines.yml:/usr/share/logstash/config/pipelines.yml
      - ./volumes/logstash/config/queries/:/usr/share/logstash/config/queries/
    logging:
      driver: "json-file"
      options:
        max-size: "10k"
        max-file: "10"

and logstash-DockerFile is as :

FROM docker.elastic.co/logstash/logstash:7.9.3

# Download JDBC connector for Logstash
RUN curl -L --output "mysql-connector-java-8.0.22.tar.gz" "https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.22.tar.gz" \
    && tar -xf "mysql-connector-java-8.0.22.tar.gz" "mysql-connector-java-8.0.22/mysql-connector-java-8.0.22.jar" \
    && mv "mysql-connector-java-8.0.22/mysql-connector-java-8.0.22.jar" "mysql-connector-java-8.0.22.jar" \
    && rm -r "mysql-connector-java-8.0.22" "mysql-connector-java-8.0.22.tar.gz"

ENTRYPOINT ["/usr/local/bin/docker-entrypoint"]

Now, my question is that if I setup logstash as in the docker-compose and logstash dockerfile in the repo, will it work for Python as well ?

Because JDBC works for Java and my application is built upon Django and Django-Ninja ??
Overall, I am trying to sync mysql with Elasticsearch, so if my api is doing any create, update or delete it should also reflect in Elasticsearch.

9

I'm not sure how Python is related here but anyway, one of the major problem you might have to solve at first is to use the same version for the Elasticsearch cluster (8.12.2) and Logstash which is currently using 7.9.3. This won't work.

Again I don't really see what you want to do. I mean: what is the use case...

If your goal is to have your application, storing data in MySQL and indexed into Elasticsearch (for search), then indeed Logstash can help for that. But I'd recommend modifying the application layer if possible and send data to elasticsearch in the same "transaction" as you are sending your data to the database.

I shared most of my thoughts there: https://david.pilato.fr/blog/2015-05-09-advanced-search-for-your-legacy-application/

Have also a look at this "live coding" recording.

10

Thanks a lot. Let me do some research.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.