# Logstash dockerfile issue while syncing elasticsearch with mysql using python **URL:** \ **Category:** Logstash\ **Tags:** docker\ **Created:** [March 9, 2024, 3:38pm UTC](https://discuss.elastic.co/t/logstash-dockerfile-issue-while-syncing-elasticsearch-with-mysql-using-python/355083 "2024-03-09T15:38:28Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![abhinavtyagi](https://avatars.discourse-cdn.com/v4/letter/a/e79b87/32.png) [@abhinavtyagi](https://discuss.elastic.co/u/abhinavtyagi)\ **Post date:** [March 9, 2024, 3:38pm UTC](https://discuss.elastic.co/t/logstash-dockerfile-issue-while-syncing-elasticsearch-with-mysql-using-python/355083/1 "2024-03-09T15:38:28Z")
Below is the logstash dockerfile taken from [GitHub - r13i/sync-elasticsearch-mysql: Using Logstash to synchronize an Elasticsearch index with MySQL data](https://github.com/r13i/sync-elasticsearch-mysql) but note that I am using Python and JDBC works with Java. How can I setup logstash with Python. Note that I am using mysql as my database and api's are built using django-ninja. So, kind of I want to connect logstash with MySQL using Python. Please reply if any other detail is required. ```auto FROM docker.elastic.co/logstash/logstash:7.9.3 # Download JDBC connector for Logstash RUN curl -L --output "mysql-connector-java-8.0.22.tar.gz" "https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.22.tar.gz" \ && tar -xf "mysql-connector-java-8.0.22.tar.gz" "mysql-connector-java-8.0.22/mysql-connector-java-8.0.22.jar" \ && mv "mysql-connector-java-8.0.22/mysql-connector-java-8.0.22.jar" "mysql-connector-java-8.0.22.jar" \ && rm -r "mysql-connector-java-8.0.22" "mysql-connector-java-8.0.22.tar.gz" ENTRYPOINT ["/usr/local/bin/docker-entrypoint"] ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [March 9, 2024, 3:47pm UTC](https://discuss.elastic.co/t/logstash-dockerfile-issue-while-syncing-elasticsearch-with-mysql-using-python/355083/2 "2024-03-09T15:47:37Z")
> [@abhinavtyagi](#): > > How can I setup logstash with Python. Hello, it is not clear what you mean with this. Logstash is a java application, it will use the JDBC to connect to relation databases. Logstash will connect to your database and send the data to the configured output, which normally is Elasticsearch, your python code/api would have nothing to do in this process. You have something like this: [Database] -\> Logstash] -\> [Elasticsearch] ---
**Author:** ![abhinavtyagi](https://avatars.discourse-cdn.com/v4/letter/a/e79b87/32.png) [@abhinavtyagi](https://discuss.elastic.co/u/abhinavtyagi)\ **Post date:** [March 9, 2024, 5:51pm UTC](https://discuss.elastic.co/t/logstash-dockerfile-issue-while-syncing-elasticsearch-with-mysql-using-python/355083/3 "2024-03-09T17:51:08Z")
Then, will the code shared in the logstash dockerfile work for connecting with the MySQL database ? ---
**Author:** ![abhinavtyagi](https://avatars.discourse-cdn.com/v4/letter/a/e79b87/32.png) [@abhinavtyagi](https://discuss.elastic.co/u/abhinavtyagi)\ **Post date:** [March 10, 2024, 5:38am UTC](https://discuss.elastic.co/t/logstash-dockerfile-issue-while-syncing-elasticsearch-with-mysql-using-python/355083/4 "2024-03-10T05:38:51Z")
@leandrojmp ?? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 10, 2024, 8:04am UTC](https://discuss.elastic.co/t/logstash-dockerfile-issue-while-syncing-elasticsearch-with-mysql-using-python/355083/5 "2024-03-10T08:04:51Z")
Please be patient in waiting for responses to your question and refrain from pinging multiple times asking for a response or opening multiple topics for the same question. This is a community forum, it may take time for someone to reply to your question. For more information please refer to the [Community Code of Conduct](https://www.elastic.co/community/codeofconduct) specifically the section "Be patient". Also, please refrain from pinging folks directly, this is a forum and anyone that participates might be able to assist you. If you are in need of a service with an SLA that covers response times for questions then you may want to consider talking to us about a [subscription](https://www.elastic.co/subscriptions). It's fine to answer on your own thread after 2 or 3 days (not including weekends) if you don't have an answer. ---
**Author:** ![abhinavtyagi](https://avatars.discourse-cdn.com/v4/letter/a/e79b87/32.png) [@abhinavtyagi](https://discuss.elastic.co/u/abhinavtyagi)\ **Post date:** [March 10, 2024, 8:27am UTC](https://discuss.elastic.co/t/logstash-dockerfile-issue-while-syncing-elasticsearch-with-mysql-using-python/355083/6 "2024-03-10T08:27:53Z")
Okay, any idea about the problem I shared above ? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 10, 2024, 10:25am UTC](https://discuss.elastic.co/t/logstash-dockerfile-issue-while-syncing-elasticsearch-with-mysql-using-python/355083/7 "2024-03-10T10:25:42Z")
> Then, will the code shared in the logstash dockerfile work for connecting with the MySQL database ? I don't think so but you did not clarify what you are trying to do. ---
**Author:** ![abhinavtyagi](https://avatars.discourse-cdn.com/v4/letter/a/e79b87/32.png) [@abhinavtyagi](https://discuss.elastic.co/u/abhinavtyagi)\ **Post date:** [March 10, 2024, 11:38am UTC](https://discuss.elastic.co/t/logstash-dockerfile-issue-while-syncing-elasticsearch-with-mysql-using-python/355083/8 "2024-03-10T11:38:53Z")
In the first post, I shared a GitHub repository which I am using to sync MySQL with Elasticsearch. Note that I am using docker and in my compose file I configured Elasticsearch and kibana and upto this point everything is running. Issue comes with the logstash, I am not getting how to setup logstash. My docker-compose.yml is as below: ```auto version: "1" services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:8.12.2 container_name: elasticsearch environment: - node.name=elasticsearch - cluster.name=es-docker-cluster - discovery.seed_hosts=elasticsearch - cluster.initial_master_nodes=elasticsearch - bootstrap.memory_lock=true - xpack.security.enabled=false - "ES_JAVA_OPTS=-Xms512m -Xmx512m" ulimits: memlock: soft: -1 hard: -1 volumes: - ./volumes/elasticsearch:/usr/share/elasticsearch/data logging: driver: "json-file" options: max-size: "10k" max-file: "10" ports: - 9200:9200 networks: - elastic kibana: image: docker.elastic.co/kibana/kibana:8.12.2 container_name: kibana environment: - node.name=kibana - cluster.name=es-docker-cluster - ELASTICSEARCH_HOSTS=http://elasticsearch:9200 ports: - 5601:5601 networks: - elastic depends_on: - elasticsearch volumes: data: driver: local networks: elastic: driver: bridge ``` If you look at the docker-compose in the repo, it's defining service for logstash as: ```auto logstash: build: context: . dockerfile: Dockerfile-logstash container_name: logstash # restart: on-failure depends_on: - mysql - elasticsearch volumes: - ./volumes/logstash/pipeline/:/usr/share/logstash/pipeline/ - ./volumes/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml - ./volumes/logstash/config/pipelines.yml:/usr/share/logstash/config/pipelines.yml - ./volumes/logstash/config/queries/:/usr/share/logstash/config/queries/ logging: driver: "json-file" options: max-size: "10k" max-file: "10" ``` and logstash-DockerFile is as : ```auto FROM docker.elastic.co/logstash/logstash:7.9.3 # Download JDBC connector for Logstash RUN curl -L --output "mysql-connector-java-8.0.22.tar.gz" "https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.22.tar.gz" \ && tar -xf "mysql-connector-java-8.0.22.tar.gz" "mysql-connector-java-8.0.22/mysql-connector-java-8.0.22.jar" \ && mv "mysql-connector-java-8.0.22/mysql-connector-java-8.0.22.jar" "mysql-connector-java-8.0.22.jar" \ && rm -r "mysql-connector-java-8.0.22" "mysql-connector-java-8.0.22.tar.gz" ENTRYPOINT ["/usr/local/bin/docker-entrypoint"] ``` Now, my question is that if I setup logstash as in the docker-compose and logstash dockerfile in the repo, will it work for Python as well ? Because JDBC works for Java and my application is built upon Django and Django-Ninja ?? Overall, I am trying to sync mysql with Elasticsearch, so if my api is doing any create, update or delete it should also reflect in Elasticsearch. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 10, 2024, 3:17pm UTC](https://discuss.elastic.co/t/logstash-dockerfile-issue-while-syncing-elasticsearch-with-mysql-using-python/355083/9 "2024-03-10T15:17:27Z")
I'm not sure how Python is related here but anyway, one of the major problem you might have to solve at first is to use the same version for the Elasticsearch cluster (8.12.2) and Logstash which is currently using 7.9.3. This won't work. > [@abhinavtyagi](#): > > Overall, I am trying to sync mysql with Elasticsearch, so if my api is doing any create, update or delete it should also reflect in Elasticsearch. Again I don't really see what you want to do. I mean: what is the use case... If your goal is to have your application, storing data in MySQL and indexed into Elasticsearch (for search), then indeed Logstash can help for that. But I'd recommend modifying the application layer if possible and send data to elasticsearch in the same "transaction" as you are sending your data to the database. I shared most of my thoughts there: [https://david.pilato.fr/blog/2015-05-09-advanced-search-for-your-legacy-application/](http://david.pilato.fr/blog/2015/05/09/advanced-search-for-your-legacy-application/) Have also a look at this ["live coding" recording](https://www.elastic.co/fr/blog/how-to-add-powerful-search-existing-sql-applications-elasticsearch-video-tutorial). ---
**Author:** ![abhinavtyagi](https://avatars.discourse-cdn.com/v4/letter/a/e79b87/32.png) [@abhinavtyagi](https://discuss.elastic.co/u/abhinavtyagi)\ **Post date:** [March 10, 2024, 4:30pm UTC](https://discuss.elastic.co/t/logstash-dockerfile-issue-while-syncing-elasticsearch-with-mysql-using-python/355083/10 "2024-03-10T16:30:39Z")
Thanks a lot. Let me do some research. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 7, 2024, 4:30pm UTC](https://discuss.elastic.co/t/logstash-dockerfile-issue-while-syncing-elasticsearch-with-mysql-using-python/355083/11 "2024-04-07T16:30:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.