# Advantages of getting data from Elastic using Python **URL:** \ **Category:** Elasticsearch\ **Tags:** language-clients\ **Created:** [June 2, 2023, 7:03pm UTC](https://discuss.elastic.co/t/advantages-of-getting-data-from-elastic-using-python/335118 "2023-06-02T19:03:54Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Jeferson\_Schiavinato](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeferson_schiavinato/32/118384_2.png) [@Jeferson\_Schiavinato](https://discuss.elastic.co/u/Jeferson_Schiavinato)\ **Post date:** [June 2, 2023, 7:03pm UTC](https://discuss.elastic.co/t/advantages-of-getting-data-from-elastic-using-python/335118/1 "2023-06-02T19:03:54Z")
Hello guys, I am a begginer at ELK and I am studying elasticsearch package in python. I have a shell script which reads large logs, extract data and send them to Zabbix. Theses large log files are now at Elasticsearch. My doubt is what is the best option to do the same role our shell script does: Use Python for getting data from Elastic or simply use Python with Pandas to directly read my log files in its origin server? Could you please also provide me the advantages of getting data from Elastic in such a situation? Thansk in advance ---
**Author:** ![Alex\_Salgado-Elastic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_salgado-elastic/32/103081_2.png) [@Alex\_Salgado-Elastic](https://discuss.elastic.co/u/Alex_Salgado-Elastic)\ **Post date:** [June 12, 2023, 6:28pm UTC](https://discuss.elastic.co/t/advantages-of-getting-data-from-elastic-using-python/335118/2 "2023-06-12T18:28:34Z")
Hi @Jeferson_Schiavinato , Just to clarify, in your new scenario: 1- What is the origin of your logs? 2- What is the destination of the logs? 3- What is the frequency of reading per day? 4- What is the estimated size of the logs? Thanks ---
**Author:** ![Jeferson\_Schiavinato](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeferson_schiavinato/32/118384_2.png) [@Jeferson\_Schiavinato](https://discuss.elastic.co/u/Jeferson_Schiavinato)\ **Post date:** [June 12, 2023, 7:31pm UTC](https://discuss.elastic.co/t/advantages-of-getting-data-from-elastic-using-python/335118/3 "2023-06-12T19:31:18Z")
Hi Alex, Here are the information: Just to clarify, in your new scenario: 1- What is the origin of your logs? R: A Telco Platform that CRUD subscribers in databases using REST/SOAP requests. 2- What is the destination of the logs? R: Firstly It was stored in a Linux Server and now we sent them do a Server with Elasticsearch running on it. 3- What is the frequency of reading per day? R: Every 5 minutes now that the traffic is small. When the traffic grows up it will change to every 15 minutes 4- What is the estimated size of the logs? R: It is about 5k now , however it tends to turn into 30 a 50M when we migrate the traffic that was projected to it. ---
**Author:** ![Alex\_Salgado-Elastic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_salgado-elastic/32/103081_2.png) [@Alex\_Salgado-Elastic](https://discuss.elastic.co/u/Alex_Salgado-Elastic)\ **Post date:** [June 14, 2023, 5:16pm UTC](https://discuss.elastic.co/t/advantages-of-getting-data-from-elastic-using-python/335118/4 "2023-06-14T17:16:31Z")
> [@Jeferson\_Schiavinato](#): > > Telco Platform that CRUD subscribers in databases using REST/SOAP Hi @Jeferson_Schiavinato , There are tons of ways to ingest your logs into Elasticsearch, as you can see in this article: - [Seven ways to ingest data into Elasticsearch!](https://aravind.dev/elastic-data-ingest/). So, I suggest you start simple, like using Beats, then python(if need), then include Kafka to robust stream and improve scalability, etc. So, study more ways, and update your architecture. I'm sending you some good articles and videos that can help: - [Scaling Data Ingestion with Elastic | Elastic Videos](https://www.elastic.co/pt/webinars/scaling-data-ingestion-with-elastic) - [Ingestion methods | Elastic Cloud Enterprise Reference [3.2] | Elastic](https://www.elastic.co/guide/en/cloud-enterprise/3.2/ece-ingest-methods.html) - [Ingest logs from a Python application using Filebeat | Elasticsearch Service Documentation | Elastic](https://www.elastic.co/guide/en/cloud/current/ec-getting-started-search-use-cases-python-logs.html) - [https://www.youtube.com/watch?v=VolhAzadyaU](https://www.youtube.com/watch?v=VolhAzadyaU) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 12, 2023, 5:16pm UTC](https://discuss.elastic.co/t/advantages-of-getting-data-from-elastic-using-python/335118/5 "2023-07-12T17:16:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.