Advantages of getting data from Elastic using Python

Elastic Stack Elasticsearch
1

Hello guys,

I am a begginer at ELK and I am studying elasticsearch package in python. I have a shell script which reads large logs, extract data and send them to Zabbix. Theses large log files are now at Elasticsearch. My doubt is what is the best option to do the same role our shell script does: Use Python for getting data from Elastic or simply use Python with Pandas to directly read my log files in its origin server? Could you please also provide me the advantages of getting data from Elastic in such a situation?

Thansk in advance

2

Hi @Jeferson_Schiavinato ,

Just to clarify, in your new scenario:
1- What is the origin of your logs?
2- What is the destination of the logs?
3- What is the frequency of reading per day?
4- What is the estimated size of the logs?

Thanks

3

Hi Alex,

Here are the information:

Just to clarify, in your new scenario:
1- What is the origin of your logs?
R: A Telco Platform that CRUD subscribers in databases using REST/SOAP requests.
2- What is the destination of the logs?
R: Firstly It was stored in a Linux Server and now we sent them do a Server with Elasticsearch running on it.
3- What is the frequency of reading per day?
R: Every 5 minutes now that the traffic is small. When the traffic grows up it will change to every 15 minutes
4- What is the estimated size of the logs?
R: It is about 5k now , however it tends to turn into 30 a 50M when we migrate the traffic that was projected to it.

4

Hi @Jeferson_Schiavinato ,

There are tons of ways to ingest your logs into Elasticsearch, as you can see in this article:

So, I suggest you start simple, like using Beats, then python(if need), then include Kafka to robust stream and improve scalability, etc. So, study more ways, and update your architecture. I'm sending you some good articles and videos that can help:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.