How logstash parse log files internally?


(Clash Of Clanss) #1

I am new here for ELK(ElasticSearch,Logstash,kibana). I would like to know how logstash parse log files and upload into an ElasticSearch. Lets take an example, If I want to load around 250GB of log files data into an ElasticSearch from S3 storage by using logstash. In logstash how that log files are parsed and send into an elasticsearch, is it stored the whole S3 log files data in local memory and then parsed that log files and then put into an elasticsearch? or it will parsed by reading one by one line from S3 Storage without downloading all the log files from S3 storage. And also how to increase the read and write throughput in logstash. is it support multithread process?

Thanks,


(Magnus Bäck) #2

or it will parsed by reading one by one line from S3 Storage without downloading all the log files from S3 storage.

The files are downloaded locally, but I don't remember if they're downloaded one by one (i.e. you'll never have more than one file locally at any points in time) or if all matching files are continuously downloaded independent of the processing.

And also how to increase the read and write throughput in logstash. is it support multithread process?

Yes. See e.g. https://www.elastic.co/guide/en/logstash/current/tuning-logstash.html


(Clash Of Clanss) #3

Thanks for responding... but if the files are downloaded locally then how it will be efficient? because S3 file size is very large then how to parse that log files efficiently?


(Magnus Bäck) #4

but if the files are downloaded locally then how it will be efficient?

Why would it not be efficient? What do you suggest instead?


(Clash Of Clanss) #5

No I am not suggested... I like to know clear idea about like if all files are downloaded around 250GB of data in local memory how it could be parsing all the data from that local memory.


(Magnus Bäck) #6

They're downloaded to disk, not to memory.


(Clash Of Clanss) #7

yeah you are right but is it downloaded all the log files into disk or one by one from s3 storage? because if it download all the files from s3, the disk storage will not stored all the files.


(Magnus Bäck) #8

As I said, I don't know if they're downloaded all at once or one by one as the parsing progresses.


(Clash Of Clanss) #9

ohh it's okay thanks man.


(system) #10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.