How logstash parse log files internally?

Elastic Stack Logstash
1

I am new here for ELK(ElasticSearch,Logstash,kibana). I would like to know how logstash parse log files and upload into an ElasticSearch. Lets take an example, If I want to load around 250GB of log files data into an ElasticSearch from S3 storage by using logstash. In logstash how that log files are parsed and send into an elasticsearch, is it stored the whole S3 log files data in local memory and then parsed that log files and then put into an elasticsearch? or it will parsed by reading one by one line from S3 Storage without downloading all the log files from S3 storage. And also how to increase the read and write throughput in logstash. is it support multithread process?

Thanks,

2

or it will parsed by reading one by one line from S3 Storage without downloading all the log files from S3 storage.

The files are downloaded locally, but I don't remember if they're downloaded one by one (i.e. you'll never have more than one file locally at any points in time) or if all matching files are continuously downloaded independent of the processing.

And also how to increase the read and write throughput in logstash. is it support multithread process?

Yes. See e.g. Tuning and Profiling Logstash Performance | Logstash Reference [8.11] | Elastic

3

Thanks for responding... but if the files are downloaded locally then how it will be efficient? because S3 file size is very large then how to parse that log files efficiently?

4

but if the files are downloaded locally then how it will be efficient?

Why would it not be efficient? What do you suggest instead?

5

No I am not suggested... I like to know clear idea about like if all files are downloaded around 250GB of data in local memory how it could be parsing all the data from that local memory.

6

They're downloaded to disk, not to memory.

7

yeah you are right but is it downloaded all the log files into disk or one by one from s3 storage? because if it download all the files from s3, the disk storage will not stored all the files.

8

As I said, I don't know if they're downloaded all at once or one by one as the parsing progresses.

9

ohh it's okay thanks man.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.