ElasticSearch for the Log Search in Zipped Archives Usecase

Elastic Stack Elasticsearch
1

Hi,
I have the following usecase and I want to find out if I can use elasticsearch for it. If yes, then how can I implement it.
Usecase:
I want to search log archives for any kind of string.
I have log archives from different servers. There is a directory for each server, containing zipped files of logs. Each zipped file contains exactly one type of log file. There are different kind of log types, but essentially all the log files are text files.
Each server directory can contain anywhere from 1000 to 50K zipped files. Size of the archive is in access of 2TB.
Currently I have the log paths hosted on apache and can download individual zipfiles, but cannot search for a string across the zipped files.
Could you please suggest if elastic search can be implemented for this scenario. If so, how to go about it.

Thanks

2

The easiest thing to do IMO is to gunzip the file and run filebeat on it then gzip it again.
Once everything is in elasticsearch, that's fine.

Otherwise, you can may be gunzip on the fly with LS with https://www.elastic.co/guide/en/logstash/current/plugins-codecs-gzip_lines.html but I'm not sure.

3

The archives are on a different machine. There is a rsynch process which gets the logs fromt he host machine to archive machine. I cannot install anything on host machine like filebeat. Are you suggesting to first unzip the files run filebeat on archive machine and zip it again?

Thanks
Manu

6

Yes. That's what I meant.