Upload Large Log File

Note: I've already Google-ed before I ask.

I see this issue [ML] Uploading a file over 100MB via Kibana · Issue #61735 · elastic/kibana · GitHub which allowed to upload less than 1G files.

As a dev, I know it would be hard to upload such a large file via browser and http.

Some part of my job is to analyze static log files from 3rd-party partners. These files are structured, but since from different devices, they are using totally different separators , but still plain text. And due to the difference in device configuration which is I cannot control, the file size is from 10MiB to 6GiB.

If the file is large than 1G per-file, instead of splitting it to multiple slices, is it possible to feed the log files to ELK stack?

I see filebeat, but it seems used to gather streamed logs, instead of feeding those cold logs. If filebeat could help with me, please give me some link.

Thanks in advance.

I'm not faimilar with filebeat but Logstash (L of ELK) could watch some folder for new file, convert them and send to Elasticsearch.

You may use:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.