Upload Large Log File

Elastic Stack Kibana
1

Note: I've already Google-ed before I ask.

I see this issue [ML] Uploading a file over 100MB via Kibana · Issue #61735 · elastic/kibana · GitHub which allowed to upload less than 1G files.

As a dev, I know it would be hard to upload such a large file via browser and http.

Some part of my job is to analyze static log files from 3rd-party partners. These files are structured, but since from different devices, they are using totally different separators , but still plain text. And due to the difference in device configuration which is I cannot control, the file size is from 10MiB to 6GiB.

If the file is large than 1G per-file, instead of splitting it to multiple slices, is it possible to feed the log files to ELK stack?

I see filebeat, but it seems used to gather streamed logs, instead of feeding those cold logs. If filebeat could help with me, please give me some link.

Thanks in advance.

2

I'm not faimilar with filebeat but Logstash (L of ELK) could watch some folder for new file, convert them and send to Elasticsearch.

You may use:

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.