Add An archived Flat File

[macdadi112 (Mac)]

Hi

Is there a way in which to view archived files in elastic search from a month ago?

Thanks

[dadoonet (David Pilato)]

How did you archive them?

[macdadi112 (Mac)]

They were archived after 2 weeks and involves minio.

So what I want to do is be able to replay them as and when I want.

Thanks

[dadoonet (David Pilato)]

Sounds like similar to:

Archived minio files

Hi What I have is files get archived to minio after a certain duration. So how would I play these files again? Thanks and apologies if I sent this to the wrong group, Mac

Replay files from Minio

Hi What I have is files get archived to minio after a certain duration. So how would I play these files again (retrieved by logstash)? Thanks and apologies if I sent this to the wrong group, Mac

Right?

[macdadi112 (Mac)]

Yes, thats the same issue but started this one after thinking about what I was trying to achieve.

It could be replayed in Logstash -> elasticsearch

Or as suggested above directly in elasticsearch.

Apologies if I shouldn't have duplicated the issue.

[dadoonet (David Pilato)]

How was the data archived in Minio?

[macdadi112 (Mac)]

It was archived using minio running via docker.

It is a flat file and is json format once you look at it.

Please note that logstash is running as a container via docker.

[dadoonet (David Pilato)]

I don't understand so I don't know.
You did not share what exactly has been stored in Minio. What the files look like...

May be have a look at the s3 input plugin of logstash.

[dadoonet (David Pilato)]

So I believe you can use filebeat to stream all lines and use the json option. See https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-log.html#filebeat-input-log-config-json

If you have any further question on this, please open a new question in #beats:filebeat