Get dataset into Elastic

[CD9820]

Hello,

I developed a script that gathers information from a range of physical servers (hardware health state, firmware versions, security settings, ...).
The gathered dataset is written to a json file. As a test I 've uploaded the generated json file to our Elastic stack and created some test visualizations in Kibana. This all works as expected.
I want the script to run once a week and the generated json file should be automatically forwarded to Elastic. Since the data is not metric nor log (the script always generates a new output file) I am wondering what the recommended way is to get this data in Elastic? I guess every time a new file is injected in Elastic a new index should be created? Is it possible to use filebeat in this case or should I create a new index and inject the data into Elastic from within the script?

Best regards,
Christophe

[carly.richmond]

Hi @CD9820,

Welcome! What is your script written in? There are a few ways to get your data into Elasticsearch.

1. Depending on the language you've written your script you could add logic using an Elasticsearch client to create a new index and bulk ingest your data.
2. You could use Filebeat as you suggest to pick up any new file matching the path (regex or full path) and ingest into Elasticsearch. You can create rolling indices using ILM which is compatible with Filebeat.
3. If you want to do more advanced preprocessing you could also look at Logstash for ingesting the file as well. This is also compatible with ILM.

Hopefully one of those options will help you!

[CD9820]

Thank you, the script is written in Python.

[carly.richmond]

Thanks for confirming @CD9820. There is a Python client available if you decide to go down the option 1 route.

Hope that helps!

[willemdh]

Because you are not interested in the history, this looks like a monolithic index and it might be an idea to store the data in sth like state-* indices instead of indexing into logs-* or metrics-*.

You could use the serial number as doc id.