# Get dataset into Elastic **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [February 20, 2024, 3:06pm UTC](https://discuss.elastic.co/t/get-dataset-into-elastic/353710 "2024-02-20T15:06:14Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![CD9820](https://avatars.discourse-cdn.com/v4/letter/c/e9c0ed/32.png) [@CD9820](https://discuss.elastic.co/u/CD9820)\ **Post date:** [February 20, 2024, 3:06pm UTC](https://discuss.elastic.co/t/get-dataset-into-elastic/353710/1 "2024-02-20T15:06:14Z")
Hello, I developed a script that gathers information from a range of physical servers (hardware health state, firmware versions, security settings, ...). The gathered dataset is written to a json file. As a test I 've uploaded the generated json file to our Elastic stack and created some test visualizations in Kibana. This all works as expected. I want the script to run once a week and the generated json file should be automatically forwarded to Elastic. Since the data is not metric nor log (the script always generates a new output file) I am wondering what the recommended way is to get this data in Elastic? I guess every time a new file is injected in Elastic a new index should be created? Is it possible to use filebeat in this case or should I create a new index and inject the data into Elastic from within the script? Best regards, Christophe ---
**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\ **Post date:** [February 21, 2024, 12:30pm UTC](https://discuss.elastic.co/t/get-dataset-into-elastic/353710/2 "2024-02-21T12:30:30Z")
Hi @CD9820, Welcome! What is your script written in? There are a few ways to get your data into Elasticsearch. 1. Depending on the language you've written your script you could add logic using an [Elasticsearch client](https://www.elastic.co/guide/en/elasticsearch/client/index.html) to create a new index and bulk ingest your data. 2. You could use [Filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html) as you suggest to pick up any new file matching the path (regex or full path) and ingest into Elasticsearch. You can create rolling indices using [ILM which is compatible with Filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html). 3. If you want to do more advanced preprocessing you could also look at [Logstash](https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html) for ingesting the file as well. [This is also compatible with ILM](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ilm). Hopefully one of those options will help you! ---
**Author:** ![CD9820](https://avatars.discourse-cdn.com/v4/letter/c/e9c0ed/32.png) [@CD9820](https://discuss.elastic.co/u/CD9820)\ **Post date:** [February 21, 2024, 6:57pm UTC](https://discuss.elastic.co/t/get-dataset-into-elastic/353710/3 "2024-02-21T18:57:03Z")
Thank you, the script is written in Python. ---
**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\ **Post date:** [February 22, 2024, 9:48am UTC](https://discuss.elastic.co/t/get-dataset-into-elastic/353710/4 "2024-02-22T09:48:54Z")
Thanks for confirming @CD9820. There is a [Python client](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/index.html) available if you decide to go down the option 1 route. Hope that helps! ---
**Author:** ![willemdh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/willemdh/32/16922_2.png) [@willemdh](https://discuss.elastic.co/u/willemdh)\ **Post date:** [February 23, 2024, 11:43am UTC](https://discuss.elastic.co/t/get-dataset-into-elastic/353710/5 "2024-02-23T11:43:26Z")
Because you are not interested in the history, this looks like a monolithic index and it might be an idea to store the data in sth like state-\* indices instead of indexing into logs-\* or metrics-\*. You could use the serial number as doc id. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 22, 2024, 1:43pm UTC](https://discuss.elastic.co/t/get-dataset-into-elastic/353710/6 "2024-03-22T13:43:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.