# Streaming custom data source to Elastic

**URL:** <https://discuss.elastic.co/t/streaming-custom-data-source-to-elastic/202613>\
**Category:** Elasticsearch\
**Created:** [October 8, 2019, 4:58am UTC](https://discuss.elastic.co/t/streaming-custom-data-source-to-elastic/202613 "2019-10-08T04:58:51Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ajhstn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajhstn/32/24629_2.png) [@ajhstn](https://discuss.elastic.co/u/ajhstn)\
**Post date:** [October 8, 2019, 4:58am UTC](https://discuss.elastic.co/t/streaming-custom-data-source-to-elastic/202613/1 "2019-10-08T04:58:51Z")

</div>

Hello - This is a question around best practices for streaming custom data to Elastic.

Currently I have a script to pull 1 event from my API and then post it to `/indexname/_doc`. This works great, but now i am asking, what is the best way to run this continuously?

I thought that if i pull the data down (in a script) and save it to a **JSON** document on disk, then configure `Filebeat` to ship that data to Elastic is **one** option, but i'm wanting to ask the advice of others before i try take this route.

Obviously there is the **Bulk API** that could be coded straight into the script, however as `Filebeat` also uses the **Bulk API** under the covers, why re-invent the wheel.

Thank you for feedback.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 8, 2019, 8:27am UTC](https://discuss.elastic.co/t/streaming-custom-data-source-to-elastic/202613/2 "2019-10-08T08:27:13Z")

</div>

What kind of script it is? Java based? Pure unix script?

There are pros and cons in both methods but you surely know it.

Using filebeat is very good because indeed you can benefit from the retry mechanism in case of failure on elasticsearch side for example.  
On the other hand, you will have to write/update a file and have to think about cleaning the old files.

My guts feeling is that if I'm using Java for example, I'd use the bulk processor and accumulate documents in it and let it flush to elasticsearch. But in that case you need to deal with errors "manually".

One other solution is to write your documents to a message queue system (like Kafka, Redis, RabbitMQ...) and use Logstash to read the event queue.

My 0.05 cents. Happy to hear from others.

---

<div class="post-metadata">

**Author:** ![ajhstn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajhstn/32/24629_2.png) [@ajhstn](https://discuss.elastic.co/u/ajhstn)\
**Post date:** [October 8, 2019, 8:42pm UTC](https://discuss.elastic.co/t/streaming-custom-data-source-to-elastic/202613/3 "2019-10-08T20:42:44Z")

</div>

Hi David, thank you for the feedback, that is helpful. I may try Filebeat first, see how that goes and what the management side of things will look like.

The script itself is Windows Powershell, but this is more a POC, and will likely move this to python or use the elastic nodejs client.

From an architecture perspective, is running a script either continuously or every 30/60 seconds in a cron a good use case for containers?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 5, 2019, 8:51pm UTC](https://discuss.elastic.co/t/streaming-custom-data-source-to-elastic/202613/4 "2019-11-05T20:51:57Z")

</div>

Closed on November 5, 2019, 8:51pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.