# Adding state storage backend options to the libbeat framework?

**URL:** <https://discuss.elastic.co/t/adding-state-storage-backend-options-to-the-libbeat-framework/101080>\
**Category:** Beats\
**Created:** [September 19, 2017, 11:45pm UTC](https://discuss.elastic.co/t/adding-state-storage-backend-options-to-the-libbeat-framework/101080 "2017-09-19T23:45:46Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dustin\_Decker](https://avatars.discourse-cdn.com/v4/letter/d/e495f1/32.png) [@Dustin\_Decker](https://discuss.elastic.co/u/Dustin_Decker)\
**Post date:** [September 19, 2017, 11:45pm UTC](https://discuss.elastic.co/t/adding-state-storage-backend-options-to-the-libbeat-framework/101080/1 "2017-09-19T23:45:46Z")

</div>

For beats that don't depend on a local system resource, such as a beat for a web API, having a some standard options to store state remotely would be a huge win when running on infrastructure without persistent disk storage.

Providing an [`io.ReadWriter`](https://golang.org/pkg/io/#ReadWriter) backed by a configurable `local`, `elasticsearch`, `etcd`, or `consul` storage backend as part of the standard beats framework might be a good way to do it. Developers can be free to read and write their state blobs, and users can have more flexibility in their deployments.

---

<div class="post-metadata">

**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\
**Post date:** [September 20, 2017, 12:22pm UTC](https://discuss.elastic.co/t/adding-state-storage-backend-options-to-the-libbeat-framework/101080/2 "2017-09-20T12:22:16Z")

</div>

Just to make sure I understand, you are referring to custom Beats, and you want libbeat to never write to local disk, right?

If it could be done in a generic way, we'd be open to have that in libbeat, but I'm not sure how easy it will be. Can you try opening a PoC pull request? Doesn't need to be complete, but just to demonstrate the concept.

---

<div class="post-metadata">

**Author:** ![Dustin\_Decker](https://avatars.discourse-cdn.com/v4/letter/d/e495f1/32.png) [@Dustin\_Decker](https://discuss.elastic.co/u/Dustin_Decker)\
**Post date:** [September 20, 2017, 2:21pm UTC](https://discuss.elastic.co/t/adding-state-storage-backend-options-to-the-libbeat-framework/101080/3 "2017-09-20T14:21:42Z")

</div>

Yes, I'm proposing adding a configurable storage backend for state tracking in libbeat.

So for Filebeat, for example:  
What is now `filebeat.registry_file: registry` would instead become:

```auto
state:
  storage_driver: local
  storage_opts:
    path: registry

```

A different beat, such as one that retrieves Google Suite Oauth authorizations, you could use something like this, and run it on Kubernetes without local storage:

```auto
state:
  storage_driver: etcd
  storage_opts:
    address: "127.0.0.1:8500"
    token: "3940uf3ij32094j43"
    key: "co.elastic.gsuitebeat.state"

```

I'd be happy to make a PoC when I find time.

**The feature issue is being tracked here: [https://github.com/elastic/beats/issues/5375](https://github.com/elastic/beats/issues/5375)**

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [October 4, 2017, 10:54am UTC](https://discuss.elastic.co/t/adding-state-storage-backend-options-to-the-libbeat-framework/101080/4 "2017-10-04T10:54:51Z")

</div>

It's an interesting idea. Currently the state store for filebeat and winlogbeat are custom per beat. We hope to change this, so to have a common API for storing/updating state in libbeat in the future. As first iteration we still want to go for local files, but e.g. considering k8s we might indeed opt to support other backends. But with backends based on networks, the chance of things going wrong will increase as well.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 10, 2017, 11:46pm UTC](https://discuss.elastic.co/t/adding-state-storage-backend-options-to-the-libbeat-framework/101080/5 "2017-10-10T23:46:16Z")

</div>

This topic was automatically closed after 21 days. New replies are no longer allowed.