# Custom Beat - spool disk queue

**URL:** <https://discuss.elastic.co/t/custom-beat-spool-disk-queue/180622>\
**Category:** Beats\
**Created:** [May 10, 2019, 9:43pm UTC](https://discuss.elastic.co/t/custom-beat-spool-disk-queue/180622 "2019-05-10T21:43:31Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sentient](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sentient/32/34996_2.png) [@sentient](https://discuss.elastic.co/u/sentient)\
**Post date:** [May 10, 2019, 9:43pm UTC](https://discuss.elastic.co/t/custom-beat-spool-disk-queue/180622/1 "2019-05-10T21:43:31Z")

</div>

I have a simple community beat that sends messages received on a udp channel to elasticsearch.

```auto
	bt.client, err = b.Publisher.ConnectWith(beat.ClientConfig{
		PublishMode: beat.GuaranteedSend,
		WaitClose: 10 * time.Second,
	})
        .... 
       bt.client.PublishAll(bt.buffer)

```

the data is being send to elasticsearch. all ok.  
but when I configure to the the

```auto
queue:
    spool:
       file:

```

I see the data being flushed to disk, and it still makes it to elasticsearch.

But how do I send an acknowledgement (and to where) so that the queue spool file can remove that record.

It seems over time that I'm running out of disk space in the ring

Or don't I have to code anything for this and it is done behind the scene?

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [May 14, 2019, 2:02pm UTC](https://discuss.elastic.co/t/custom-beat-spool-disk-queue/180622/2 "2019-05-14T14:02:21Z")

</div>

The spool file has a max size. Once this size is reached, it won't allocate more space. Internally it's a simple dynamic transactional on-disk queue. It writes events into pages, potentially having multiple events in a page, or events spanning mulitple pages. A page size is 4KB by default.

Once all events in a page have been ACKed, the pages (in case of events spanning multiple pages) are returned to the free list immediately and have a high chance to be reused immediately.

Although the actual on-disk usage is low, the file can still grow, based on allocation patterns.

> Or don't I have to code anything for this and it is done behind the scene?

All magic is behind the scenes.

---

<div class="post-metadata">

**Author:** ![sentient](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sentient/32/34996_2.png) [@sentient](https://discuss.elastic.co/u/sentient)\
**Post date:** [May 14, 2019, 5:46pm UTC](https://discuss.elastic.co/t/custom-beat-spool-disk-queue/180622/3 "2019-05-14T17:46:07Z")

</div>

Thanks Steffens,

I did quite some testing the last fews days. And most of the 'behind the scenes' work.  
I can stop elasticsearch nodes. Let the events queue up. Restart elasticsearch and pending events are being send over.

But in production under heavy load. I get this error:

```auto
2019-05-10T18:28:10.029Z ERROR [publisher] spool/inbroker.go:544 Spool flush failed with: pq/writer-flush: txfile/tx-alloc-pages: file='/var/lib/statsdbeat/spool.dat' tx=0: transaction failed during commit: not enough memory to allocate 255 data page(s)

```

I'm using the default 4k page size. I have a large pre-allocated disk queue. And 4Gb of internal memory.

The error suggest an out of memory error. But it happens when it tries to commit the events to file. pq/writer\_flush

The 4k page size is the file block allocation size. So making that smaller won't help  
Is it internal memory. Or do I need a larger disk queue size. Or just the input stream is too high for the output stream to process?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 11, 2019, 5:46pm UTC](https://discuss.elastic.co/t/custom-beat-spool-disk-queue/180622/4 "2019-06-11T17:46:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.