Offline monitoring or Spooling into a disk

Hello Elastic team,

I really need your help on this. I am trying to setup the internal Queue, so when my system is offline for some time and after connecting it back to the network, i could still receive a metricbeat data in Kibana, during the system was offline. I configured everything correct, here is the conf

queue.mem:
events: 4096
flush.min_events: 512
flush.timeout: 5s queue.spool:
file:
path: "${path.data}/spool.dat"
size: 2048MiB
page_size: 16KiB
write:
buffer_size: 100MiB
flush.timeout: 5s
flush.events: 1024

but it can only provide data for only 30 minutes. For example if my system disconnected at 10 am, and i connect it back to the network at 12 pm, i can only only see the data up untill 10:30 am.

I ve also tried configure logstash, but samething there as well. Do you have any idea on how we can fix it ?

thank you,
Ratmir

Your config snippet is a tad difficult to read, but I would make sure have queue.spool configured properly and try and set a fairly high file size, which it looks like you have. https://www.elastic.co/guide/en/beats/metricbeat/current/configuring-internal-queue.html#configuration-internal-queue-spool

You might also want to make sure you've set max_retries properly so you're not dropping events: https://www.elastic.co/guide/en/beats/metricbeat/7.3/elasticsearch-output.html#_literal_max_retries_literal

However, considering your circumstances and the fact that you're disconnected for a number of hours, I would recommend trying to have something between metricbeat and ES that's available for longer periods of time.

Thank you very much for replying me.
I have increased the size file to: 4096MiB and also set max_retries to 500, backoff.max and backoff.init to 120s, but looks like still same output.

What would you recommend to have between metricbeat and ES to help me increase time? I aslo checked the file path where the data while the system is off should go, but coundt find /spool.dat file. I checked the /var/lib/metricbeat and its empty

Anything that can remain up while ES is down that you can use as a "buffer" for extended periods of time.

Are you sure file spooling is enabled? If you check your metricbeat log, it should print a message warning you that it's on. Also, try commenting out all of the queue.mem config and just leaving queue.spool configured.

didn't see any messages that it's on. Do i have to do anything to enable spooling ? i configured it and restarted metricbeat service, should't it be enabled after that?

Just commented queue.mem, will try to disconnect again

still same 25 min. i disconnected it at 12pm and connected back at 1 pm, and it only restores data up until 12:25.
I noticed that even i i will comment all spooling configuration, it still remember those 25 minutes. I think queue.spool configuration not working at all.

Can you post your entire metricbeat config using the code formatting with backtics?

Y mean metricbeat.yml file?

Yes.

Thank you, Alex I figured what was the problem. Apparently it was something with permissions.

The only question i have now is, the size of my /spool.dat file increased by 300MB and after connecting it back to the network, it didn't empty /spool.dat file. How can we set events to immediately remove from the queue?