Metricbeat fail safe/over function on output

Elastic Stack Beats
1

Is there a way in which i can save the data from metric beat even though the output destination is not available?
Like for example elasticsearch host can't be reach, it will save first the data then continue once the connections is available again.

Or are the any third party application which I can store the pending data of metric beat?

TYIA!

2

There is a new disk spooling feature that you can use to write metrics to disk. When the output destination comes back online then the data stored in the disk backed queue will be forwarded to ES.

https://www.elastic.co/guide/en/beats/metricbeat/6.3/configuring-internal-queue.html#configuration-internal-queue-spool

1 Like
3

Hi @andrewkroh. I tried to implement the queue spool but i got problem. I couldn't start the metricbeat service after the implementation.
Kindly check the config below.

 #==========================  Modules configuration ============================

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false


queue.spool:
  file:
    path: "${path.data}/spool.dat"
    size: 300GiB
    page_size: 16KiB
  write:
    buffer_size: 10MiB
    flush.timeout: 5s
    flush.events: 1024

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
   hosts: ["192.168.200.254:9200"]
4

What error is the log file?

What version of Metricbeat? This was introduced in 6.3.

5

I'm running on metricbeat 6.3.1

Here is the error from the log file:

 metricbeat stopped.
2018-08-02T12:09:17.067+0800 ERROR instance/beat.go:691 Exiting: error initializing publisher: file permissions must be more strict (required permissions: -rw-------, actual permissions: -rw-rw-rw-)
6

Based on the error, you need to change the permissions on your config files.

Try sudo chmod 0400 /etc/metricbeat/metricbeat.yml

7

My metricbeat is on Windows 10 machine.

I tried to chmod the yml in unix but still an error occur

8

Just to confirm,

it the config is valid?

queue.spool:
  file:
    path: "${path.data}/spool.dat"
    size: 300GiB
    page_size: 16KiB
  write:
    buffer_size: 10MiB
    flush.timeout: 5s
    flush.events: 1024
9

It works on my machine using 6.3.1, but I'm not on Windows. After learning that you are using Windows I am surprised to see that file permission error because AFAIK windows does not use POSIX permissions like show in the error, but has a DACL.

Can you try a workaround by adding permissions: 0777.

queue.spool:
  file:
    path: "${path.data}/spool.dat"
    size: 300GiB
    page_size: 16KiB
    permissions: 0777
  write:
    buffer_size: 10MiB
    flush.timeout: 5s
    flush.events: 1024
10

I opened an issue.

1 Like
11

Hi @andrewkroh, i tried to set permissions, but still I cannot start the service of metricbeat.

12

got new error message.

metricbeat stopped.
2018-08-03T11:53:47.570+0800 ERROR instance/beat.go:691 Exiting: error initializing publisher: spool queue: failed to open file at path 'C:\ProgramData\metricbeat/spool.dat': file C:\ProgramData\metricbeat/spool.dat can not be locked right now
13

Sorry, it looks like you worked around the problem only to hit another one. The spooling to disk feature is still marked as beta. Can you please open a new issue on Github for the last error you hit and we'll get it fixed. I'm quite sure it's some kind of bug specific to Windows.

14

Hi @andrewkroh, thanks. i already open an issue in github.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.