Metricbeat elasticsearch module not showing index fields

Elastic Stack Beats
,
1

node and node_stats are working but not seeing index fileds coming into metricbeat.

  • module: elasticsearch
    metricsets:
    • node

    - node_stats

    • index
    • index_summary

    - shard

    period: 10s
    hosts: ["http://...:9200"]
2

Hi Frank,

I'm unable to replicate this. What version of Metricbeat are you on? Do you see any errors in the logs either from Metricbeat or from Elasticsearch? If you run Metricbeat with -d -e '*' do you see the fields present in the JSON data being sent? Finally, could you please repost your Elasticsearch module configuration and preserve the formatting so that we can make certain is is valid? Thanks.

3

metricbeat version 6.6.0 (amd64)
Elasticsearch is 6.4.3.

Not sure how to preserve the formatting though.

I was abe to get index fields from another cluster that is Elasticsearch 6.6.0. Not sure if that is the issue or not.

I am attaching the config and log output

metricbeat_ana_201903061111
metricbeat_ana_201903061111.png1190×719 74.3 KB

4

Could you try to enabled only index metricset and run metricbeat with -e -d "*" option and paste the output here? (it's going to be quite a lot).

To format the output best use 3 ticks before and after the code.

5 
sudo cat /etc/metricbeat/modules.d/elasticsearch.yml
- module: elasticsearch
  metricsets:
  # - node
  # - node_stats
  - index
  # - index_summary
  # - shard
  period: 10s
  hosts: ["http://*.*.*.*:9200"]
6

Output is too large

2019-03-12T14:59:59.032Z        WARN    [cfgwarn]       index/index.go:49       BETA: the elasticsearch/index metricset is beta
7

Is there another place you could post the output where we can see it? Perhaps using a GitHub gist?

8
9

Thanks for the ouptut. Based on https://gist.github.com/frankfoti/a436adf9dcc149b1b750e8d4ccb843b0#file-gistfile1-txt-L565 we see that it is started. Any chance you could disable the system module so we have less noise in the log.

Unfortunately the log you posted does not contain enough info to do a first fetch of the index metricset. You have to run it at least 19.9s with your settings above.

10

Output updated on gist

11

Ok, I just found a very good hint:

2019-03-14T12:46:52.853Z        DEBUG   [elasticsearch] index/index.go:71       trying to fetch index stats from a non-master node

The index stats can only be fetched from the master node. This is to make sure the same index stats are only reported ones as they are same across all nodes. I assume the node you are talking to here is not he master?

12

That was the issue. Answers some confusion I had why some were working and not others. Also, as the node switches from active master to master... it turns on and off. All set thank you. Gist snippet updated.

Thanks

13

Glad it's working

@shaunak I wonder if we should keep some state in the metricset and log a message on the Info every time the metricset switches from talking to master or not talking to master to have something for the user on the Info level. I expect master not to constantly move around :crossed_fingers:

14

I'd rather not keep state. The message in the log that Metricbeat is not talking to a master seems good enough to me. Since that check happens in the metricset's Fetch method, it will run every time Metricbeat polls Elasticsearch.

So if a master switch happens, Metricbeat would suddenly start emitting the log message when it wasn't previously. I think this is good enough if we're looking at logs over a long enough period of time?

15

The log messages I had in mind:

As soon as the Metricset starts talking to a master (once):

Start collecting elasticsearch.index metricset data as talking to a master node.

As soon as not talking anymore to master, logging once:

Stop collecting elasticsearch.index metricset data as not talking to a master node

Both log messages happen only once when the switch happens.

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.