Filebeat monitoring metrics not visible in ElasticSearch

Hi,
I have posted this question on SO but could not get a good answer yet .... Basically I think I have configured Fielbeat + ES according to the docs , to send Filebeat monitored metrics into ES - but nothing is showing up in ES.

All the details and configuration are in the post:

any ideas what I am missing?
thank you
Marina

Hi!

I see the following configuration:

output.elasticsearch:
  enabled: true
  index: "ibc-parsed-logs"
  parameters.pipeline: "geoip-info"
  hosts: ${ES_HOSTS}

  # Authentication credentials - either API key or username/password.
  api_key: ${ES_API_KEY}

In this, are the values you are using correct (hosts etc) ?

Also I would suggest you run Filebeat with debug logging enabled so as to check if there is better indicator in debug messages.

Hi, Chris, thank you for your reply.
The Elasticsearch settings are correct - because the events I am ingesting form the GCP Pubsub input are getting indexed into ES - I can see them in Kibana/via queries.
It is only the Filebeat metrics that are not getting sent/indexed into ES.

What specific logging should I turn on for Filebeat to debug further?
thank you!

Hey, please have a look at Configure logging | Filebeat Reference [8.4] | Elastic. When you have the logs feel free to share them here so as I can have a look.

Thanks, Chris. So, I have enabled DEBUG for all categories, and got a log file with tons of logs :slight_smile:
From what I see - the Filebeat connect to a correct ES instance, and is trying to send metrics too - but they just are not preserved" in the ES cluster in any index that I can find.

here is an excerpt from the logs that I think shows that the filebeat is sending metrics to ES:

2022-10-04T14:25:32.917Z	DEBUG	[esclientleg]	transport/logging.go:41	Completed dialing successfully	{"network": "tcp", "address": "XXX.us-east4.gcp.elastic-cloud.com:443"}
2022-10-04T14:25:32.945Z	DEBUG	[esclientleg]	eslegclient/connection.go:272	Ping status code: 200
2022-10-04T14:25:32.946Z	INFO	[esclientleg]	eslegclient/connection.go:273	Attempting to connect to Elasticsearch version 7.15.0
2022-10-04T14:25:32.947Z	DEBUG	[esclientleg]	eslegclient/connection.go:328	GET https://XXX.us-east4.gcp.elastic-cloud.com:443/_xpack?filter_path=features.monitoring.enabled  <nil>
2022-10-04T14:25:32.982Z	DEBUG	[monitoring]	elasticsearch/client.go:101	XPack monitoring is enabled
2022-10-04T14:25:32.983Z	INFO	[monitoring]	elasticsearch/elasticsearch.go:244	Successfully connected to X-Pack Monitoring endpoint.
2022-10-04T14:25:32.984Z	DEBUG	[monitoring]	elasticsearch/elasticsearch.go:250	Finish monitoring endpoint init loop.
2022-10-04T14:25:32.984Z	INFO	[monitoring]	elasticsearch/elasticsearch.go:258	Start monitoring state metrics snapshot loop with period 1m0s.
2022-10-04T14:25:32.984Z	INFO	[monitoring]	elasticsearch/elasticsearch.go:258	Start monitoring stats metrics snapshot loop with period 10s.
2022-10-04T14:25:41.061Z	DEBUG	[input]	input/input.go:139	Run input
2022-10-04T14:25:42.959Z	DEBUG	[monitoring]	processing/processors.go:203	Publish event: {
  "@timestamp": "2022-10-04T14:25:42.950Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.15.0",
    "type": "beats_stats",
    "interval_ms": 10000,
    "params": {
      "interval": "10s"
    }
  },
  "beat": {
    "type": "filebeat",
    "version": "7.15.0",
    "name": "9975cbe98075",
    "host": "9975cbe98075",
    "uuid": "08e8a88e-e214-4d48-a65c-d5b5226776a5"
  },
  "metrics": {
    "system": {
      "cpu": {
        "cores": 8
      },
      "load": {
        "1": 0.04,
        "5": 0.01,
        "15": 0,
        "norm": {
          "1": 0.005,
          "5": 0.0013,
          "15": 0
        }
      }
    },
    "beat": {
      "cgroup": {
        "cpuacct": {
          "id": "/",
          "total": {
            "ns": 596922278
          }
        },
        "memory": {
          "id": "/",
          "mem": {
            "limit": {
              "bytes": 9223372036854771712
            },
            "usage": {
              "bytes": 46735360
            }
          }
        },
        "cpu": {
          "stats": {
            "periods": 0,
            "throttled": {
              "ns": 0,
              "periods": 0
            }
          },
          "id": "/",
          "cfs": {
            "period": {
              "us": 100000
            },
            "quota": {
              "us": 0
            }
          }
        }
      },
      "handles": {
        "open": 20,
        "limit": {
          "hard": 1048576,
          "soft": 1048576
        }
      },
      "info": {
        "uptime": {
          "ms": 12033
        },
        "ephemeral_id": "3dac65ba-ee80-4333-8eeb-e46106b369c8",
        "version": "7.15.0"
      },
      "memstats": {
        "memory_alloc": 13034112,
        "memory_sys": 76104712,
        "gc_next": 21276432,
        "rss": 116137984,
        "memory_total": 64822632
      },
      "cpu": {
        "total": {
          "time": {
            "ms": 549
          },
          "value": 540,
          "ticks": 540
        },
        "user": {
          "time": {
            "ms": 323
          },
          "ticks": 320
        },
        "system": {
          "ticks": 220,
          "time": {
            "ms": 226
          }
        }
      },
      "runtime": {
        "goroutines": 71
      }
    },
    "registrar": {
      "states": {
        "current": 0,
        "update": 0,
        "cleanup": 0
      },
      "writes": {
        "success": 0,
        "total": 0,
        "fail": 0
      }
    },
    "filebeat": {
      "harvester": {
        "started": 0,
        "closed": 0,
        "running": 0,
        "open_files": 0,
        "skipped": 0
      },
      "input": {
        "netflow": {
          "packets": {
            "dropped": 0,
            "received": 0
          },
          "flows": 0
        },
        "log": {
          "files": {
            "renamed": 0,
            "truncated": 0
          }
        }
      },
      "events": {
        "done": 0,
        "active": 0,
        "added": 0
      }
    },
    "libbeat": {
      "output": {
        "read": {
          "bytes": 0,
          "errors": 0
        },
        "type": "elasticsearch",
        "events": {
          "batches": 0,
          "total": 0,
          "acked": 0,
          "failed": 0,
          "dropped": 0,
          "duplicates": 0,
          "active": 0,
          "toomany": 0
        },
        "write": {
          "errors": 0,
          "bytes": 0
        }
      },
      "pipeline": {
        "clients": 1,
        "events": {
          "active": 0,
          "total": 0,
          "filtered": 0,
          "published": 0,
          "failed": 0,
          "dropped": 0,
          "retry": 0
        },
        "queue": {
          "acked": 0,
          "max_events": 4096
        }
      },
      "config": {
        "scans": 0,
        "reloads": 0,
        "module": {
          "starts": 0,
          "stops": 0,
          "running": 0
        }
      }
    }
  }
}
2022-10-04T14:25:42.964Z	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to backoff(monitoring(https://XXX.us-east4.gcp.elastic-cloud.com:443))
2022-10-04T14:25:42.964Z	DEBUG	[monitoring]	elasticsearch/client.go:66	Monitoring client: connect.
2022-10-04T14:25:42.965Z	DEBUG	[esclientleg]	eslegclient/connection.go:249	ES Ping(url=https://XXX.us-east4.gcp.elastic-cloud.com:443)
2022-10-04T14:25:42.964Z	INFO	[monitoring]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2022-10-04T14:25:42.966Z	INFO	[monitoring]	pipeline/retry.go:223	  done
2022-10-04T14:25:43.015Z	DEBUG	[esclientleg]	eslegclient/connection.go:272	Ping status code: 200
2022-10-04T14:25:43.016Z	INFO	[esclientleg]	eslegclient/connection.go:273	Attempting to connect to Elasticsearch version 7.15.0
2022-10-04T14:25:43.017Z	DEBUG	[esclientleg]	eslegclient/connection.go:328	GET https://XXX.us-east4.gcp.elastic-cloud.com:443/_xpack?filter_path=features.monitoring.enabled  <nil>
2022-10-04T14:25:43.205Z	DEBUG	[monitoring]	elasticsearch/client.go:101	XPack monitoring is enabled
2022-10-04T14:25:43.207Z	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to backoff(monitoring(https://XXX.us-east4.gcp.elastic-cloud.com:443)) established
2022-10-04T14:25:43.239Z	DEBUG	[monitoring]	memqueue/ackloop.go:160	ackloop: receive ack [0: 0, 1]

However, when I go to ES, I do not see any indices created for metrics... the only indices I see there are for my events/data, and for APM .

Here is full log: filebeat full log - 8021ff33

After further struggles with getting the actual ES logs (which is not that simple on hosted ES clusters at elastic.io!!) - I finally was able to access them and noticed many of the following WARNings:

16:32:50.998
elasticsearch.server
[elasticsearch.server][WARN] Authentication to realm found failed - Password authentication failed for ec-local-beats-monitor

I'm not sure who this user might be: "ec-local-beats-monitor" ?? and where it is configured/set - definitely not in my filebeat.yml config for the Filebeat process...

Could this be related to the issue with loosing/ not indexing Filebeat metrics data into ES?

Hi @ppine7 !

ec-local-beats-monitor is an internal user managed by the Elasticsearch service. The error is curious but shouldn't cause any problems collecting metrics from your filebeat process.

As for the filebeat metrics, the config you posted on stack overflow looks reasonable. Monitor Filebeat | Filebeat Reference [8.4] | Elastic is the main doc page for that. Either internal collection or metricbeat collection are supported.

Once it's working you should see the beats in the kibana stack monitoring UI which is driven by the .monitoring-* indices. You shouldn't need to change any cluster settings since the Elasticsearch service (elastic.io, cloud.elastic.co) should already enable the necessary settings.

If the problem persists you might want to open a case on https://cloud.elastic.co/support so we can check for any issues with your cluster.

Hi, Mat, thank you for your reply!
Indeed, there are no .monitoring-* indices at all, not a single one, for beats or anything else ...
I did open a case with the Elastic support - so hopefully we will solve this mystery ....

thanks,
Marina

No problem :slight_smile: Hopefully support can help get things sorted out.

One thing to note is that since the indices are internal, they might not show up in certain views. If you're not sure I'd recommend trying a GET .monitoring-beats-*/_search in dev tools.

If you get no hits then indeed the monitoring data isn't coming in. So we'll need to figure out why.

Make sure to include your filebeat configuration for support to check for errors there as well.

Thanks, Mat!
Just tried:

no indices:

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : 0.0,
    "hits" : [ ]
  }
}

thank you for the ideas to try - I am going in circles on this one :slight_smile:

Hi again @ppine7 !

So I think we might have some bugs here. I'll have to do a little more research but I think:

  1. when using API keys, filebeat internal collection writes to the wrong ES by default. This might also happen regardless of API keys (it should write through the prod deployment to the monitoring deployment, but that didn't happen in my tests)
  2. it might also be inspecting the UUID incorrectly, or caching it somehow (in my tests, I had the monitoring deployment ES UUID on some docs that landed in the production deployment)
  3. kibana isn't showing a "Standalone" entry for a single filebeat using internal collection, which it should (so to work around this, you'll need elasticsearch monitoring enabled as well)

I had to work around all 3 of those locally to get your setup working as expected.

Here's how the configuration ended up: Example filebeat monitoring setup · GitHub

This assumes both a "prod" deployment (to receive the logs) and a "monitoring" deployment (to receive the monitoring data) and monitoring needs to be configured for the "prod" ES as well. On cloud.elastic.co, you can do this using the "Logs & Metrics" section of the UI.

If it's a single deployment, you can have it report the monitoring data to itself, but that can cause performance degradation as scale increases.

Hi, Mat,
Thank you so much for spending the time to troubleshoot and test out possible solutions to this issue! Really appreciate it!

So, as it happens, I have received the same suggestions as your Points 1 and 2 (not 3 , yet) from the Elastic support case I have opened. and here is the updated config of filebeat.yml I have tried:


# =============================== Elastic Cloud ================================
cloud.id: '${CLOUD_ID}'

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  enabled: true
  index: "ibc-parsed-logs"
  parameters.pipeline: "geoip-info"
  hosts: ${ES_HOSTS}
  protocol: "https"
  api_key: ${ES_API_KEY}

# ================================== Logging ===================================
logging.metrics.enabled: true
logging.enabled: true
logging.level: debug
logging.to_files: true
logging.files:
  path: /usr/share/filebeat/f_logs
  name: filebeat
  keepfiles: 10
  permissions: 0640
logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
monitoring.enabled: true
monitoring.cluster_uuid: '${CLOUD_ID}'

monitoring.elasticsearch:
  hosts: ${ES_HOSTS}
  api_key: ${ES_API_KEY}

with everything else being the same.

It still fails to send the monitoring data to ES - but now I could also see the following error in the filebeat logs:

2022-10-17T01:44:56.239Z	INFO	[monitoring]	elasticsearch/elasticsearch.go:231	Failed to connect to Elastic X-Pack Monitoring. Either Elasticsearch X-Pack monitoring is not enabled or Elasticsearch is not available. Will keep retrying. Error: cannot connect underlying Elasticsearch client: 403 Forbidden: <html lang="en"><head><title>Elastic</title><link href="/44040/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.css" rel="stylesheet"/>

I have not heard back from the support case about this new error - but it gives an idea that maybe there are some permissions missing in the ES cluster for filebeat to send the monitoring data in?

Again, it send the real data/events to ES just fine, they are indexed and I can see them, it's only the monitoring events that have this the issue. For the record, our main and monitoring ES clusters are one and the same.

One more interesting thing I noticed about your setup:
you specify connection URL to ES cluster[s] using port 443;
I specify port 9243:
For example something like this:
-e ES_HOSTS="https://mycluster.kb.us-east4.gcp.elastic-cloud.com:9243"
Does it matter?

Again, thank you for spending so much time on this issue!
Marina

1 Like

Hi @ppine7

The cluster uuid is not the cloud id... 2 different things

per the docs

to get a cluster’s cluster_uuid , call the GET / API against that production cluster.

In fact if you are sending the filebeat monitoring data to the same cloud instance as the as the filebeat data the only monitoring setting you need is the following everything else is inherited from the other settings / elasticsearch output settings

monitoring.enabled: true

I do this all the time works with just that.

That is the kibana endpoint not the elasticsearch endpoint that would explain a lot...

"https://mycluster.kb.us-east4.gcp.elastic-cloud.com:9243"
...................^^<--- should be .es.

Should be

"https://mycluster.es.us-east4.gcp.elastic-cloud.com:9243"

Also make sure you are using the correct format for the API keys there are different formats ... not base64 but the Beats Format

I just turned on Monitoring for the First Time on a new Cluster....

Just set the following...

output.elasticsearch:
  hosts: ["https://abcdefg.es.us-west2.gcp.elastic-cloud.com:443"]
  api_key: "pQasdsdfasdfgsScRN_T_T9:X70m3osadfBgHFO4-bkg"

`monitoring.enabled: true`

And it works right away

1 Like

Glad to hear you're making progress on the support case @ppine7! And thanks @stephenb for the demonstration. :orange_heart:

It's probably best to stay focused on your support case since it's where we can share details about your deployment.

I'll stay subscribed here incase other questions arise.

Just a note that I did a local test today on es/kibana/filebeat main and the Stack Monitoring UI was able to show a standalone beat with this config:

filebeat.inputs:
- type: filestream
  id: logs
  paths:
    - /logs/*.log

monitoring:
  enabled: true

output.elasticsearch:
  hosts: "http://localhost:9200"
  api_key: "(beats key)"

As well as a similar standalone cluster on ESS 8.4.3

The monitoring data still doesn't go via the production cluster, but I think it's likely I'm missing some nuance and this is working as expected. I think once support has a look at your deployment, we should be able to sort out the gap for you @ppine7 .

Thank you Mat and Stephen!!!!
So, after reading all your suggestions - decided to bite the bullet and start from scratch! :slight_smile:

so I did the following:

  1. got a new ES8.4.3 cloud cluster - clean, with no additional monitoring settings
  2. updated my filebeat.yml to use minimal monitoring config - only set the " monitoring.enabled: true" setting
    here is what it looks like now: (a couple of extra settings for pipelines - leaving them here just in case they might be a problem - unlikely)
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  enabled: true
  index: "ibc-parsed-logs"
  parameters.pipeline: "geoip-info"
  hosts: ${ES_HOSTS}
  protocol: "https"
  # Authentication credentials - either API key or username/password.
  api_key: ${ES_API_KEY}

  # The maximum number of events to bulk in a single Elasticsearch bulk API index request.
  # The default is 50.
  #bulk_max_size: 100

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - decode_json_fields:
      fields: ["message"]
      add_error_key: true
      document_id: "event_uuid"

# ================================== Logging ===================================

logging.metrics.enabled: true
logging.enabled: true

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: debug
logging.to_files: true
logging.files:
  path: /usr/share/filebeat/f_logs
  name: filebeat
  keepfiles: 10
  permissions: 0640

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
monitoring.enabled: true

  1. make sure I have correct cloud.id, 'hosts' and api key values (beats format)

Now, when I run filebeat, I do not see any errors anymore (!) and I can see in filebeat's logs that the monitoring events are being sent to the ES cluster:

{"log.level":"debug","@timestamp":"2022-10-18T14:55:21.509Z","log.logger":"monitoring","log.origin":{"file.name":"processing/processors.go","file.line":210},"message":"Publish event: {\n  \"@timestamp\": \"2022-10-18T14:55:21.505Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.4.3\",\n    \"type\": \"beats_stats\",\n    \"interval_ms\": 10000,\n    \"params\": {\n      \"interval\": \"10s\"\n    }\n  },\n  \"metrics\": {\n    \"registrar\": {\n      \"states\": {\n        \"current\": 0,\n        \"update\": 0,\n        \"cleanup\": 0\n      },\n      \"writes\": {\n        \"total\": 0,\n        \"fail\": 0,\n        \"success\": 0\n      }\n    },\n    \"filebeat\": {\n      \"input\": {\n        \"netflow\": {\n          \"flows\": 0,\n          \"packets\": {\n            \"received\": 0,\n            \"dropped\": 0\n          }\n        },\n        \"log\": {\n          \"files\": {\n            \"renamed\": 0,\n            \"truncated\": 0\n          }\n        }\n      },\n      \"events\": {\n        \"added\": 0,\n        \"done\": 0,\n        \"active\": 0\n      },\n      \"harvester\": {\n        \"closed\": 0,\n        \"running\": 0,\n        \"open_files\": 0,\n        \"skipped\": 0,\n        \"started\": 0\n      }\n    },\n    \"libbeat\": {\n      \"pipeline\": {\n        \"queue\": {\n          \"max_events\": 4096,\n          \"acked\": 0\n        },\n        \"clients\": 1,\n        \"events\": {\n          \"retry\": 0,\n          \"active\": 0,\n          \"total\": 0,\n          \"filtered\": 0,\n          \"published\": 0,\n          \"failed\": 0,\n          \"dropped\": 0\n        }\n      },\n      \"config\": {\n        \"module\": {\n          \"running\": 0,\n          \"starts\": 0,\n          \"stops\": 0\n        },\n        \"scans\": 0,\n        \"reloads\": 0\n      },\n      \"output\": {\n        \"read\": {\n          \"bytes\": 0,\n          \"errors\": 0\n        },\n        \"type\": \"elasticsearch\",\n        \"events\": {\n          \"failed\": 0,\n          \"dropped\": 0,\n          \"duplicates\": 0,\n          \"active\": 0,\n          \"toomany\": 0,\n          \"batches\": 0,\n          \"total\": 0,\n          \"acked\": 0\n        },\n        \"write\": {\n          \"bytes\": 0,\n          \"errors\": 0\n        }\n      }\n    },\n    \"beat\": {\n      \"cpu\": {\n        \"total\": {\n          \"time\": {\n            \"ms\": 1820\n          },\n          \"value\": 1820,\n          \"ticks\": 1820\n        },\n        \"user\": {\n          \"ticks\": 700,\n          \"time\": {\n            \"ms\": 700\n          }\n        },\n        \"system\": {\n          \"ticks\": 1120,\n          \"time\": {\n            \"ms\": 1120\n          }\n        }\n      },\n      \"runtime\": {\n        \"goroutines\": 76\n      },\n      \"info\": {\n        \"version\": \"8.4.3\",\n        \"uptime\": {\n          \"ms\": 180354\n        },\n        \"ephemeral_id\": \"68d58d94-3678-4cda-9759-7348b4354bd9\",\n        \"name\": \"filebeat\"\n      },\n      \"cgroup\": {\n        \"cpuacct\": {\n          \"id\": \"/\",\n          \"total\": {\n            \"ns\": 1875475443\n          }\n        },\n        \"memory\": {\n          \"id\": \"/\",\n          \"mem\": {\n            \"usage\": {\n              \"bytes\": 52338688\n            },\n            \"limit\": {\n              \"bytes\": 9223372036854771712\n            }\n          }\n        },\n        \"cpu\": {\n          \"id\": \"/\",\n          \"cfs\": {\n            \"quota\": {\n              \"us\": 0\n            },\n            \"period\": {\n              \"us\": 100000\n            }\n          },\n          \"stats\": {\n            \"periods\": 0,\n            \"throttled\": {\n              \"periods\": 0,\n              \"ns\": 0\n            }\n          }\n        }\n      },\n      \"handles\": {\n        \"open\": 19,\n        \"limit\": {\n          \"soft\": 1048576,\n          \"hard\": 1048576\n        }\n      },\n      \"memstats\": {\n        \"memory_sys\": 34161672,\n        \"gc_next\": 20490032,\n        \"rss\": 138559488,\n        \"memory_total\": 67309832,\n        \"memory_alloc\": 13446248\n      }\n    },\n    \"system\": {\n      \"cpu\": {\n        \"cores\": 8\n      },\n      \"load\": {\n        \"1\": 0.09,\n        \"5\": 0.1,\n        \"15\": 0.04,\n        \"norm\": {\n          \"15\": 0.005,\n          \"1\": 0.0113,\n          \"5\": 0.0125\n        }\n      }\n    }\n  },\n  \"beat\": {\n    \"uuid\": \"1fc0f527-3173-46be-a029-ca70c420c1c7\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.4.3\",\n    \"name\": \"ccf8bd83efd8\",\n    \"host\": \"ccf8bd83efd8\"\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}

{"log.level":"debug","@timestamp":"2022-10-18T14:55:21.545Z","log.logger":"monitoring","log.origin":{"file.name":"memqueue/eventloop.go","file.line":197},"message":"handle ACKs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-18T14:55:21.545Z","log.logger":"monitoring","log.origin":{"file.name":"memqueue/eventloop.go","file.line":216},"message":"try ack index: (idx=0, i=0, seq=0)\n","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-18T14:55:21.546Z","log.logger":"monitoring","log.origin":{"file.name":"memqueue/eventloop.go","file.line":220},"message":"no state set","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-18T14:55:21.547Z","log.logger":"monitoring","log.origin":{"file.name":"memqueue/eventloop.go","file.line":199},"message":"handle ACK took: 2.2609ms","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-18T14:55:21.547Z","log.logger":"monitoring","log.origin":{"file.name":"memqueue/ackloop.go","file.line":95},"message":"ackloop: return ack to broker loop:1","service.name":"filebeat","ecs.version":"1.6.0"}

HOWEVER, when I go to Stack Monitoring in the ES cluster - I still see nothing - no .monitoring indices are create.... And in fact, I don't see anything at ll like what you showed....
This is what I get when go to Stack Monitoring:

I feel I am sooooo close and yet ....
Is this something in configuration or permissions missing, potentially?

Thank you !!

Ok @ppine7 yes I think you are close...

Now you need to do one more thing I think this IS perhaps a bug... you will not see the metricbeat monitoring without the rest of the cluster monitoring.

So go to the cloud console and turn on metrics monitoring for your cluster and ship the data to the same cluster.

You can skip the logs for now...

Then when you go back to monitoring you should see the cluster + the metricbeat monitoring.

Thanks, Stephen!
Ok, enabled Metrics per your advise. Now we can see *some" monitoring in the Stack Monitoring: of ES itself and Kibana:

but not of Beats :frowning:

Checking in Dev Tools - the only monitoring -related indices that appeared now are these:

GET /_cat/indices/.monitoring-*

green open .monitoring-es-7-2022.10.18                   ZSLCzJRBRiS2r4qihIw71w 1 1  385 34 690.3kb 347.2kb
green open .ds-.monitoring-kibana-8-mb-2022.10.18-000001 Xvq_P_9NRiKy3hYxPwBwmQ 1 1 1010  0   2.3mb   1.5mb
green open .ds-.monitoring-es-8-mb-2022.10.18-000001     2H4hLmyGS8q2-EQfbuynQQ 1 1 8395  0  11.8mb   6.1mb
green open .monitoring-kibana-7-2022.10.18               4kofsQNNTzylE-YWoEMYXg 1 1   76  0 371.8kb 165.7kb

and I can still see in the Filebeat logs that it is trying to send the metrics data:

{"log.level":"debug","@timestamp":"2022-10-18T18:58:33.891Z","log.logger":"monitoring","log.origin":{"file.name":"processing/processors.go","file.line":210},"message":"Publish event: {\n  \"@timestamp\": \"2022-10-18T18:58:33.886Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.4.3\",\n    \"type\": \"beats_stats\",\n    \"interval_ms\": 10000,\n    \"params\": {\n      \"interval\": \"10s\"\n    },\n    \"cluster_uuid\": \"9PxnN-9PToumnPWp0L4grQ\"\n  },\n  \"beat\": {\n    \"uuid\": \"031e0829-3ab6-40a4-af63-8a6209c0d53d\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.4.3\",\n    \"name\": \"572d93de94ee\",\n    \"host\": \"572d93de94ee\"\n  },\n  \"metrics\": {\n    \"libbeat\": {\n      \"pipeline\": {\n        \"events\": {\n          \"dropped\": 0,\n          \"retry\": 1,\n          \"active\": 0,\n          \"total\": 1,\n          \"filtered\": 0,\n          \"published\": 1,\n          \"failed\": 0\n        },\n        \"queue\": {\n          \"max_events\": 4096,\n          \"acked\": 1\n        },\n        \"clients\": 1\n      },\n      \"config\": {\n        \"reloads\": 0,\n        \"module\": {\n          \"stops\": 0,\n          \"running\": 0,\n          \"starts\": 0\n        },\n        \"scans\": 0\n      },\n      \"output\": {\n        \"events\": {\n          \"dropped\": 1,\n          \"duplicates\": 0,\n          \"active\": 0,\n          \"toomany\": 0,\n          \"batches\": 1,\n          \"total\": 1,\n          \"acked\": 0,\n          \"failed\": 0\n        },\n        \"write\": {\n          \"errors\": 0,\n          \"bytes\": 3611\n        },\n        \"read\": {\n          \"bytes\": 3458,\n          \"errors\": 1\n        },\n        \"type\": \"elasticsearch\"\n      }\n    },\n    \"beat\": {\n      \"cpu\": {\n        \"total\": {\n          \"time\": {\n            \"ms\": 2860\n          },\n          \"value\": 2860,\n          \"ticks\": 2860\n        },\n        \"user\": {\n          \"time\": {\n            \"ms\": 1220\n          },\n          \"ticks\": 1220\n        },\n        \"system\": {\n          \"ticks\": 1640,\n          \"time\": {\n            \"ms\": 1640\n          }\n        }\n      },\n      \"runtime\": {\n        \"goroutines\": 54\n      },\n      \"info\": {\n        \"ephemeral_id\": \"43298808-e0b4-490c-b516-b56328f33b75\",\n        \"name\": \"filebeat\",\n        \"version\": \"8.4.3\",\n        \"uptime\": {\n          \"ms\": 250626\n        }\n      },\n      \"cgroup\": {\n        \"cpu\": {\n          \"cfs\": {\n            \"period\": {\n              \"us\": 100000\n            },\n            \"quota\": {\n              \"us\": 0\n            }\n          },\n          \"stats\": {\n            \"periods\": 0,\n            \"throttled\": {\n              \"periods\": 0,\n              \"ns\": 0\n            }\n          },\n          \"id\": \"/\"\n        },\n        \"cpuacct\": {\n          \"id\": \"/\",\n          \"total\": {\n            \"ns\": 2961870248\n          }\n        },\n        \"memory\": {\n          \"id\": \"/\",\n          \"mem\": {\n            \"limit\": {\n              \"bytes\": 9223372036854771712\n            },\n            \"usage\": {\n              \"bytes\": 61280256\n            }\n          }\n        }\n      },\n      \"handles\": {\n        \"open\": 12,\n        \"limit\": {\n          \"hard\": 1048576,\n          \"soft\": 1048576\n        }\n      },\n      \"memstats\": {\n        \"memory_total\": 71525392,\n        \"memory_alloc\": 13243936,\n        \"memory_sys\": 34685960,\n        \"gc_next\": 21063872,\n        \"rss\": 140554240\n      }\n    },\n    \"system\": {\n      \"cpu\": {\n        \"cores\": 8\n      },\n      \"load\": {\n        \"1\": 0.23,\n        \"5\": 0.22,\n        \"15\": 0.16,\n        \"norm\": {\n          \"1\": 0.0288,\n          \"5\": 0.0275,\n          \"15\": 0.02\n        }\n      }\n    },\n    \"registrar\": {\n      \"states\": {\n        \"update\": 0,\n        \"cleanup\": 0,\n        \"current\": 0\n      },\n      \"writes\": {\n        \"total\": 0,\n        \"fail\": 0,\n        \"success\": 0\n      }\n    },\n    \"filebeat\": {\n      \"harvester\": {\n        \"started\": 0,\n        \"closed\": 0,\n        \"running\": 0,\n        \"open_files\": 0,\n        \"skipped\": 0\n      },\n      \"input\": {\n        \"log\": {\n          \"files\": {\n            \"renamed\": 0,\n            \"truncated\": 0\n          }\n        },\n        \"netflow\": {\n          \"flows\": 0,\n          \"packets\": {\n            \"received\": 0,\n            \"dropped\": 0\n          }\n        }\n      },\n      \"events\": {\n        \"added\": 1,\n        \"done\": 1,\n        \"active\": 0\n      }\n    }\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}

Something else possibly missing?

that should be

pipeline: "geoip-info"

That parameters is putting info into the headers which I do not think you want to do, I don't think that is the problem but it not correct..but it could be a problem...

After you start filebeat with the monitoring enabled....

You should see interesting that it says -7- this is filebeat / metricbeat 8.4.1

GET _cat/indices/.monitoring-beats-*?v

health status index                                        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .monitoring-beats-7-2022.10.18               RQ3yi65GR66LIDFADeWPWw   1   1          9            0    313.7kb        156.8kb

note there may be one but that is the internal monitoring of the APM / Fleet agent

.ds-.monitoring-beats-8-mb-2022.10.18-000001

And you should see something like this in your filebeat logs
this the Connection to Monitoring!

{"log.level":"info","@timestamp":"2022-10-18T14:09:25.099-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(monitoring(https://**************.es.us-west2.gcp.elastic-cloud.com:443)) established","service.name":"filebeat","ecs.version":"1.6.0"}

You might commenting out all the logging stuff just to test, I do not have any of it.

This is my entire filebeat not just snippets it is the entire working filebeat

filebeat.inputs:
- type: log
  enabled: true
  paths:
   -  /Users/sbrown/workspace/sample-data/discuss/container-mixed/*.log

output.elasticsearch:
  hosts: ["https://*********.es.us-west2.gcp.elastic-cloud.com:443"]
  api_key: "asdfasdfasdfoHNQkCvBgHFO4-bkg"

monitoring.enabled: true

1 Like

@ppine7 I'd check if the cluster_uuid shown on the documents in .monitoring-beats-* matches the cluster_uuid shown in the stack monitoring UI (it'll be in the browser bar, or GET / on the monitored cluster).

Like I mentioned above

  1. it might also be inspecting the UUID incorrectly, or caching it somehow (in my tests, I had the monitoring deployment ES UUID on some docs that landed in the production deployment)

At one point in my testing, filebeat seemed to be getting sending docs with the wrong cluster UUID. I didn't figure out why.

The UI expects either a cluster UUID that matches an ES cluster that's also being monitored or a blank cluster UUID (these will show up as a "Standalone Cluster" entry in the UI).

If the docs have a non-monitored cluster UUID, they won't show up in the Stack Monitoring UI.

1 Like