Hi,
I have posted this question on SO but could not get a good answer yet .... Basically I think I have configured Fielbeat + ES according to the docs , to send Filebeat monitored metrics into ES - but nothing is showing up in ES.
All the details and configuration are in the post:
any ideas what I am missing?
thank you
Marina
Hi!
I see the following configuration:
output.elasticsearch:
enabled: true
index: "ibc-parsed-logs"
parameters.pipeline: "geoip-info"
hosts: ${ES_HOSTS}
# Authentication credentials - either API key or username/password.
api_key: ${ES_API_KEY}
In this, are the values you are using correct (hosts etc) ?
Also I would suggest you run Filebeat with debug logging enabled so as to check if there is better indicator in debug messages.
Hi, Chris, thank you for your reply.
The Elasticsearch settings are correct - because the events I am ingesting form the GCP Pubsub input are getting indexed into ES - I can see them in Kibana/via queries.
It is only the Filebeat metrics that are not getting sent/indexed into ES.
What specific logging should I turn on for Filebeat to debug further?
thank you!
Hey, please have a look at Configure logging | Filebeat Reference [8.4] | Elastic. When you have the logs feel free to share them here so as I can have a look.
Thanks, Chris. So, I have enabled DEBUG for all categories, and got a log file with tons of logs 
From what I see - the Filebeat connect to a correct ES instance, and is trying to send metrics too - but they just are not preserved" in the ES cluster in any index that I can find.
here is an excerpt from the logs that I think shows that the filebeat is sending metrics to ES:
2022-10-04T14:25:32.917Z DEBUG [esclientleg] transport/logging.go:41 Completed dialing successfully {"network": "tcp", "address": "XXX.us-east4.gcp.elastic-cloud.com:443"}
2022-10-04T14:25:32.945Z DEBUG [esclientleg] eslegclient/connection.go:272 Ping status code: 200
2022-10-04T14:25:32.946Z INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.15.0
2022-10-04T14:25:32.947Z DEBUG [esclientleg] eslegclient/connection.go:328 GET https://XXX.us-east4.gcp.elastic-cloud.com:443/_xpack?filter_path=features.monitoring.enabled <nil>
2022-10-04T14:25:32.982Z DEBUG [monitoring] elasticsearch/client.go:101 XPack monitoring is enabled
2022-10-04T14:25:32.983Z INFO [monitoring] elasticsearch/elasticsearch.go:244 Successfully connected to X-Pack Monitoring endpoint.
2022-10-04T14:25:32.984Z DEBUG [monitoring] elasticsearch/elasticsearch.go:250 Finish monitoring endpoint init loop.
2022-10-04T14:25:32.984Z INFO [monitoring] elasticsearch/elasticsearch.go:258 Start monitoring state metrics snapshot loop with period 1m0s.
2022-10-04T14:25:32.984Z INFO [monitoring] elasticsearch/elasticsearch.go:258 Start monitoring stats metrics snapshot loop with period 10s.
2022-10-04T14:25:41.061Z DEBUG [input] input/input.go:139 Run input
2022-10-04T14:25:42.959Z DEBUG [monitoring] processing/processors.go:203 Publish event: {
"@timestamp": "2022-10-04T14:25:42.950Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.15.0",
"type": "beats_stats",
"interval_ms": 10000,
"params": {
"interval": "10s"
}
},
"beat": {
"type": "filebeat",
"version": "7.15.0",
"name": "9975cbe98075",
"host": "9975cbe98075",
"uuid": "08e8a88e-e214-4d48-a65c-d5b5226776a5"
},
"metrics": {
"system": {
"cpu": {
"cores": 8
},
"load": {
"1": 0.04,
"5": 0.01,
"15": 0,
"norm": {
"1": 0.005,
"5": 0.0013,
"15": 0
}
}
},
"beat": {
"cgroup": {
"cpuacct": {
"id": "/",
"total": {
"ns": 596922278
}
},
"memory": {
"id": "/",
"mem": {
"limit": {
"bytes": 9223372036854771712
},
"usage": {
"bytes": 46735360
}
}
},
"cpu": {
"stats": {
"periods": 0,
"throttled": {
"ns": 0,
"periods": 0
}
},
"id": "/",
"cfs": {
"period": {
"us": 100000
},
"quota": {
"us": 0
}
}
}
},
"handles": {
"open": 20,
"limit": {
"hard": 1048576,
"soft": 1048576
}
},
"info": {
"uptime": {
"ms": 12033
},
"ephemeral_id": "3dac65ba-ee80-4333-8eeb-e46106b369c8",
"version": "7.15.0"
},
"memstats": {
"memory_alloc": 13034112,
"memory_sys": 76104712,
"gc_next": 21276432,
"rss": 116137984,
"memory_total": 64822632
},
"cpu": {
"total": {
"time": {
"ms": 549
},
"value": 540,
"ticks": 540
},
"user": {
"time": {
"ms": 323
},
"ticks": 320
},
"system": {
"ticks": 220,
"time": {
"ms": 226
}
}
},
"runtime": {
"goroutines": 71
}
},
"registrar": {
"states": {
"current": 0,
"update": 0,
"cleanup": 0
},
"writes": {
"success": 0,
"total": 0,
"fail": 0
}
},
"filebeat": {
"harvester": {
"started": 0,
"closed": 0,
"running": 0,
"open_files": 0,
"skipped": 0
},
"input": {
"netflow": {
"packets": {
"dropped": 0,
"received": 0
},
"flows": 0
},
"log": {
"files": {
"renamed": 0,
"truncated": 0
}
}
},
"events": {
"done": 0,
"active": 0,
"added": 0
}
},
"libbeat": {
"output": {
"read": {
"bytes": 0,
"errors": 0
},
"type": "elasticsearch",
"events": {
"batches": 0,
"total": 0,
"acked": 0,
"failed": 0,
"dropped": 0,
"duplicates": 0,
"active": 0,
"toomany": 0
},
"write": {
"errors": 0,
"bytes": 0
}
},
"pipeline": {
"clients": 1,
"events": {
"active": 0,
"total": 0,
"filtered": 0,
"published": 0,
"failed": 0,
"dropped": 0,
"retry": 0
},
"queue": {
"acked": 0,
"max_events": 4096
}
},
"config": {
"scans": 0,
"reloads": 0,
"module": {
"starts": 0,
"stops": 0,
"running": 0
}
}
}
}
}
2022-10-04T14:25:42.964Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(monitoring(https://XXX.us-east4.gcp.elastic-cloud.com:443))
2022-10-04T14:25:42.964Z DEBUG [monitoring] elasticsearch/client.go:66 Monitoring client: connect.
2022-10-04T14:25:42.965Z DEBUG [esclientleg] eslegclient/connection.go:249 ES Ping(url=https://XXX.us-east4.gcp.elastic-cloud.com:443)
2022-10-04T14:25:42.964Z INFO [monitoring] pipeline/retry.go:219 retryer: send unwait signal to consumer
2022-10-04T14:25:42.966Z INFO [monitoring] pipeline/retry.go:223 done
2022-10-04T14:25:43.015Z DEBUG [esclientleg] eslegclient/connection.go:272 Ping status code: 200
2022-10-04T14:25:43.016Z INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.15.0
2022-10-04T14:25:43.017Z DEBUG [esclientleg] eslegclient/connection.go:328 GET https://XXX.us-east4.gcp.elastic-cloud.com:443/_xpack?filter_path=features.monitoring.enabled <nil>
2022-10-04T14:25:43.205Z DEBUG [monitoring] elasticsearch/client.go:101 XPack monitoring is enabled
2022-10-04T14:25:43.207Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(monitoring(https://XXX.us-east4.gcp.elastic-cloud.com:443)) established
2022-10-04T14:25:43.239Z DEBUG [monitoring] memqueue/ackloop.go:160 ackloop: receive ack [0: 0, 1]
However, when I go to ES, I do not see any indices created for metrics... the only indices I see there are for my events/data, and for APM .
Here is full log: filebeat full log - 8021ff33
After further struggles with getting the actual ES logs (which is not that simple on hosted ES clusters at elastic.io!!) - I finally was able to access them and noticed many of the following WARNings:
16:32:50.998
elasticsearch.server
[elasticsearch.server][WARN] Authentication to realm found failed - Password authentication failed for ec-local-beats-monitor
I'm not sure who this user might be: "ec-local-beats-monitor" ?? and where it is configured/set - definitely not in my filebeat.yml config for the Filebeat process...
Could this be related to the issue with loosing/ not indexing Filebeat metrics data into ES?
Hi @ppine7 !
ec-local-beats-monitor is an internal user managed by the Elasticsearch service. The error is curious but shouldn't cause any problems collecting metrics from your filebeat process.
As for the filebeat metrics, the config you posted on stack overflow looks reasonable. Monitor Filebeat | Filebeat Reference [8.4] | Elastic is the main doc page for that. Either internal collection or metricbeat collection are supported.
Once it's working you should see the beats in the kibana stack monitoring UI which is driven by the .monitoring-* indices. You shouldn't need to change any cluster settings since the Elasticsearch service (elastic.io, cloud.elastic.co) should already enable the necessary settings.
If the problem persists you might want to open a case on https://cloud.elastic.co/support so we can check for any issues with your cluster.
Hi, Mat, thank you for your reply!
Indeed, there are no .monitoring-* indices at all, not a single one, for beats or anything else ...
I did open a case with the Elastic support - so hopefully we will solve this mystery ....
thanks,
Marina
No problem Hopefully support can help get things sorted out.
One thing to note is that since the indices are internal, they might not show up in certain views. If you're not sure I'd recommend trying a GET .monitoring-beats-*/_search in dev tools.
If you get no hits then indeed the monitoring data isn't coming in. So we'll need to figure out why.
Make sure to include your filebeat configuration for support to check for errors there as well.
Thanks, Mat!
Just tried:
no indices:
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 0,
"successful" : 0,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : 0.0,
"hits" : [ ]
}
}
thank you for the ideas to try - I am going in circles on this one
Hi again @ppine7 !
So I think we might have some bugs here. I'll have to do a little more research but I think:
I had to work around all 3 of those locally to get your setup working as expected.
Here's how the configuration ended up: Example filebeat monitoring setup · GitHub
This assumes both a "prod" deployment (to receive the logs) and a "monitoring" deployment (to receive the monitoring data) and monitoring needs to be configured for the "prod" ES as well. On cloud.elastic.co, you can do this using the "Logs & Metrics" section of the UI.
If it's a single deployment, you can have it report the monitoring data to itself, but that can cause performance degradation as scale increases.
Hi, Mat,
Thank you so much for spending the time to troubleshoot and test out possible solutions to this issue! Really appreciate it!
So, as it happens, I have received the same suggestions as your Points 1 and 2 (not 3 , yet) from the Elastic support case I have opened. and here is the updated config of filebeat.yml I have tried:
# =============================== Elastic Cloud ================================
cloud.id: '${CLOUD_ID}'
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
enabled: true
index: "ibc-parsed-logs"
parameters.pipeline: "geoip-info"
hosts: ${ES_HOSTS}
protocol: "https"
api_key: ${ES_API_KEY}
# ================================== Logging ===================================
logging.metrics.enabled: true
logging.enabled: true
logging.level: debug
logging.to_files: true
logging.files:
path: /usr/share/filebeat/f_logs
name: filebeat
keepfiles: 10
permissions: 0640
logging.selectors: ["*"]
# ============================= X-Pack Monitoring ==============================
monitoring.enabled: true
monitoring.cluster_uuid: '${CLOUD_ID}'
monitoring.elasticsearch:
hosts: ${ES_HOSTS}
api_key: ${ES_API_KEY}
with everything else being the same.
It still fails to send the monitoring data to ES - but now I could also see the following error in the filebeat logs:
2022-10-17T01:44:56.239Z INFO [monitoring] elasticsearch/elasticsearch.go:231 Failed to connect to Elastic X-Pack Monitoring. Either Elasticsearch X-Pack monitoring is not enabled or Elasticsearch is not available. Will keep retrying. Error: cannot connect underlying Elasticsearch client: 403 Forbidden: <html lang="en"><head><title>Elastic</title><link href="/44040/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.css" rel="stylesheet"/>
I have not heard back from the support case about this new error - but it gives an idea that maybe there are some permissions missing in the ES cluster for filebeat to send the monitoring data in?
Again, it send the real data/events to ES just fine, they are indexed and I can see them, it's only the monitoring events that have this the issue. For the record, our main and monitoring ES clusters are one and the same.
One more interesting thing I noticed about your setup:
you specify connection URL to ES cluster[s] using port 443;
I specify port 9243:
For example something like this:
-e ES_HOSTS="https://mycluster.kb.us-east4.gcp.elastic-cloud.com:9243"
Does it matter?
Again, thank you for spending so much time on this issue!
Marina
Hi @ppine7
The cluster uuid is not the cloud id... 2 different things
per the docs
to get a cluster’s
cluster_uuid, call theGET /API against that production cluster.
In fact if you are sending the filebeat monitoring data to the same cloud instance as the as the filebeat data the only monitoring setting you need is the following everything else is inherited from the other settings / elasticsearch output settings
monitoring.enabled: true
I do this all the time works with just that.
That is the kibana endpoint not the elasticsearch endpoint that would explain a lot...
"https://mycluster.kb.us-east4.gcp.elastic-cloud.com:9243"
...................^^<--- should be .es.
Should be
"https://mycluster.es.us-east4.gcp.elastic-cloud.com:9243"
Also make sure you are using the correct format for the API keys there are different formats ... not base64 but the Beats Format
I just turned on Monitoring for the First Time on a new Cluster....
Just set the following...
output.elasticsearch:
hosts: ["https://abcdefg.es.us-west2.gcp.elastic-cloud.com:443"]
api_key: "pQasdsdfasdfgsScRN_T_T9:X70m3osadfBgHFO4-bkg"
`monitoring.enabled: true`
And it works right away
Glad to hear you're making progress on the support case @ppine7! And thanks @stephenb for the demonstration. 
It's probably best to stay focused on your support case since it's where we can share details about your deployment.
I'll stay subscribed here incase other questions arise.
Just a note that I did a local test today on es/kibana/filebeat main and the Stack Monitoring UI was able to show a standalone beat with this config:
filebeat.inputs:
- type: filestream
id: logs
paths:
- /logs/*.log
monitoring:
enabled: true
output.elasticsearch:
hosts: "http://localhost:9200"
api_key: "(beats key)"
As well as a similar standalone cluster on ESS 8.4.3
The monitoring data still doesn't go via the production cluster, but I think it's likely I'm missing some nuance and this is working as expected. I think once support has a look at your deployment, we should be able to sort out the gap for you @ppine7 .
Thank you Mat and Stephen!!!!
So, after reading all your suggestions - decided to bite the bullet and start from scratch!
so I did the following:
monitoring.enabled: true" setting# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
enabled: true
index: "ibc-parsed-logs"
parameters.pipeline: "geoip-info"
hosts: ${ES_HOSTS}
protocol: "https"
# Authentication credentials - either API key or username/password.
api_key: ${ES_API_KEY}
# The maximum number of events to bulk in a single Elasticsearch bulk API index request.
# The default is 50.
#bulk_max_size: 100
# ================================= Processors =================================
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- decode_json_fields:
fields: ["message"]
add_error_key: true
document_id: "event_uuid"
# ================================== Logging ===================================
logging.metrics.enabled: true
logging.enabled: true
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: debug
logging.to_files: true
logging.files:
path: /usr/share/filebeat/f_logs
name: filebeat
keepfiles: 10
permissions: 0640
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
logging.selectors: ["*"]
# ============================= X-Pack Monitoring ==============================
monitoring.enabled: true
Now, when I run filebeat, I do not see any errors anymore (!) and I can see in filebeat's logs that the monitoring events are being sent to the ES cluster:
{"log.level":"debug","@timestamp":"2022-10-18T14:55:21.509Z","log.logger":"monitoring","log.origin":{"file.name":"processing/processors.go","file.line":210},"message":"Publish event: {\n \"@timestamp\": \"2022-10-18T14:55:21.505Z\",\n \"@metadata\": {\n \"beat\": \"filebeat\",\n \"type\": \"_doc\",\n \"version\": \"8.4.3\",\n \"type\": \"beats_stats\",\n \"interval_ms\": 10000,\n \"params\": {\n \"interval\": \"10s\"\n }\n },\n \"metrics\": {\n \"registrar\": {\n \"states\": {\n \"current\": 0,\n \"update\": 0,\n \"cleanup\": 0\n },\n \"writes\": {\n \"total\": 0,\n \"fail\": 0,\n \"success\": 0\n }\n },\n \"filebeat\": {\n \"input\": {\n \"netflow\": {\n \"flows\": 0,\n \"packets\": {\n \"received\": 0,\n \"dropped\": 0\n }\n },\n \"log\": {\n \"files\": {\n \"renamed\": 0,\n \"truncated\": 0\n }\n }\n },\n \"events\": {\n \"added\": 0,\n \"done\": 0,\n \"active\": 0\n },\n \"harvester\": {\n \"closed\": 0,\n \"running\": 0,\n \"open_files\": 0,\n \"skipped\": 0,\n \"started\": 0\n }\n },\n \"libbeat\": {\n \"pipeline\": {\n \"queue\": {\n \"max_events\": 4096,\n \"acked\": 0\n },\n \"clients\": 1,\n \"events\": {\n \"retry\": 0,\n \"active\": 0,\n \"total\": 0,\n \"filtered\": 0,\n \"published\": 0,\n \"failed\": 0,\n \"dropped\": 0\n }\n },\n \"config\": {\n \"module\": {\n \"running\": 0,\n \"starts\": 0,\n \"stops\": 0\n },\n \"scans\": 0,\n \"reloads\": 0\n },\n \"output\": {\n \"read\": {\n \"bytes\": 0,\n \"errors\": 0\n },\n \"type\": \"elasticsearch\",\n \"events\": {\n \"failed\": 0,\n \"dropped\": 0,\n \"duplicates\": 0,\n \"active\": 0,\n \"toomany\": 0,\n \"batches\": 0,\n \"total\": 0,\n \"acked\": 0\n },\n \"write\": {\n \"bytes\": 0,\n \"errors\": 0\n }\n }\n },\n \"beat\": {\n \"cpu\": {\n \"total\": {\n \"time\": {\n \"ms\": 1820\n },\n \"value\": 1820,\n \"ticks\": 1820\n },\n \"user\": {\n \"ticks\": 700,\n \"time\": {\n \"ms\": 700\n }\n },\n \"system\": {\n \"ticks\": 1120,\n \"time\": {\n \"ms\": 1120\n }\n }\n },\n \"runtime\": {\n \"goroutines\": 76\n },\n \"info\": {\n \"version\": \"8.4.3\",\n \"uptime\": {\n \"ms\": 180354\n },\n \"ephemeral_id\": \"68d58d94-3678-4cda-9759-7348b4354bd9\",\n \"name\": \"filebeat\"\n },\n \"cgroup\": {\n \"cpuacct\": {\n \"id\": \"/\",\n \"total\": {\n \"ns\": 1875475443\n }\n },\n \"memory\": {\n \"id\": \"/\",\n \"mem\": {\n \"usage\": {\n \"bytes\": 52338688\n },\n \"limit\": {\n \"bytes\": 9223372036854771712\n }\n }\n },\n \"cpu\": {\n \"id\": \"/\",\n \"cfs\": {\n \"quota\": {\n \"us\": 0\n },\n \"period\": {\n \"us\": 100000\n }\n },\n \"stats\": {\n \"periods\": 0,\n \"throttled\": {\n \"periods\": 0,\n \"ns\": 0\n }\n }\n }\n },\n \"handles\": {\n \"open\": 19,\n \"limit\": {\n \"soft\": 1048576,\n \"hard\": 1048576\n }\n },\n \"memstats\": {\n \"memory_sys\": 34161672,\n \"gc_next\": 20490032,\n \"rss\": 138559488,\n \"memory_total\": 67309832,\n \"memory_alloc\": 13446248\n }\n },\n \"system\": {\n \"cpu\": {\n \"cores\": 8\n },\n \"load\": {\n \"1\": 0.09,\n \"5\": 0.1,\n \"15\": 0.04,\n \"norm\": {\n \"15\": 0.005,\n \"1\": 0.0113,\n \"5\": 0.0125\n }\n }\n }\n },\n \"beat\": {\n \"uuid\": \"1fc0f527-3173-46be-a029-ca70c420c1c7\",\n \"type\": \"filebeat\",\n \"version\": \"8.4.3\",\n \"name\": \"ccf8bd83efd8\",\n \"host\": \"ccf8bd83efd8\"\n }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-18T14:55:21.545Z","log.logger":"monitoring","log.origin":{"file.name":"memqueue/eventloop.go","file.line":197},"message":"handle ACKs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-18T14:55:21.545Z","log.logger":"monitoring","log.origin":{"file.name":"memqueue/eventloop.go","file.line":216},"message":"try ack index: (idx=0, i=0, seq=0)\n","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-18T14:55:21.546Z","log.logger":"monitoring","log.origin":{"file.name":"memqueue/eventloop.go","file.line":220},"message":"no state set","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-18T14:55:21.547Z","log.logger":"monitoring","log.origin":{"file.name":"memqueue/eventloop.go","file.line":199},"message":"handle ACK took: 2.2609ms","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-18T14:55:21.547Z","log.logger":"monitoring","log.origin":{"file.name":"memqueue/ackloop.go","file.line":95},"message":"ackloop: return ack to broker loop:1","service.name":"filebeat","ecs.version":"1.6.0"}
HOWEVER, when I go to Stack Monitoring in the ES cluster - I still see nothing - no .monitoring indices are create.... And in fact, I don't see anything at ll like what you showed....
This is what I get when go to Stack Monitoring:
I feel I am sooooo close and yet ....
Is this something in configuration or permissions missing, potentially?
Thank you !!
Ok @ppine7 yes I think you are close...
Now you need to do one more thing I think this IS perhaps a bug... you will not see the metricbeat monitoring without the rest of the cluster monitoring.
So go to the cloud console and turn on metrics monitoring for your cluster and ship the data to the same cluster.
You can skip the logs for now...
Then when you go back to monitoring you should see the cluster + the metricbeat monitoring.
Thanks, Stephen!
Ok, enabled Metrics per your advise. Now we can see *some" monitoring in the Stack Monitoring: of ES itself and Kibana:
but not of Beats 
Checking in Dev Tools - the only monitoring -related indices that appeared now are these:
GET /_cat/indices/.monitoring-*
green open .monitoring-es-7-2022.10.18 ZSLCzJRBRiS2r4qihIw71w 1 1 385 34 690.3kb 347.2kb
green open .ds-.monitoring-kibana-8-mb-2022.10.18-000001 Xvq_P_9NRiKy3hYxPwBwmQ 1 1 1010 0 2.3mb 1.5mb
green open .ds-.monitoring-es-8-mb-2022.10.18-000001 2H4hLmyGS8q2-EQfbuynQQ 1 1 8395 0 11.8mb 6.1mb
green open .monitoring-kibana-7-2022.10.18 4kofsQNNTzylE-YWoEMYXg 1 1 76 0 371.8kb 165.7kb
and I can still see in the Filebeat logs that it is trying to send the metrics data:
{"log.level":"debug","@timestamp":"2022-10-18T18:58:33.891Z","log.logger":"monitoring","log.origin":{"file.name":"processing/processors.go","file.line":210},"message":"Publish event: {\n \"@timestamp\": \"2022-10-18T18:58:33.886Z\",\n \"@metadata\": {\n \"beat\": \"filebeat\",\n \"type\": \"_doc\",\n \"version\": \"8.4.3\",\n \"type\": \"beats_stats\",\n \"interval_ms\": 10000,\n \"params\": {\n \"interval\": \"10s\"\n },\n \"cluster_uuid\": \"9PxnN-9PToumnPWp0L4grQ\"\n },\n \"beat\": {\n \"uuid\": \"031e0829-3ab6-40a4-af63-8a6209c0d53d\",\n \"type\": \"filebeat\",\n \"version\": \"8.4.3\",\n \"name\": \"572d93de94ee\",\n \"host\": \"572d93de94ee\"\n },\n \"metrics\": {\n \"libbeat\": {\n \"pipeline\": {\n \"events\": {\n \"dropped\": 0,\n \"retry\": 1,\n \"active\": 0,\n \"total\": 1,\n \"filtered\": 0,\n \"published\": 1,\n \"failed\": 0\n },\n \"queue\": {\n \"max_events\": 4096,\n \"acked\": 1\n },\n \"clients\": 1\n },\n \"config\": {\n \"reloads\": 0,\n \"module\": {\n \"stops\": 0,\n \"running\": 0,\n \"starts\": 0\n },\n \"scans\": 0\n },\n \"output\": {\n \"events\": {\n \"dropped\": 1,\n \"duplicates\": 0,\n \"active\": 0,\n \"toomany\": 0,\n \"batches\": 1,\n \"total\": 1,\n \"acked\": 0,\n \"failed\": 0\n },\n \"write\": {\n \"errors\": 0,\n \"bytes\": 3611\n },\n \"read\": {\n \"bytes\": 3458,\n \"errors\": 1\n },\n \"type\": \"elasticsearch\"\n }\n },\n \"beat\": {\n \"cpu\": {\n \"total\": {\n \"time\": {\n \"ms\": 2860\n },\n \"value\": 2860,\n \"ticks\": 2860\n },\n \"user\": {\n \"time\": {\n \"ms\": 1220\n },\n \"ticks\": 1220\n },\n \"system\": {\n \"ticks\": 1640,\n \"time\": {\n \"ms\": 1640\n }\n }\n },\n \"runtime\": {\n \"goroutines\": 54\n },\n \"info\": {\n \"ephemeral_id\": \"43298808-e0b4-490c-b516-b56328f33b75\",\n \"name\": \"filebeat\",\n \"version\": \"8.4.3\",\n \"uptime\": {\n \"ms\": 250626\n }\n },\n \"cgroup\": {\n \"cpu\": {\n \"cfs\": {\n \"period\": {\n \"us\": 100000\n },\n \"quota\": {\n \"us\": 0\n }\n },\n \"stats\": {\n \"periods\": 0,\n \"throttled\": {\n \"periods\": 0,\n \"ns\": 0\n }\n },\n \"id\": \"/\"\n },\n \"cpuacct\": {\n \"id\": \"/\",\n \"total\": {\n \"ns\": 2961870248\n }\n },\n \"memory\": {\n \"id\": \"/\",\n \"mem\": {\n \"limit\": {\n \"bytes\": 9223372036854771712\n },\n \"usage\": {\n \"bytes\": 61280256\n }\n }\n }\n },\n \"handles\": {\n \"open\": 12,\n \"limit\": {\n \"hard\": 1048576,\n \"soft\": 1048576\n }\n },\n \"memstats\": {\n \"memory_total\": 71525392,\n \"memory_alloc\": 13243936,\n \"memory_sys\": 34685960,\n \"gc_next\": 21063872,\n \"rss\": 140554240\n }\n },\n \"system\": {\n \"cpu\": {\n \"cores\": 8\n },\n \"load\": {\n \"1\": 0.23,\n \"5\": 0.22,\n \"15\": 0.16,\n \"norm\": {\n \"1\": 0.0288,\n \"5\": 0.0275,\n \"15\": 0.02\n }\n }\n },\n \"registrar\": {\n \"states\": {\n \"update\": 0,\n \"cleanup\": 0,\n \"current\": 0\n },\n \"writes\": {\n \"total\": 0,\n \"fail\": 0,\n \"success\": 0\n }\n },\n \"filebeat\": {\n \"harvester\": {\n \"started\": 0,\n \"closed\": 0,\n \"running\": 0,\n \"open_files\": 0,\n \"skipped\": 0\n },\n \"input\": {\n \"log\": {\n \"files\": {\n \"renamed\": 0,\n \"truncated\": 0\n }\n },\n \"netflow\": {\n \"flows\": 0,\n \"packets\": {\n \"received\": 0,\n \"dropped\": 0\n }\n }\n },\n \"events\": {\n \"added\": 1,\n \"done\": 1,\n \"active\": 0\n }\n }\n }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
Something else possibly missing?
that should be
pipeline: "geoip-info"
That parameters is putting info into the headers which I do not think you want to do, I don't think that is the problem but it not correct..but it could be a problem...
After you start filebeat with the monitoring enabled....
You should see interesting that it says -7- this is filebeat / metricbeat 8.4.1
GET _cat/indices/.monitoring-beats-*?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .monitoring-beats-7-2022.10.18 RQ3yi65GR66LIDFADeWPWw 1 1 9 0 313.7kb 156.8kb
note there may be one but that is the internal monitoring of the APM / Fleet agent
.ds-.monitoring-beats-8-mb-2022.10.18-000001
And you should see something like this in your filebeat logs
this the Connection to Monitoring!
{"log.level":"info","@timestamp":"2022-10-18T14:09:25.099-0700","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(monitoring(https://**************.es.us-west2.gcp.elastic-cloud.com:443)) established","service.name":"filebeat","ecs.version":"1.6.0"}
You might commenting out all the logging stuff just to test, I do not have any of it.
This is my entire filebeat not just snippets it is the entire working filebeat
filebeat.inputs:
- type: log
enabled: true
paths:
- /Users/sbrown/workspace/sample-data/discuss/container-mixed/*.log
output.elasticsearch:
hosts: ["https://*********.es.us-west2.gcp.elastic-cloud.com:443"]
api_key: "asdfasdfasdfoHNQkCvBgHFO4-bkg"
monitoring.enabled: true
@ppine7 I'd check if the cluster_uuid shown on the documents in .monitoring-beats-* matches the cluster_uuid shown in the stack monitoring UI (it'll be in the browser bar, or GET / on the monitored cluster).
Like I mentioned above
- it might also be inspecting the UUID incorrectly, or caching it somehow (in my tests, I had the monitoring deployment ES UUID on some docs that landed in the production deployment)
At one point in my testing, filebeat seemed to be getting sending docs with the wrong cluster UUID. I didn't figure out why.
The UI expects either a cluster UUID that matches an ES cluster that's also being monitored or a blank cluster UUID (these will show up as a "Standalone Cluster" entry in the UI).
If the docs have a non-monitored cluster UUID, they won't show up in the Stack Monitoring UI.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.
