Hi, Mat, Stephen - thank you for the further investigation!
So, to answer the questions and suggestions in order:
-
@stephenb regrading the pipeline... This was one of the pain points to get working, took me awhile, as there seems to be a bug in ES/Filebeat, where specifying the "pipeline: xxx" per documentation does NOT work, and the only way to get it working is to use "parameters.pipeline: xxx" instead. Here is the bug report that helped me find this workaround: [Filebeat] 7.8 Filebeat output elasticsearch pipeline broken · Issue #20342 · elastic/beats · GitHub
-
logs that @stephenb mentioned: yes, I do see the lines similar to what you are referring to:
{"log.level":"info","@timestamp":"2022-10-19T13:51:27.671Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(monitoring(https://1e432b5236c***.us-east4.gcp.elastic-cloud.com:443)) established","service.name":"filebeat","ecs.version":"1.6.0"}
and here is a bigger chunk of logs that shows connections, pings and sending of the monitoring events to this URL:
{"log.level":"debug","@timestamp":"2022-10-19T13:51:17.577Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":346},"message":"GET https://1e432b5236c***.us-east4.gcp.elastic-cloud.com:443/_xpack?filter_path=features.monitoring.enabled <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-19T13:51:17.600Z","log.logger":"monitoring","log.origin":{"file.name":"elasticsearch/client.go","file.line":99},"message":"XPack monitoring is enabled","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-19T13:51:17.600Z","log.logger":"monitoring","log.origin":{"file.name":"elasticsearch/elasticsearch.go","file.line":234},"message":"Successfully connected to X-Pack Monitoring endpoint.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-19T13:51:17.601Z","log.logger":"monitoring","log.origin":{"file.name":"elasticsearch/elasticsearch.go","file.line":240},"message":"Finish monitoring endpoint init loop.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-19T13:51:17.601Z","log.logger":"monitoring","log.origin":{"file.name":"elasticsearch/elasticsearch.go","file.line":248},"message":"Start monitoring stats metrics snapshot loop with period 10s.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-19T13:51:17.601Z","log.logger":"monitoring","log.origin":{"file.name":"elasticsearch/elasticsearch.go","file.line":248},"message":"Start monitoring state metrics snapshot loop with period 1m0s.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-19T13:51:27.348Z","log.logger":"input","log.origin":{"file.name":"input/input.go","file.line":137},"message":"Run input","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-19T13:51:27.620Z","log.logger":"monitoring","log.origin":{"file.name":"processing/processors.go","file.line":210},"message":"Publish event: {\n \"@timestamp\": \"2022-10-19T13:51:27.602Z\",\n \"@metadata\": {\n \"beat\": \"filebeat\",\n \"type\": \"_doc\",\n \"version\": \"8.4.3\",\n \"type\": \"beats_stats\",\n \"interval_ms\": 10000,\n \"params\": {\n \"interval\": \"10s\"\n }\n },\n \"beat\": {\n \"type\": \"filebeat\",\n \"version\": \"8.4.3\",\n \"name\": \"c0383dd47a87\",\n \"host\": \"c0383dd47a87\",\n \"uuid\": \"ed5a0b2c-3ced-43da-a7ce-282f2383eca8\"\n },\n \"metrics\": {\n \"filebeat\": {\n \"events\": {\n \"added\": 0,\n \"done\": 0,\n \"active\": 0\n },\n \"harvester\": {\n \"closed\": 0,\n \"running\": 0,\n \"open_files\": 0,\n \"skipped\": 0,\n \"started\": 0\n },\n \"input\": {\n \"log\": {\n \"files\": {\n \"renamed\": 0,\n \"truncated\": 0\n }\n },\n \"netflow\": {\n \"packets\": {\n \"received\": 0,\n \"dropped\": 0\n },\n \"flows\": 0\n }\n }\n },\n \"libbeat\": {\n \"output\": {\n \"events\": {\n \"duplicates\": 0,\n \"active\": 0,\n \"toomany\": 0,\n \"batches\": 0,\n \"total\": 0,\n \"acked\": 0,\n \"failed\": 0,\n \"dropped\": 0\n },\n \"write\": {\n \"bytes\": 0,\n \"errors\": 0\n },\n \"read\": {\n \"errors\": 0,\n \"bytes\": 0\n },\n \"type\": \"elasticsearch\"\n },\n \"pipeline\": {\n \"clients\": 1,\n \"events\": {\n \"total\": 0,\n \"filtered\": 0,\n \"published\": 0,\n \"failed\": 0,\n \"dropped\": 0,\n \"retry\": 0,\n \"active\": 0\n },\n \"queue\": {\n \"max_events\": 4096,\n \"acked\": 0\n }\n },\n \"config\": {\n \"scans\": 0,\n \"reloads\": 0,\n \"module\": {\n \"running\": 0,\n \"starts\": 0,\n \"stops\": 0\n }\n }\n },\n \"beat\": {\n \"info\": {\n \"version\": \"8.4.3\",\n \"uptime\": {\n \"ms\": 10735\n },\n \"ephemeral_id\": \"45393707-8a6c-4e36-a7dc-78134b21cbdd\",\n \"name\": \"filebeat\"\n },\n \"cgroup\": {\n \"cpuacct\": {\n \"id\": \"/\",\n \"total\": {\n \"ns\": 2488286354\n }\n },\n \"memory\": {\n \"id\": \"/\",\n \"mem\": {\n \"limit\": {\n \"bytes\": 9223372036854771712\n },\n \"usage\": {\n \"bytes\": 56520704\n }\n }\n },\n \"cpu\": {\n \"cfs\": {\n \"period\": {\n \"us\": 100000\n },\n \"quota\": {\n \"us\": 0\n }\n },\n \"stats\": {\n \"periods\": 0,\n \"throttled\": {\n \"periods\": 0,\n \"ns\": 0\n }\n },\n \"id\": \"/\"\n }\n },\n \"handles\": {\n \"limit\": {\n \"hard\": 1048576,\n \"soft\": 1048576\n },\n \"open\": 20\n },\n \"memstats\": {\n \"memory_sys\": 34423816,\n \"gc_next\": 20466608,\n \"rss\": 138305536,\n \"memory_total\": 59005528,\n \"memory_alloc\": 10707568\n },\n \"cpu\": {\n \"user\": {\n \"ticks\": 1420,\n \"time\": {\n \"ms\": 1420\n }\n },\n \"system\": {\n \"ticks\": 940,\n \"time\": {\n \"ms\": 940\n }\n },\n \"total\": {\n \"value\": 2360,\n \"ticks\": 2360,\n \"time\": {\n \"ms\": 2360\n }\n }\n },\n \"runtime\": {\n \"goroutines\": 76\n }\n },\n \"system\": {\n \"cpu\": {\n \"cores\": 8\n },\n \"load\": {\n \"5\": 0.02,\n \"15\": 0,\n \"norm\": {\n \"1\": 0.0088,\n \"5\": 0.0025,\n \"15\": 0\n },\n \"1\": 0.07\n }\n },\n \"registrar\": {\n \"states\": {\n \"current\": 0,\n \"update\": 0,\n \"cleanup\": 0\n },\n \"writes\": {\n \"total\": 0,\n \"fail\": 0,\n \"success\": 0\n }\n }\n }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-19T13:51:27.623Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(monitoring(https://1e432b5236c***.us-east4.gcp.elastic-cloud.com:443))","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-19T13:51:27.624Z","log.logger":"monitoring","log.origin":{"file.name":"elasticsearch/client.go","file.line":64},"message":"Monitoring client: connect.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-19T13:51:27.626Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":267},"message":"ES Ping(url=https://1e432b5236c***.us-east4.gcp.elastic-cloud.com:443)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-19T13:51:27.648Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":290},"message":"Ping status code: 200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-19T13:51:27.649Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.4.3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-19T13:51:27.650Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":346},"message":"GET https://1e432b5236c***.us-east4.gcp.elastic-cloud.com:443/_xpack?filter_path=features.monitoring.enabled <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-19T13:51:27.670Z","log.logger":"monitoring","log.origin":{"file.name":"elasticsearch/client.go","file.line":99},"message":"XPack monitoring is enabled","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-19T13:51:27.671Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(monitoring(https://1e432b5236c***.us-east4.gcp.elastic-cloud.com:443)) established","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-19T13:51:27.698Z","log.logger":"monitoring","log.origin":{"file.name":"memqueue/eventloop.go","file.line":197},"message":"handle ACKs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-19T13:51:27.700Z","log.logger":"monitoring","log.origin":{"file.name":"memqueue/eventloop.go","file.line":216},"message":"try ack index: (idx=0, i=0, seq=0)\n","service.name":"filebeat","ecs.version":"1.6.0"}
{"
Interesting that this URL: https://1e432b5236c***.us-east4.gcp.elastic-cloud.com:443 is NOT what I am specifying as "hosts" in the filebeat.yml.... In the filebeat.yml I am specifying a real name of the cluster, something like "my-new-es-cluster" - like "https://my-new-es8-cluster.es.us-east4.gcp.elastic-cloud.com:9243".... So it is getting translated into the UUID-type form somewhere ...
- indices: unfortunately, I do not have any indices that have "beat" and "monitoring" at the same time in the name, not what @stephenb showed in his cluster... I do have one 'metricbeat' index - but it is empty.
A few searches I did:
GET /_cat/indices/*monitoring-*
results:
green open .ds-.monitoring-kibana-8-mb-2022.10.18-000001 Xvq_P_9NRiKy3hYxPwBwmQ 1 1 36050 0 18.7mb 9.3mb
green open .monitoring-kibana-7-2022.10.18 4kofsQNNTzylE-YWoEMYXg 1 1 76 0 371.8kb 165.7kb
green open .ds-.monitoring-es-8-mb-2022.10.18-000001 2H4hLmyGS8q2-EQfbuynQQ 1 1 265395 0 315.2mb 158mb
green open .monitoring-es-7-2022.10.18 ZSLCzJRBRiS2r4qihIw71w 1 1 385 34 690.3kb 347.2kb
GET /_cat/indices/*beat*
results:
green open .ds-metricbeat-8.4.3-2022.10.18-000001 wq0Wt0kaSIae9kXP4b2Iug 1 1 0 0 450b 225b
GET .ds-metricbeat-8.4.3-2022.10.18-000001/_search
results:
{
"took": 0,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 0,
"relation": "eq"
},
"max_score": null,
"hits": []
}
}
- now comparing to the actual UUID of the one and only ES cluster I have (data+monitoring) , per @matschaffer suggestion:
when I do
GET /_cluster/state
I get this:
{
"cluster_name": "1e432b5236c***",
"cluster_uuid": "9PxnN-9PT***",
"version": 2013,
so it is the cluster_name that Filebeat is using, not the UUID...per its logs:
"message":"Connection to backoff(monitoring(https://1e432b5236c***.us-east4.gcp.elastic-cloud.com:443)) established"
is this a problem?
Thank you!!!
Marina
.... monitoring and GEO-enriched events!