Hi, this question comes as a result of solving one mystery in this post: Filebeat monitoring metrics not visible in ElasticSearch - #30 by stephenb - huge Thank You to @stephenb for his help! - where Filebeat monitoring metrics were not being indexed into Elasticsearch IF a GEOIP processing pipeline was enabled in the filebeat.yml config.
So this post's goal is now solve the follow up mystery - WHY having a GEOIP pipeline enabled is causing all monitoring events from the Filebeat to be dropped/not indexed into ES?
Here is the geoip pipeline I have created in ES:
PUT _ingest/pipeline/geoip-info
{
"description": "Add geoip info",
"processors": [
{
"geoip": {
"field": "message.remote_ip",
"target_field": "message.remote_ip_geo",
"ignore_missing": true
}
}
]
}
This is the filebeat config I'm using to process events from GCP PubSub, enrich with GEO info using the geoip pipeline, push into ES (this part of event processing works just fine!) - and at the same time enabling Filebeat monitoring metrics to also be sent into the same ES cluster
###################### Filebeat Configuration Example #########################
queue.mem:
events: 4096
flush.min_events: 2048
flush.timeout: 1s
# ============================== Filebeat inputs ===============================
filebeat.inputs:
- type: gcp-pubsub
enabled: true
project_id: ${PROJECT_ID}
topic: ${PUBSUB_INPUT_TOPIC}
subscription.name: ${SUBSCRIPTION_NAME}
fields_under_root: true
# ======================= Elasticsearch template setting =======================
setup.template.name: "ibc-parsed-logs"
setup.template.pattern: "ibc-parsed-logs-*"
setup.template.json.enabled: true
setup.template.json.path: "ibc_es_template.json"
setup.template.json.name: "ibc-parsed-logs-template"
setup.template.enabled: true
setup.ilm.enabled: false
# =============================== Elastic Cloud ================================
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
# ENG IBC ES
#cloud.id: '${CLOUD_ID}'
# ================================== Outputs ===================================
output.console:
enabled: false
pretty: true
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
enabled: true
index: "ibc-parsed-logs"
#parameters.pipeline: "geoip-info"
hosts: ${ES_HOSTS}
protocol: "https"
api_key: ${ES_API_KEY}
# ============================= X-Pack Monitoring ==============================
monitoring.enabled: true
monitoring.cluster_uuid: "9PxnN-9Pxxx"
# ================================= Processors =================================
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- decode_json_fields:
fields: ["message"]
add_error_key: true
document_id: "event_uuid"
# ================================== Logging ===================================
logging.metrics.enabled: true
logging.enabled: true
logging.level: debug
logging.to_files: true
logging.files:
path: /usr/share/filebeat/f_logs
name: filebeat
keepfiles: 10
permissions: 0640
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
logging.selectors: ["*"]
Now, IF the geoip pipeline line is commented out in the filebeat.yml:
#parameters.pipeline: "geoip-info"
then In the filebeat logs I can see the monitoring events are indeed sent to ES:
{"log.level":"debug","@timestamp":"2022-10-19T18:18:22.649Z","log.logger":"monitoring","log.origin":{"file.name":"elasticsearch/client.go","file.line":99},"message":"XPack monitoring is enabled","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-19T18:18:22.649Z","log.logger":"monitoring","log.origin":{"file.name":"elasticsearch/elasticsearch.go","file.line":234},"message":"Successfully connected to X-Pack Monitoring endpoint.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-19T18:18:22.650Z","log.logger":"monitoring","log.origin":{"file.name":"elasticsearch/elasticsearch.go","file.line":240},"message":"Finish monitoring endpoint init loop.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-19T18:18:22.650Z","log.logger":"monitoring","log.origin":{"file.name":"elasticsearch/elasticsearch.go","file.line":248},"message":"Start monitoring state metrics snapshot loop with period 1m0s.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-19T18:18:22.650Z","log.logger":"monitoring","log.origin":{"file.name":"elasticsearch/elasticsearch.go","file.line":248},"message":"Start monitoring stats metrics snapshot loop with period 10s.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-19T18:18:32.433Z","log.logger":"input","log.origin":{"file.name":"input/input.go","file.line":137},"message":"Run input","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-19T18:18:32.625Z","log.logger":"monitoring","log.origin":{"file.name":"processing/processors.go","file.line":210},"message":"Publish event: {\n \"@timestamp\": \"2022-10-19T18:18:32.618Z\",\n \"@metadata\": {\n \"beat\": \"filebeat\",\n \"type\": \"_doc\",\n \"version\": \"8.4.3\",\n \"interval_ms\": 10000,\n \"params\": {\n \"interval\": \"10s\"\n },\n \"cluster_uuid\": \"9PxnN-9Pxxx\",\n \"type\": \"beats_stats\"\n },\n \"beat\": {\n \"version\": \"8.4.3\",\n \"name\": \"54a5f02bc902\",\n \"host\": \"54a5f02bc902\",\n \"uuid\": \"c8eccb7e-f287-4e23-8f58-4af91ccb1a8d\",\n \"type\": \"filebeat\"\n },\n \"metrics\": {\n \"beat\": {\n \"cpu\": {\n \"total\": {\n \"value\": 430,\n \"ticks\": 430,\n \"time\": {\n \"ms\": 430\n }\n },\n \"user\": {\n \"time\": {\n \"ms\": 250\n },\n \"ticks\": 250\n },\n \"system\": {\n \"ticks\": 180,\n \"time\": {\n \"ms\": 180\n }\n }\n },\n \"runtime\": {\n \"goroutines\": 76\n },\n \"info\": {\n \"uptime\": {\n \"ms\": 10343\n },\n \"ephemeral_id\": \"74839794-ee85-4d0a-8e4a-580ccb19df7a\",\n \"name\": \"filebeat\",\n \"version\": \"8.4.3\"\n },\n \"cgroup\": {\n \"cpuacct\": {\n \"total\": {\n \"ns\": 495299162\n },\n \"id\": \"/\"\n },\n \"memory\": {\n \"mem\": {\n \"limit\": {\n \"bytes\": 9223372036854771712\n },\n \"usage\": {\n \"bytes\": 51306496\n }\n },\n \"id\": \"/\"\n },\n \"cpu\": {\n \"id\": \"/\",\n \"cfs\": {\n \"period\": {\n \"us\": 100000\n },\n \"quota\": {\n \"us\": 0\n }\n },\n \"stats\": {\n \"periods\": 0,\n \"throttled\": {\n \"periods\": 0,\n \"ns\": 0\n }\n }\n }\n },\n \"handles\": {\n \"limit\": {\n \"hard\": 1048576,\n \"soft\": 1048576\n },\n \"open\": 20\n },\n \"memstats\": {\n \"gc_next\": 18153696,\n \"rss\": 137138176,\n \"memory_total\": 58737312,\n \"memory_alloc\": 13683544,\n \"memory_sys\": 34423816\n }\n },\n \"system\": {\n \"cpu\": {\n \"cores\": 8\n },\n \"load\": {\n \"1\": 0.02,\n \"5\": 0.03,\n \"15\": 0,\n \"norm\": {\n \"5\": 0.0038,\n \"15\": 0,\n \"1\": 0.0025\n }\n }\n },\n \"registrar\": {\n \"states\": {\n \"current\": 0,\n \"update\": 0,\n \"cleanup\": 0\n },\n \"writes\": {\n \"success\": 0,\n \"total\": 0,\n \"fail\": 0\n }\n },\n \"filebeat\": {\n \"events\": {\n \"active\": 0,\n \"added\": 0,\n \"done\": 0\n },\n \"harvester\": {\n \"running\": 0,\n \"open_files\": 0,\n \"skipped\": 0,\n \"started\": 0,\n \"closed\": 0\n },\n \"input\": {\n \"netflow\": {\n \"packets\": {\n \"received\": 0,\n \"dropped\": 0\n },\n \"flows\": 0\n },\n \"log\": {\n \"files\": {\n \"renamed\": 0,\n \"truncated\": 0\n }\n }\n }\n },\n \"libbeat\": {\n \"config\": {\n \"scans\": 0,\n \"reloads\": 0,\n \"module\": {\n \"starts\": 0,\n \"stops\": 0,\n \"running\": 0\n }\n },\n \"output\": {\n \"events\": {\n \"batches\": 0,\n \"total\": 0,\n \"acked\": 0,\n \"failed\": 0,\n \"dropped\": 0,\n \"duplicates\": 0,\n \"active\": 0,\n \"toomany\": 0\n },\n \"write\": {\n \"errors\": 0,\n \"bytes\": 0\n },\n \"read\": {\n \"bytes\": 0,\n \"errors\": 0\n },\n \"type\": \"elasticsearch\"\n },\n \"pipeline\": {\n \"queue\": {\n \"acked\": 0,\n \"max_events\": 4096\n },\n \"clients\": 1,\n \"events\": {\n \"filtered\": 0,\n \"published\": 0,\n \"failed\": 0,\n \"dropped\": 0,\n \"retry\": 0,\n \"active\": 0,\n \"total\": 0\n }\n }\n }\n }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
and I can see the corresponding .monitoring-beat-xxx index in ES with the monitoring events:
GET /_cat/indices/*monitoring*
results:
green open .monitoring-es-7-2022.10.18 ZSLCzJRBRiS2r4qihIw71w 1 1 385 34 690.3kb 347.2kb
green open .ds-.monitoring-kibana-8-mb-2022.10.18-000001 Xvq_P_9NRiKy3hYxPwBwmQ 1 1 43220 0 22.7mb 11.3mb
green open .monitoring-beats-7-2022.10.19 3HuDjP9NTFeBRjI5ElGkVQ 1 1 84 0 798.9kb 386.7kb
green open .ds-.monitoring-es-8-mb-2022.10.18-000001 2H4hLmyGS8q2-EQfbuynQQ 1 1 318236 0 382.2mb 190.4mb
green open .monitoring-kibana-7-2022.10.18 4kofsQNNTzylE-YWoEMYXg 1 1 76 0 371.8kb 165.7kb
GET .monitoring-beats-7-2022.10.19/_search
result:
"hits": {
"total": {
"value": 88,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": ".monitoring-beats-7-2022.10.19",
"_id": "Fkd48YMBgEVm2LhbFsxT",
"_score": 1,
"_source": {
"timestamp": "2022-10-19T18:18:42.617Z",
"interval_ms": 10000,
"cluster_uuid": "9PxnN-9Pxxx",
"type": "beats_stats",
"beats_stats": {
"beat": {
"uuid": "c8eccb7e-f287-4e23-8f58-4af91ccb1a8d",
"type": "filebeat",
"version": "8.4.3",
"name": "54a5f02bc902",
"host": "54a5f02bc902"
},
"metrics": {
"beat": {
"cgroup": {
"cpu": {
"id": "/",
"cfs": {
"period": {
"us": 100000
BUT, if I enable the geoip pipeline (uncomment) - I see the same monitoring events being shown as sent to the ES in the filebeat logs - but no actual events reach ES and no .monitoring-beat-xxx index is created...
Any idea why? 
Thank you!!!
- and I am almost there , except that my local Filebeat does not want to talk to the local ES8 - and I posted a different question on that: