Monitoring using metricbeat

parthmaniar · June 11, 2020, 2:51pm

I turned on self-monitoring that I want to roll-back and migrate to metricbeat based monitoring for the cluster. I clicked on set self-monitoring which led to the following:

I searched elasticsearch.yml but could not locate this settings. Where can I find this setting to revert it.

I have carried out entire guide to start monitoring using metricbeat & following are the configuration:

A single system is hosting metricbeat and elasticsearch. I have run the following command successfully;

metricbeat modules enable elasticsearch-xpack

File: elasticsearch.yml

# Module: elasticsearch
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.6/metricbeat-module-elasticsearch.html

- module: elasticsearch
  metricsets:
    - node
    - node_stats
  period: 10s
  #hosts: ["http://localhost:9200"]
  hosts: ["https://IP of the node:9200"]
  protocol: "https"
  username: "ID"
  password: "password"
  ssl.certificate_authorities: /etc/metricbeat/elasticsearch-ca.pem
  ssl.verification_mode: none
  #username: "user"
  #password: "secret"

File: elasticsearch-xpack.yml

 Module: elasticsearch
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.7/metricbeat-module-elasticsearch.html

- module: elasticsearch
  metricsets:
    - ccr
    - cluster_stats
    - enrich
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["https://IP of the node:9200"]
  protocol: "https"
  username: "id"
  password: "password"
  ssl.certificate_authorities: /etc/metricbeat/elasticsearch-ca.pem
  ssl.verification_mode: none
  xpack.enabled: true

I have done the following to set up metricbeat based monitoring however the migration wizad is unable to find the logs. I am getting following prompt only:

Finally, I am student running this setup on a VM on a NAS. My processor is constantly at ~95%. Is there a computation offset to send these logs first to a Raspberry Pi running logstash pipeline and later to elasticsearch in batches? Maybe help me get some compute back on the VM?

RPi running logstash:

Thank you.

chrisronline · June 11, 2020, 6:44pm

Hi @parthmaniar,

Happy to help get you sorted here.

Let's take a look at the monitoring data and see what's going on.

Can you return the results of GET _cat/indices/.monitoring* on the monitoring cluster?

parthmaniar · June 11, 2020, 6:58pm

Hi @chrisronline thank you very much for assisting.

Before I answer, its important to know that I am facing stability issues so I issued the following command few minutes back (in case it matters)

PUT https://IP of ES:9200/_cluster/settings
{
    "persistent": {
        "xpack": {
            "monitoring": {
                "collection": {
                    "enabled": null
            }
        }
    },
    "transient": {}
}
}

Here is the output you asked for:

green open .monitoring-es-7-2020.06.11          C3mUbMHvQreLeJt_qf6OQg 1 0 606092 125791 225.3mb 225.3mb
green open .monitoring-kibana-7-2020.06.11      env7GHFmRHWvaixxSJFGHw 1 0   2632      0 692.4kb 692.4kb
green open .monitoring-logstash-7-mb-2020.06.11 TDPe2LU0SIiOl0wJOHZGgg 1 0 413259      0  13.6mb  13.6mb
green open .monitoring-es-7-mb-2020.06.11       2ja6LnUZSMytBSMy7_2PIQ 1 0 131247      0   121mb   121mb

chrisronline · June 29, 2020, 1:21pm

Can you run this against your monitoring cluster and return with the results?

POST .monitoring-es-*/_search
{
  "size": 0,
  "aggs": {
    "clusters": {
      "terms": {
        "field": "cluster_uuid",
        "size": 20
      },
      "aggs": {
        "index": {
          "terms": {
            "field": "_index",
            "size": 1
          }
        },
        "types": {
          "terms": {
            "field": "type",
            "size": 10
          },
          "aggs": {
            "last_seen": {
              "max": {
                "field": "timestamp"
              }
            }
          }
        }
      }
    }
  }
}

parthmaniar · June 29, 2020, 6:59pm

chrisronline:

POST .monitoring-es-*/_search
{
  "size": 0,
  "aggs": {
    "clusters": {
      "terms": {
        "field": "cluster_uuid",
        "size": 20
      },
      "aggs": {
        "index": {
          "terms": {
            "field": "_index",
            "size": 1
          }
        },
        "types": {
          "terms": {
            "field": "type",
            "size": 10
          },
          "aggs": {
            "last_seen": {
              "max": {
                "field": "timestamp"
              }
            }
          }
        }
      }
    }
  }
}

Sure, thank you very much for replying. Here is the output.

{
  "took" : 6700,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "clusters" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "-26z29GGRyWZ-8MuAvjaWw",
          "doc_count" : 3337955,
          "types" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "index_stats",
                "doc_count" : 2507185,
                "last_seen" : {
                  "value" : 1.593456846013E12,
                  "value_as_string" : "2020-06-29T18:54:06.013Z"
                }
              },
              {
                "key" : "shards",
                "doc_count" : 657245,
                "last_seen" : {
                  "value" : 1.593456562847E12,
                  "value_as_string" : "2020-06-29T18:49:22.847Z"
                }
              },
              {
                "key" : "enrich_coordinator_stats",
                "doc_count" : 34802,
                "last_seen" : {
                  "value" : 1.593456842875E12,
                  "value_as_string" : "2020-06-29T18:54:02.875Z"
                }
              },
              {
                "key" : "index_recovery",
                "doc_count" : 34766,
                "last_seen" : {
                  "value" : 1.593456843891E12,
                  "value_as_string" : "2020-06-29T18:54:03.891Z"
                }
              },
              {
                "key" : "indices_stats",
                "doc_count" : 34759,
                "last_seen" : {
                  "value" : 1.593456844176E12,
                  "value_as_string" : "2020-06-29T18:54:04.176Z"
                }
              },
              {
                "key" : "node_stats",
                "doc_count" : 34738,
                "last_seen" : {
                  "value" : 1.593456843837E12,
                  "value_as_string" : "2020-06-29T18:54:03.837Z"
                }
              },
              {
                "key" : "cluster_stats",
                "doc_count" : 34460,
                "last_seen" : {
                  "value" : 1.593456843099E12,
                  "value_as_string" : "2020-06-29T18:54:03.099Z"
                }
              }
            ]
          },
          "index" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 2338479,
            "buckets" : [
              {
                "key" : ".monitoring-es-7-mb-2020.06.28",
                "doc_count" : 999476
              }
            ]
          }
        }
      ]
    }
  }
}

chrisronline · July 1, 2020, 8:01pm

Thanks for that. I'm going to slightly change the query and if you can return the results again, that'd be great.

POST .monitoring-es-*/_search
{
  "size": 0,
  "aggs": {
    "clusters": {
      "terms": {
        "field": "cluster_uuid",
        "size": 20
      },
      "aggs": {
        "index": {
          "terms": {
            "field": "_index",
            "size": 100,
          }
        },
        "types": {
          "terms": {
            "field": "type",
            "size": 10
          },
          "aggs": {
            "last_seen": {
              "max": {
                "field": "timestamp"
              }
            }
          }
        }
      }
    }
  }
}

parthmaniar · July 1, 2020, 8:39pm

Absolutely no problem. I couldn't not the output for your new query as it is showing a construct error:

chrisronline · July 6, 2020, 2:03pm

Yes, sorry. I had a typo there. Please remove the trailing , so it is:

POST .monitoring-es-*/_search
{
  "size": 0,
  "aggs": {
    "clusters": {
      "terms": {
        "field": "cluster_uuid",
        "size": 20
      },
      "aggs": {
        "index": {
          "terms": {
            "field": "_index",
            "size": 100
          }
        },
        "types": {
          "terms": {
            "field": "type",
            "size": 10
          },
          "aggs": {
            "last_seen": {
              "max": {
                "field": "timestamp"
              }
            }
          }
        }
      }
    }
  }

parthmaniar · July 6, 2020, 3:18pm

Thank you very much. Here is the output:

{
  "took" : 4888,
  "timed_out" : false,
  "_shards" : {
    "total" : 6,
    "successful" : 6,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "clusters" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "-26z29GGRyWZ-8MuAvjaWw",
          "doc_count" : 4662968,
          "types" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "index_stats",
                "doc_count" : 3464211,
                "last_seen" : {
                  "value" : 1.593979847751E12,
                  "value_as_string" : "2020-07-05T20:10:47.751Z"
                }
              },
              {
                "key" : "shards",
                "doc_count" : 958138,
                "last_seen" : {
                  "value" : 1.593979599307E12,
                  "value_as_string" : "2020-07-05T20:06:39.307Z"
                }
              },
              {
                "key" : "enrich_coordinator_stats",
                "doc_count" : 48406,
                "last_seen" : {
                  "value" : 1.593979848536E12,
                  "value_as_string" : "2020-07-05T20:10:48.536Z"
                }
              },
              {
                "key" : "index_recovery",
                "doc_count" : 48253,
                "last_seen" : {
                  "value" : 1.59397984066E12,
                  "value_as_string" : "2020-07-05T20:10:40.660Z"
                }
              },
              {
                "key" : "indices_stats",
                "doc_count" : 48235,
                "last_seen" : {
                  "value" : 1.593979842932E12,
                  "value_as_string" : "2020-07-05T20:10:42.932Z"
                }
              },
              {
                "key" : "node_stats",
                "doc_count" : 48172,
                "last_seen" : {
                  "value" : 1.593979842847E12,
                  "value_as_string" : "2020-07-05T20:10:42.847Z"
                }
              },
              {
                "key" : "cluster_stats",
                "doc_count" : 47553,
                "last_seen" : {
                  "value" : 1.593979845203E12,
                  "value_as_string" : "2020-07-05T20:10:45.203Z"
                }
              }
            ]
          },
          "index" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : ".monitoring-es-7-mb-2020.07.05",
                "doc_count" : 924151
              },
              {
                "key" : ".monitoring-es-7-mb-2020.07.04",
                "doc_count" : 876188
              },
              {
                "key" : ".monitoring-es-7-mb-2020.07.03",
                "doc_count" : 867776
              },
              {
                "key" : ".monitoring-es-7-mb-2020.06.30",
                "doc_count" : 750670
              },
              {
                "key" : ".monitoring-es-7-mb-2020.07.02",
                "doc_count" : 636445
              },
              {
                "key" : ".monitoring-es-7-mb-2020.07.01",
                "doc_count" : 607738
              }
            ]
          }
        }
      ]
    }
  }
}

system · August 3, 2020, 5:18pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.