Not seeing any data in the monitoring tab

Elastic Stack Kibana
21

If you are referring to the out put am seeing in kibana for heartbeat .

{
  "_index": "heartbeat-2019.05.08-000001",
  "_type": "_doc",
  "_id": "rMbhnWoB3maiDp4b8tnO",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-05-09T18:36:30.778Z",
    "tls": {
      "rtt": {
        "handshake": {
          "us": 17418
        }
      },
      "certificate_not_valid_before": "0001-01-01T00:00:00.000Z"
    },
    "fields": {
      "version": "10.6"
    },
    "agent": {
      "ephemeral_id": "3749dae6-fa12-445e-a247-b01ac8b6a4be",
      "hostname": "hostname1.changed.com",
      "id": "2ad2e338-bbe9-4068-b6fc-fc60e7cc1d71",
      "version": "7.0.0",
      "type": "heartbeat"
    },
    "monitor": {
      "type": "http",
      "check_group": "157de181-7289-11e9-a7d1-005056ac9316",
      "ip": "127.0.0.1",
      "status": "up",
      "duration": {
        "us": 40478
      },
      "id": "auto-http-0XEB9699FEE4978E58",
      "name": ""
    },
    "summary": {
      "up": 1,
      "down": 0
    },
    "tcp": {
      "rtt": {
        "connect": {
          "us": 144
        }
      }
    },
    "event": {
      "dataset": "uptime"
    },
    "ecs": {
      "version": "1.0.0"
    },
    "host": {
      "name": "hostname1.changed.com",
      "hostname": "hostname1.changed.com",
      "architecture": "x86_64",
      "os": {
        "platform": "rhel",
        "version": "7.5 (Maipo)",
        "family": "redhat",
        "name": "Red Hat Enterprise Linux Server",
        "kernel": "3.10.0-862.14.4.el7.x86_64",
        "codename": "Maipo"
      },
      "id": "0709cd0acc86463da7d518c28f145a66",
      "containerized": true
    },
    "http": {
      "rtt": {
        "response_header": {
          "us": 1722
        },
        "validate": {
          "us": 1773
        },
        "content": {
          "us": 50
        },
        "total": {
          "us": 39971
        },
        "write_request": {
          "us": 48
        }
      },
      "response": {
        "status_code": 200
      }
    },
    "resolve": {
      "rtt": {
        "us": 306
      },
      "ip": "127.0.0.1"
    },
    "url": {
      "scheme": "https",
      "domain": "hostname1.changed.com",
      "port": 7443,
      "path": "/home",
      "full": "https://hostname1.changed.com:7443"
    }
  },
  "fields": {
    "@timestamp": [
      "2019-05-09T18:36:30.778Z"
    ],
    "tls.certificate_not_valid_before": [
      "0001-01-01T00:00:00.000Z"
    ]
  },
  "sort": [
    1557426990778
  ]
}
22

What confuses me is that you don't have any _source in the response and I think you should.

What does this show?

POST .monitoring-es-*/_search
{
  "size": 1,
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "type": "cluster_stats"
          }
        }
      ]
    }
  },
  "sort": {
    "timestamp": {
      "order": "desc"
    }
  }
}
23

Can you tell me where I can add that. Do I need to delete any template and add it.

{
  "took" : 11,
  "timed_out" : false,
  "_shards" : {
    "total" : 8,
    "successful" : 8,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : [
      {
        "_index" : ".monitoring-es-7-2019.05.09",
        "_type" : "_doc",
        "_id" : "rMfwnWoB3maiDp4bzAXs",
        "_score" : null,
        "sort" : [
          1557427965086
        ]
      }
    ]
  }
}
24

Can you run and return the result of this:
GET .monitoring-es-7-*/_mapping

25
26 
".monitoring-es-7-2019.05.09" : {
        "mappings" : {
          "dynamic" : "false",
          "_source" : {
            "enabled" : false
          },

Here is your problem. The source is disabled.

Do you see the same property set in the result of this call:
GET _template/.monitoring-es

27

I don't see _source field here.

28

If you look at the previous days monitoring indices, do you see the same _source.enabled: false? Also, look at any other .monitoring-* indices as well. I'm trying to understand what might be causing this.

29

Yes, In the previous indices I see it as false.

 "_source": {
      "enabled": false

This is for kibana

"mapping": {
    "dynamic": "false",
    "_source": {
      "enabled": false
    },
30

Can you post the response of GET _template? It will be long but we need to find out if another template exists that is applying this setting for the .monitoring-* indices (I suspect there is)

31 
.monitoring-alerts   [.monitoring-alerts-6]     0          6050399
.monitoring-es       [.monitoring-es-7-*]       0          7000099
.ml-anomalies-       [.ml-anomalies-*]          0          7000099
.watch-history-9     [.watcher-history-9*]      2147483647 
.monitoring-beats    [.monitoring-beats-7-*]    0          7000099
.monitoring-kibana   [.monitoring-kibana-7-*]   0          7000099
.watches             [.watches*]                2147483647 
.ml-notifications    [.ml-notifications]        0          7000099
.monitoring-alerts-7 [.monitoring-alerts-7]     0          7000099
.ml-state            [.ml-state*]               0          7000099
.ml-config           [.ml-config]               0          7000099
32

Found it.

 "template_1" : {
    "order" : 0,
    "index_patterns" : [
      "*"
    ],
    "settings" : {
      "index" : {
        "number_of_shards" : "1"
      }
    },
    "mappings" : {
      "_source" : {
        "enabled" : false
      }
    },
    "aliases" : { }
  },

Deleting that template will fix it. However, this might have been added for a reason so perhaps redo this so it excludes .monitoring-* indices.

Once you resolve that, the next index created (for tomorrow) will then start containing the _source and the monitoring UI should start working. Alternatively, you can delete all .monitoring-* indices (after deleting the template) as those don't even have data right now. With this route, you should start seeing monitoring data in the UI soon after.

33

Thank you. Will try your solution.

34

After deleting the template and deleting monitoring indices. I am able to see the monitoring dashboard. Thanks for helping me out.

35

@chrisronline One last thing. My uptime dashboard is also blank. Just wondering if its the same issue which is stopping the data to be shown on up-time dashboard.

36

Most likely, yes. That index template was causing every indexed document to contain no source which means there is basically no data in those documents.

37

I mean am still not seeing any thing on my uptime dashboard. I believe it gets data from heartbeat index. I already deleted the template which was causing monitoring tab to not display.

38

My Heartbeat template.

       "mappings" : {
          "_meta" : {
            "beat" : "heartbeat",
            "version" : "7.0.0"
          },
          "_source" : {
            "enabled" : true
          },
          "dynamic_templates" : [
            {
39

Check the mappings for the most recent heartbeat index. Is the source enabled or disabled there?

40

I don't see a _source field in my index mapping. But in heartbeat template I can see the _source enabled. Do you think its taking picking this change from some other template?

I tried to delete the heartbeat template and create new template but no change.