Metricbeat to Logstash to ElasticSearch - Cannot See Any Hosts Defined, But Data Is Definitely Coming In

[jthart (Jesse Hart)]

For background information, I've tried following instructions in these posts and am still encountering an issue:

• Kibana Metrics – 500 Internal Server Error
• Host data wont populate in dashboards when upgrading beats from 7.0 to 7.3 and using Logstash
• There is no data to display - Infrastructure tab

I am certain there is something simple I am missing or doing incorrectly given that I am quite new to trying to figure out ELK for a test deployment, and so I am learning as I go. The only thing present in the logs when I click on the "check for new data" button is the following:

{"type":"response","@timestamp":"2021-06-11T15:40:13-04:00","tags":["access:infra"],"pid":31618,"method":"post","statusCode":200,"req":{"url":"/api/metrics/snapshot","method":"post","headers":{"host":"10.8.1.53:5601","accept":"*/*","kbn-version":"7.12.1","accept-encoding":"gzip, deflate","accept-language":"en-us","content-type":"application/json","origin":"http://10.8.1.53:5601","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/605.1.15","connection":"keep-alive","referer":"http://10.8.1.53:5601/app/metrics/inventory?waffleFilter=(expression:%27%27,kind:kuery)&waffleTime=(currentTime:1623439586927,isAutoReloading:!f)&waffleOptions=(accountId:%27%27,autoBounds:!t,boundsOverride:(max:1,min:0),customMetrics:!(),customOptions:!(),groupBy:!(),legend:(palette:cool,reverseColors:!f,steps:10),metric:(type:cpu),nodeType:host,region:%27%27,sort:(by:name,direction:desc),view:map)","content-length":"236"},"remoteAddress":"10.45.44.6","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/605.1.15","referer":"http://10.8.1.53:5601/app/metrics/inventory?waffleFilter=(expression:%27%27,kind:kuery)&waffleTime=(currentTime:1623439586927,isAutoReloading:!f)&waffleOptions=(accountId:%27%27,autoBounds:!t,boundsOverride:(max:1,min:0),customMetrics:!(),customOptions:!(),groupBy:!(),legend:(palette:cool,reverseColors:!f,steps:10),metric:(type:cpu),nodeType:host,region:%27%27,sort:(by:name,direction:desc),view:map)"},"res":{"statusCode":200,"responseTime":97,"contentLength":28},"message":"POST /api/metrics/snapshot 200 97ms - 28.0B"}

Given that I attempted to manually load the templates (and received an acknowledge message seeming to indicate it was successful), I'm not really sure where to go next. For the sake of reference, the below steps were taken to try to load the templates manually:

1. Export the template from the metricbeat client
2. Copy the exported template to one of the elasticsearch nodes
3. Install the exported template using the following command: curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/_template/metricbeat-7.13.1 -d@metricbeat.template.json (substituting the IP instead of localhost)
4. Stopped the two logstash nodes in the cluster
5. Deleted all indices under index management related to beats
6. Deleted all index patterns in kibana that were related
7. Started logstash on the two nodes
8. Confirmed indices were recreated and looked appropriate
9. Created a new index pattern for kibana that matched the metricbeat index and confirmed data was being received

I still have no defined hosts, but do have the following partial information displayed:

Any ideas on where to go from here to resolve would be greatly appreciated. At this point I just have no knowledge of where to look next.

[stephenb (Stephen Brown)]

Hi @jthart Welcome to the community apologies that you're having some struggles getting this set up Perhaps we can help.

Assuming you want to run an architecture like this

Metricbeat (1 to Many) -> Logstash -> Elasticsearch

Basically using Logstash as a collect and pass through

Here is my recommendation try to resist the urge to make this more complex. Do not try to manually load index templates dashboards anything else follow the quick start / basic setup.

1. Clean everything up we're starting over.

2. Follow the exact steps 1 through
   6 here... At this point metricbeat should be properly configured and sending telemetry directly to Elasticsearch and you should be able to view it in Kibana in metrics app.

3. Stop metricbeat.

4. Use this Logstash config. This configuration acts as a pass through and will take any and all the metadata associated with metric beat including if there's any pipelines etcetera to be run and pass them through logstash.

################################################
# beats->logstash->es default config.
################################################
input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      user => "elastic"
      password => "secret"
    }
  }
}

4. Start Logstash. It should be listening on the beats input port 5044.

5. Edit the metricbeat.yml
   Comment out the Kibana setup and the output.elasticsearch configurations.

Configure metricbeat to point at Logstash

output.logstash:
      # The Logstash hosts
      hosts: ["localhost:5044"]

6. Start metricbeat.

Now you should be sending metricbeat data through logstash to elasticsearch with all the mappings, index templates ILM dashboards etc

Note you only need to run the setup step once whether you're collecting metrics from one host or a thousand hosts setup only needs to be run once

[jthart (Jesse Hart)]

Hi @stephenb , thank you for the response.

Following the steps provided I see the same behavior (with a notable exception, which I'll explain in a moment). For the sake of confirmation, these are the steps I took:

1. Removed all existing indices from index management
2. Stopped logstash
3. Configured metricbeat to talk to elasticsearch instead of logstash and configured the kibana section
4. Initiated metricbeat setup and confirmed the new index information was present in kibana (this is where the notable change occurred, as the template was far more detailed than before)
5. Stopped metricbeat and configured logstash as you provided
6. Started logstash and reconfigured metricbeat to point to logstash

The only symptom is still that the inventory screen claims there is nothing to display, even though there are multiple systems sending in system data via metricbeat (that is the only enabled module at the moment, intentionally).

[stephenb (Stephen Brown)]

Did you start metricbeat? (not explicitly said)

If you go to Discover and look at metricbeat-* index pattern is there anything?

How are you actually starting and stopping metricbeat and logstash?

[jthart (Jesse Hart)]

Ha, no worries. Yes, metricbeat was started.

Yes, there is a lot of data present and seemingly indexed appropriately (i.e. all of the system metrics appear to be tagged appropriately).

All packages were installed via yum and are enabled/started/stopped via systemctl, so systemctl stop metricbeat for example.

Screenshot just to show there are records and the agent is properly identified, cropped to avoid having to scrub any data.

[stephenb (Stephen Brown)]

So what does the Metrics App Look Like? (I know that was at the top)
It can take a minute to show up? but not too long, I think you said you left all defaults (10s)

Also have you gone to Dashboards -> Metricbeat ECS System Overview?

In that discover are there fields like exported as defined here

system.cpu.total.pct

The percentage of CPU time spent in states other than Idle and IOWait.

type: scaled_float

format: percent

 system.cpu.total.norm.pct

    The percentage of CPU time in states other than Idle and IOWait, normalised by the number of cores.

    type: scaled_float

    format: percent

Does it work if you go metricbeat direct to Elasticsearch?

Ohh and DOH what version(s)?

[jthart (Jesse Hart)]

Screenshots to show current state at bottom. I had not gone to the dashboard, but that looks good and is showing the proper breakdown of hosts.

I have not seen the inventory screen work in any use so far (including when I let it run for a few minutes with metricbeat connected directly to elasticsearch on another trial of the original response, thinking maybe I missed a step).

Installed versions:

• elasticsearch-7.12.1-1.x86_64
• kibana-7.12.1-1.x86_64
• metricbeat-7.13.1-1.x86_64
• logstash-7.12.1-1.x86_64

[stephenb (Stephen Brown)]

Hmm Odd ....

In Discover.... Did you check to see if you have values in host.name there is host.hostname and host.name do you have both

Also anything strange here?

in that Metrics - Explorer what happens if you try to group by host.name

[jthart (Jesse Hart)]

That was it! I never looked at the values in settings because I didn't change them directly, but at some point the "Host name" field was overwritten to be beats.hostname instead. When I changed it to host.name everything populated appropriately. I'm going to assume this happened at some point during my attempts to manually import the templates originally instead of having metricbeat handle it directly.

I greatly appreciate the assistance.