Metricbeat does not start service

Elastic Stack Beats
1

Hi guys

I've been trying to implement kibana in my aws. i have 4 instances in total

  1. elasticsearch
    2.logstash
  2. kibana,filebeat and metricbeat
  3. app instance (nginx)
    i would like, from kibana, be able to display meatricbeat of my app instance.
    ive been told to enable http module metricbeat and pass in host the app ip, which i did, and after that, i cannot start metricbeat service anymore. any suggestion please?
    this is the error.
● metricbeat.service - Metricbeat is a lightweight shipper for metrics.
   Loaded: loaded (/lib/systemd/system/metricbeat.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2020-05-05 00:10:44 UTC; 4s ago
     Docs: https://www.elastic.co/products/beats/metricbeat
  Process: 1757 ExecStart=/usr/share/metricbeat/bin/metricbeat $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=exited, status=1/FAILURE)
 Main PID: 1757 (code=exited, status=1/FAILURE)

May 05 00:10:44 ip-10-0-105-101 systemd[1]: metricbeat.service: Main process exited, code=exited, status=1/FAILURE
May 05 00:10:44 ip-10-0-105-101 systemd[1]: metricbeat.service: Failed with result 'exit-code'.
May 05 00:10:44 ip-10-0-105-101 systemd[1]: metricbeat.service: Service hold-off time over, scheduling restart.
May 05 00:10:44 ip-10-0-105-101 systemd[1]: metricbeat.service: Scheduled restart job, restart counter is at 5.
May 05 00:10:44 ip-10-0-105-101 systemd[1]: Stopped Metricbeat is a lightweight shipper for metrics..
May 05 00:10:44 ip-10-0-105-101 systemd[1]: metricbeat.service: Start request repeated too quickly.
May 05 00:10:44 ip-10-0-105-101 systemd[1]: metricbeat.service: Failed with result 'exit-code'.
May 05 00:10:44 ip-10-0-105-101 systemd[1]: Failed to start Metricbeat is a lightweight shipper for metrics..
2

Can you check /var/log/metricbeat/metricbeat.log and post that?

3

i got 2 logs:

metricbeat:

2020-05-04T23:52:02.047Z        INFO    instance/beat.go:622    Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
2020-05-04T23:52:02.048Z        INFO    instance/beat.go:630    Beat ID: 782dce8b-8418-486a-9254-567fc5e6dc36

metricbeat.1

2020-05-04T23:52:01.887Z        INFO    instance/beat.go:622    Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
2020-05-04T23:52:01.896Z        INFO    instance/beat.go:630    Beat ID: 782dce8b-8418-486a-9254-567fc5e6dc36

which are the same

4

Thanks, can you try running metricbeat manually and seeing what the output is?

5

sorry just to make sure i understand.

do you mean not provisioning my instance with metricbeat, but install it normally after the instance is app and running right?

6

if i print the journalctl i got this:

May 05 00:38:36 ip-10-0-105-101 systemd[1]: Failed to start Metricbeat is a lightweight shipper for metrics..
-- Subject: Unit metricbeat.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit metricbeat.service has failed.
-- 
-- The result is RESULT.
May 05 00:38:41 ip-10-0-105-101 sudo[1837]:   ubuntu : TTY=pts/0 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/bin/systemctl status metricbeat
May 05 00:38:41 ip-10-0-105-101 sudo[1837]: pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)
May 05 00:38:41 ip-10-0-105-101 sudo[1837]: pam_unix(sudo:session): session closed for user root
May 05 00:38:42 ip-10-0-105-101 sudo[1840]:   ubuntu : TTY=pts/0 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/bin/systemctl start metricbeat
May 05 00:38:42 ip-10-0-105-101 sudo[1840]: pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)
May 05 00:38:42 ip-10-0-105-101 systemd[1]: metricbeat.service: Start request repeated too quickly.
May 05 00:38:42 ip-10-0-105-101 systemd[1]: metricbeat.service: Failed with result 'exit-code'.
May 05 00:38:42 ip-10-0-105-101 systemd[1]: Failed to start Metricbeat is a lightweight shipper for metrics..
-- Subject: Unit metricbeat.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit metricbeat.service has failed.
-- 
-- The result is RESULT.
May 05 00:38:42 ip-10-0-105-101 sudo[1840]: pam_unix(sudo:session): session closed for user root
May 05 00:38:47 ip-10-0-105-101 filebeat[1155]: 2020-05-05T00:38:47.528Z        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"be
7

I mean run the metricbeat command directly on your host and see what the output is.

8

Finally i fixed it. Sorry for the late reply.

My mistake was so basic. When i provisioned my app instance with chef and build a image aws image, i forgot to include my beats cookbook to send data to elasticsearch. Once done that one i didnt need to activate the http forwarding.

The only thing i am missing, i can visualize only the filebeat from my app instance but not the metricbeat. Which from my kibana instance, i can see only the metric and not the filebeat. is that normal?

1 Like
9

Is filebeat running?

10

on kibana instance are both up and running..filebeat and metricbeat.

but on my app instance only filebeat is running. metricbeat no

11

Ok then you will need to start it, or install it if it's not installed.

12

it is installed, if i try to start the service it says:
sudo: unable to resolve host ip-10-0-102-48 which is the public ip of my instance

13

That shouldn't make a difference.

14

i dont know if this is important:

● metricbeat.service - Metricbeat is a lightweight shipper for metrics.
   Loaded: loaded (/lib/systemd/system/metricbeat.service; enabled; vendor preset: enabled)
   Active: failed (Result: start-limit-hit) since Tue 2020-05-05 02:01:29 UTC; 1s ago
     Docs: https://www.elastic.co/products/beats/metricbeat
  Process: 2417 ExecStart=/usr/share/metricbeat/bin/metricbeat $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=exited, status=1/FAILURE)
 Main PID: 2417 (code=exited, status=1/FAILURE)

next to fail it says limit-hit

15

It's heaps easier if you check the logs rather than this output, it'll usually show you what the problem is immediately.

Otherwise, try to run the Beat manually, like I mentioned above, and check its output.

16

you are right sorry.

if i run it manually i got this:

sudo: unable to resolve host ip-10-0-102-48
Exiting: no metricsets configured for module 'http'

i enabled my metricbeat and set my instance ip in both field. for port 80 and port 8080

17

Please post the Metricbeat HTTP module config, as you appear to be missing something.

18 
# Module: http
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.6/metricbeat-module-http.html

- module: http
  #metricsets:
  #  - json
  period: 10s
  hosts: ["3.249.247.52:80"]
  namespace: "json_namespace"
  path: "/"
  #body: ""
  #method: "GET"
  #username: "user"
  #password: "secret"
  #request.enabled: false
  #response.enabled: false
  #json.is_array: false
  #dedot.enabled: false

- module: http
  #metricsets:
  #  - server
  host: "3.249.247.52"
  port: "8080"
  enabled: false
  #paths:
  #  - path: "/foo"
  #    namespace: "foo"
  #    fields: # added to the the response in root. overwrites existing fields
  #      key: "value"
19

i spot my error. i didnt uncomment metricset and json

1 Like
20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.