Metricbeat data is not read by Logstash

Elastic Stack Logstash
1

Hi Team,

I have configured Metricbeat to capture system level metrics, Metricbeat is intsalled as a service in Redhat linux , i installed metricbeat using rpm package.
Bellow is my metricbeat configuration .

> #============================  Config Reloading ==============================
> metricbeat.config.modules:
> 
>   # Glob pattern for configuration reloading
>   path: ${path.config}/conf.d/*.yml
> 
>   # Period on which files under path should be checked for changes
>   reload.period: 10s
> 
>   # Set to true to enable config reloading
>   reload.enabled: false
> 
> # Maximum amount of time to randomly delay the start of a metricset. Use 0 to
> # disable startup delay.
> metricbeat.max_start_delay: 10s
> 
> #------------------------------- System Module -------------------------------
> - module: system
>   metricsets:
>     - cpu             # CPU usage
>     - load            # CPU load averages
>     - memory          # Memory usage
>     - network         # Network IO
>     - process         # Per process metrics
>     - process_summary # Process summary
>     - uptime          # System Uptime
>     #- core           # Per CPU core usage
>     #- diskio         # Disk IO
>     #- filesystem     # File system usage for each mountpoint
>     #- fsstat         # File system summary metrics
>     #- raid           # Raid
>     #- socket         # Sockets and connection info (linux only)
>   enabled: true
>   period: 10s
>   processes: ['.*']
> 
>   # Configure the metric types that are included by these metricsets.
>   cpu.metrics:  ["percentages"]  # The other available options are normalized_percentages and ticks.
>   core.metrics: ["percentages"]  # The other available option is ticks.
> 
> 
> #----------------------------- Logstash output ---------------------------------
> output.logstash:
>   # Boolean flag to enable or disable the output module.
>   #enabled: true
> 
>   # The Logstash hosts
>   hosts: ["localhost:5044"]

And logstash is also installed as service in Redhat linux using rpm package.

bellow is my logstash configuration where i am trying to read inputs from beat

input {
beats {
port => 5044
}
}

output {
elasticsearch {
host => "localhost:9200"
index => "metricbeat_logs"
}

}

bellow is my user and group permissions for both logstash and metricbeat

cd /etc/logstash
ls -lrt

> -rw-------. 1 apelkdev apa 1696 Aug 18 02:29 startup.options
> -rw-r--r--. 1 apelkdev apa  285 Aug 18 02:29 pipelines.yml
> -rw-r--r--. 1 apelkdev apa  342 Aug 18 02:29 logstash-sample.conf
> -rw-r--r--. 1 apelkdev apa 4466 Aug 18 02:29 log4j2.properties
> -rw-r--r--. 1 apelkdev apa 1846 Aug 18 02:29 jvm.options
> -rwxr-xr-x. 1 apelkdev apa 8154 Feb  8 07:41 logstash.yml
> drwxrwxr-x. 2 apelkdev apa 4096 Feb  8 10:29 conf.d

cd /etc/metricbeat
ls -lrt

> -rw-r--r--. 1 apelkdev apa 57809 Aug 18 00:28 metricbeat.reference.yml
> -rw-r--r--. 1 apelkdev apa 76697 Aug 18 00:28 fields.yml
> -rw-r--r--. 1 apelkdev apa     0 Feb  7 06:24 metricbeat.
> -rw-------. 1 apelkdev apa  7892 Feb  8 08:14 metricbeat.yml

and I am running both the services using bellow commands

sudo service metricbeat start
sudo service logstash start

I am really not getting what went wrong here, when we check metricbeat logs it is having captured metrics details for current timestamp, not very sure what happens to logstash and why it is not reading metricbeat events.

2

Did you check Logstash log also to see if there are any issues?

3

@admlko
Yeah everything in working fine, logstash pipeline created successfully trying to read inputs at localhost:5044

4

So you have checked that both services are actually running, Logstash is listening the socket and there are no errors in either log files?

Pretty much only one thing caught my eye, in the Metricbeat output.logstash you have commented out the enabled line. I checked the documentation and that should not be an issue, as it is enabled by default. But you might want to try to enable it manually.

Sorry I may be asking dumb questions, never actually used metricbeat.

5

let me try that

6

Hmm, set it up for fun in my lab, doesn't seem to be working out of the box how I was expecting. Might give it another shot later on when I find time.

7

Hi

issue is resolved by enabling logstash module explicitly.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.