Logstash 6.8.3 not exporting its metrics

Elastic Stack Logstash
1

I am running the rpm version of Logstash 6.8.3 on full GUI based CentOS 7 running locally on VM Player on my desktop.
Downloaded the rpm version from official site.
This version does not export its metrics via xpack.monitoring .
Interestingly, when I installed the same rpm version on minimal CentOS 7 VM running on Azure cloud, it exports its metrics to Elasticsearch.
Is this issue related to OSS builds? I don't understand what it means.

My logstash.yml file:

node.name: logstash-vm0

path.data: /var/lib/logstash

pipeline.id: server-pipeline

pipeline.workers: 2

pipeline.batch.size: 125

# pipeline.batch.delay: 50

# pipeline.unsafe_shutdown: false

#path.config: /etc/logstash/pipelines/*.conf

# config.string:

# config.test_and_exit: false

config.reload.automatic: true

config.reload.interval: 5s

# config.debug: false

# config.support_escapes: false

# modules:
#   - name: MODULE_NAME
#     var.PLUGINTYPE1.PLUGINNAME1.KEY1: VALUE


# modules:

# cloud.id: <identifier>
#
# cloud.auth: elastic:<password>

queue.type: "persisted"
path.queue: /var/lib/logstash/queue
queue.page_capacity: 64mb
queue.max_events: 0
queue.max_bytes: 1024mb
queue.checkpoint.writes: 1024
queue.checkpoint.interval: 1000
dead_letter_queue.max_bytes: 1024mb

path.dead_letter_queue: /var/lib/logstash/dead_letter_queue

# ------------ Metrics Settings --------------
# Bind address for the metrics REST endpoint
http.host: "10.0.0.7"
# Bind port for the metrics REST endpoint, this option also accept a range
# (9600-9700) and logstash will pick up the first available ports.
http.port: 9600


path.logs: /var/log/logstash

# ------------ X-Pack Settings (not applicable for OSS build)--------------
#
# X-Pack Monitoring
# https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password: "******"
xpack.monitoring.elasticsearch.hosts: "https://elasticsearch-client-vm0:9200"
xpack.monitoring.elasticsearch.ssl.certificate_authority: "/etc/logstash/certs/elasticsearch-master-vm0/ca.crt"
xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
xpack.monitoring.elasticsearch.sniffing: false
xpack.monitoring.collection.interval: 10s
xpack.monitoring.collection.pipeline.details.enabled: true

# X-Pack Management
# https://www.elastic.co/guide/en/logstash/current/logstash-centralized-pipeline-management.html
xpack.management.enabled: true
xpack.management.pipeline.id: ["server-pipeline"]
xpack.management.elasticsearch.username: "logstashadmin"
xpack.management.elasticsearch.password: "******"
xpack.management.elasticsearch.hosts: "https://elasticsearch-client-vm0:9200"
xpack.management.elasticsearch.ssl.certificate_authority: "/etc/logstash/certs/elasticsearch-master-vm0/ca.crt"
xpack.management.elasticsearch.ssl.verification_mode: certificate
xpack.management.elasticsearch.sniffing: false
xpack.management.logstash.poll_interval: 5s
2

@Mohit_Kunjir OSS builds do not contain xpack-licensed features. For more information on x-pack versus non-xpack, please see here: https://www.elastic.co/what-is/open-x-pack

3

@Mike_Place The RPM version of Logstash installed in Azure VM (CentOS 7 ) exports its metrics to Elasticsearch cluster.
The same RPM version when installed locally on my VM Player (CentOS 7) does not export the metrics to the above mentioned Elasticsearch cluster.
I went through your link, but am still failing to understand why is this so.
Please Help.

4

Most likely you pulled them from different repos. If Azure is pulling from the elastic.co repo, then it's getting the package with x-pack features whereas the second VM downloaded a package which does not bundle x-pack features. If you want to have those x-pack features, you should download your RPM from the elastic.co repo: https://www.elastic.co/downloads/logstash

5

I should also ask whether you are trying to send monitoring data from both to the same cluster or different clusters?

6

@Mike_Place I followed the steps mentioned below to install Logstash on Azure Vm (CentOS 7)

  1. Downloaded the Logstash 6.8.3 RPM version from Official Repo on Azure VM (CentOS 7) on path /home/logstash:
    wget https://artifacts.elastic.co/downloads/logstash/logstash-6.8.3.rpm
  2. Installed the Logstash 6.8.3 RPM version:
    sudo yum install ./logstash-6.8.3.rpm
  3. Replaced the existing logstash.yml present at path /etc/logstash/ with the one posted above.
  4. Started the Logstash service:
    sudo systemctl start logstash.service

When Logstash service started, it began exporting its metrics to elasticsearch-client-vm0 as per the X-Pack Monitoring configuration mentioned in logstash.yml file.
The message Connected to X-Pack monitoring... was also seen in logstash logs.
The logstash instance then could be seen in stack monitoring on Kibana.

However, when I repeated the above mentioned steps on my local VM (CentOS 7) running on VM Player the Logstash instance running locally did not export its metrics to elasticsearch-client-vm0.
Also Connected to X-Pack Monitoring... was too not seen in the local logstash logs.
I can assure you this is not due to network permissions as the Beat agents running on the local VM are exporting their metrics to elasticsearch-client-vm0.
It is only the local Logstash which is not exporting its metrics.

7

@Mike_Place I am using the same X-Pack Monitoring configuration in both the Logstash (Local & Cloud) instances.
So both Logstash instances should export their metrics to elasticsearch-client-vm0 (same cluster).

8

On the second (non-working) VM, could you please post the output of rpm -qi logstash?

9

@Mike_Place Here is the requested output:

    [mohit@localhost ~]$ rpm -qi logstash
    Name        : logstash
    Epoch       : 1
    Version     : 6.8.3
    Release     : 1
    Architecture: noarch
    Install Date: Thursday 24 October 2019 01:23:55 PM IST
    Group       : default
    Size        : 303419110
    License     : Elastic License
    Signature   : RSA/SHA512, Friday 30 August 2019 02:22:04 AM IST, Key ID d27d666cd88e
    Source RPM  : logstash-6.8.3-1.src.rpm
    Build Date  : Friday 30 August 2019 02:04:54 AM IST
    Build Host  : packer-virtualbox-iso-1559162487
    Relocations : / 
    Packager    : <vagrant@packer-virtualbox-iso-1559162487>
    Vendor      : Elasticsearch
    URL         : http://www.elasticsearch.org/overview/logstash/
    Summary     : An extensible logging pipeline
    Description :
    An extensible logging pipeline
10

I have recently seen heard some reports that suggest that there may be a problem with xpack.management and xpack.monitoring being enabled at the same time. If you comment out the xpack.management lines does anything change in regard to monitoring?

12

@Mike_Place Although I stated above that I am using the same logstash config file for both local and cloud VM, the xpack pipeline management settings have been set to false in the local VM. The Logstash running on the local VM, first, parses the logs and then sends the parsed logs to the cloud VM. The cloud VM Logstash (which uses xpack pipeline management) then just indexes the parsed logs into appropriate Elasticsearch indices. So, as you can see from the use case, I can't use centralized pipeline management for the local VM. They both run different pipelines.

Also, both X-Pack metrics and X-Pack pipeline management have been enabled in the Logstash runinng in the cloud VM, and it does exports its metrics, so no problems there.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.