# "Standalone Cluster" for Logstash monitoring

**URL:** https://discuss.elastic.co/t/standalone-cluster-for-logstash-monitoring/233898
**Category:** Logstash
**Tags:** elastic-stack-monitoring, beats-module
**Created:** [May 22, 2020, 12:05pm UTC](https://discuss.elastic.co/t/standalone-cluster-for-logstash-monitoring/233898 "2020-05-22T12:05:33Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![pup\_seba](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pup_seba/32/42988_2.png) [@pup\_seba](https://discuss.elastic.co/u/pup_seba)
#### Post date: [May 22, 2020, 12:05pm UTC](https://discuss.elastic.co/t/standalone-cluster-for-logstash-monitoring/233898/1 "2020-05-22T12:05:33Z")

</div>

Hi,

I'm trying to setup a single node with Elasticsearch, Kibana and Logstash. After adding some filebeat agents to some servers I wanted to start centralizing logs, a new cluster was created under Kibana "monitoring". Everything was "fine" while using heartbeat and metricbeats (both directly connected to Elasticsearch). Filebeat directs its output to Logstash.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/5/55e0062c3a389961f678b2d3993414976e920b7b.png)

After some digging around, it seems like it has something to do with me configuring the outputs of the filebeats to go to Logstash. These are some of the online resources I checked before posting here:  
[https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html](https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html)

> <https://github.com/elastic/beats/pull/14338>

  

> <https://github.com/elastic/beats/pull/13182>

  

> <https://github.com/elastic/logstash/pull/11106>

  

> <https://github.com/elastic/logstash/issues/11066>
>
> Background and Problem
> Logstash monitoring data is stored in .monitoring-logstash-\* indices for the Stack Monitoring application. A key field in these documents...

This is my relevant portion of logstash settings (logstash.yml) file:

```auto
node.name: "elk01"
config.reload:
  automatic: true
  interval: 3s

#xpack.monitoring.cluster_uuid: "xhb_MPjYRfeZMR4ORTAEaA"
#xpack.monitoring.cluster_uuid: xhb_MPjYRfeZMR4ORTAEaA
#xpack.monitoring.elasticsearch.cloud_id: monitoring_cluster_id: xhb_MPjYRfeZMR4ORTAEaA
#monitoring.cluster_uuid: xhb_MPjYRfeZMR4ORTAEaA
#monitoring.cluster_uuid: "xhb_MPjYRfeZMR4ORTAEaA"

xpack.monitoring:
  enabled: false
  #cluster_uuid: xhb_MPjYRfeZMR4ORTAEaA
  #cluster_uuid: "xhb_MPjYRfeZMR4ORTAEaA"
  elasticsearch:
    username: logstash_system
    password: xxxxx
    #cluster_uuid: "xhb_MPjYRfeZMR4ORTAEaA"
    hosts: ["https://fqdn:9200"]
    ssl:
      certificate_authority: /etc/elasticsearch/certs/ca.crt
      verification_mode: certificate

log.level: info

queue:
  type: persisted
  max_bytes: 10gb

```

This is the current and temporary (for test purposes only) logstash configuration I'm using.

```auto
input {
        beats {
                port => "5044"
                ssl => true
                ssl_key => '/etc/logstash/certs/elk01_pck8.key'
                ssl_certificate => '/etc/logstash/certs/elk01.crt'
                ssl_certificate_authorities => '/etc/logstash/certs/ca.crt'
        }
}
output {
  file {
    path => '/tmp/output.logstash'
  }
}

```

I left the commented lines that did not worked. The error logged for those attempts was saying that:

```auto
[2020-05-22T13:14:54,047][FATAL][logstash.runner] An unexpected error occurred! {:error=>#<ArgumentError: Setting "xpack.monitoring.cluster_uuid" hasn't been registered>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:69:in `get_setting'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:102:in `set_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:121:in `block in merge'", "org/jruby/RubyHash.java:1428:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:121:in `merge'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:179:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:284:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:242:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `<main>'"]}

```

In the hosts I have metricbeat and filebeat enabled. In the all-in-one server I don't have filebeat but I do have heartbeat. This is how filebeat.yml looks like in the servers:

```auto
filebeat.inputs:
- type: log
  enabled: true
  paths: /opt/zimbra/log/mailbox.log
  multiline.pattern: ^\d{4}-\d{2}-\d{2}
  multiline.negate: true
  multiline.match: after
- type: log
  enabled: true
  paths:
    - /opt/zimbra/log/audit.log
    - /var/log/zimbra.log
- type: log
  enabled: false
  paths:
    - /var/log/*.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true 
  reload.period: 3s
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.logstash:
  hosts: ["fqdn:5044"]
  ssl.certificate_authorities: ["/etc/filebeat/certs/ca.crt"]
  ssl.certificate: "/etc/filebeat/certs/elk01.crt"
  ssl.key: "/etc/filebeat/certs/elk01.key"
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
http.enabled: true
http.port: 5067
monitoring.enabled: false
monitoring.cluster_uuid: "xhb_MPjYRfeZMR4ORTAEaA"

```

There are no modules enabled for filebeat.

This is how metricbeat is configured (metricbeat.yml)

```auto
metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true 
  reload.period: 3s
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
setup.kibana:
  host: "https://fqdn:5601"
  ssl.certificate_authorities: ["/etc/metricbeat/certs/ca.crt"]
output.elasticsearch:
  hosts: ["https://fqdn:9200"]
  protocol: "https"
  username: "elastic"
  password: "xxxxx"
  ssl.enabled: true 
  ssl.certificate_authorities: ["/etc/metricbeat/certs/ca.crt"]
  ssl.certificate: "/etc/metricbeat/certs/elk01.crt"
  ssl.key: "/etc/metricbeat/certs/elk01.key"
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
http.enabled: true
monitoring.enabled: false
monitoring.cluster_uuid: "xhb_MPjYRfeZMR4ORTAEaA"

```

For metricbeat, I do have enabled a couple of modules which are system and beat-xpack. In the all-in-one I have some other modules enabled (logstash-xpack and kibana-xpack). Here are the confs for logstash-xpack and beat-xpack.

```auto
- module: logstash
  metricsets:
    - node
    - node_stats
  period: 10s
  xpack.enabled: true
  hosts: ["localhost:9600"]
  username: "elastic"
  password: "xxxxx"
  ssl.enabled: true 
  ssl.certificate_authorities: ["/etc/metricbeat/certs/ca.crt"]
  monitoring.override_cluster_uuid: xhb_MPjYRfeZMR4ORTAEaA
  monitoring.cluster_uuid: xhb_MPjYRfeZMR4ORTAEaA

```

```auto
- module: beat
  metricsets:
    - stats
    - state
  period: 10s
  xpack.enabled: true
  hosts: ["http://localhost:5066"]
  username: "elastic"
  password: "xxxxxx"
  ssl.enabled: true 
  ssl.certificate_authorities: ["/etc/metricbeat/certs/ca.crt"]

```

Even when logstash starts and stays up, I can see this in logs. Don't know if it is related/relevant for this issue:

```auto
[2020-05-22T13:23:00,320][WARN][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.RubyArray) has been created for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.

```

Any idea what am I doing wrong? My idea is to have a single-cluster (cluster-primero) with all the beats reporting correctly to it. I've tried to use the "beats" way of doing it instead of the "legacy monitoring" but i think I may be confusing some configurations between them both.

Thank you very much for any help you may provide.  
Edit: Looking into the .monitoring-logstash indices, I see that from a specific hour the documents sent by metricbeat to this index, are missing the "cluster\_uuid" and "logstash\_stats.process.cpu.percent". So, for some reason the "cluster\_uuid" was being sent but now is missing and thus, the "standalone cluster". I can't see how to fix this 😕

---

<div class="post-metadata">

### Author: ![Alsheh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alsheh/32/36799_2.png) [@Alsheh](https://discuss.elastic.co/u/Alsheh)
#### Post date: [May 26, 2020, 3:39am UTC](https://discuss.elastic.co/t/standalone-cluster-for-logstash-monitoring/233898/2 "2020-05-26T03:39:29Z")

</div>

ensure metricbeat and logstash are running the same minor version since fields may end up in different locations between minor versions.

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 26, 2020, 5:36am UTC](https://discuss.elastic.co/t/standalone-cluster-for-logstash-monitoring/233898/3 "2020-05-26T05:36:38Z")

</div>

Please don't post pictures of text, they are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them 🙂

---

<div class="post-metadata">

### Author: ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)
#### Post date: [May 26, 2020, 7:32pm UTC](https://discuss.elastic.co/t/standalone-cluster-for-logstash-monitoring/233898/4 "2020-05-26T19:32:24Z")

</div>

What version of Logstash are you using? Support for `monitoring.cluster_uuid` didn't exist before 7.7.0.

---

<div class="post-metadata">

### Author: ![pup\_seba](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pup_seba/32/42988_2.png) [@pup\_seba](https://discuss.elastic.co/u/pup_seba)
#### Post date: [May 29, 2020, 10:39am UTC](https://discuss.elastic.co/t/standalone-cluster-for-logstash-monitoring/233898/5 "2020-05-29T10:39:21Z")

</div>

I feel really silly 😕 I assumed that they were all in the same version as I installed them all on the same day. For some reason, logstash was in 7.6.2. I just updated it to 7.7.x (no change in repo files...). So just now they are all running on the same version and it makes complete sense that's the error as per this attribute was not available until 7.7.

It's kind of hard for me to understand how this monitoring works really. Documentation and wizards don't always talk about the same steps to configure things and on top of that for some modules the "xpack" is needed and for others is not. Still learning though...hopefully a second lab will be better configured 🙂

Thank you very much for your help!

---

<div class="post-metadata">

### Author: ![pup\_seba](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pup_seba/32/42988_2.png) [@pup\_seba](https://discuss.elastic.co/u/pup_seba)
#### Post date: [May 29, 2020, 10:42am UTC](https://discuss.elastic.co/t/standalone-cluster-for-logstash-monitoring/233898/6 "2020-05-29T10:42:48Z")

</div>

Hi!

done, I just change all the images per code. You are completle right, sorry about that.

The problem (should) be solved now as per my mistake of not realizing what version of logstash I was using (insert "doh" homer gif here).

I will need to deploy again this environment anyways, hopefully with a cleaner idea of what and how to configure things. I'm quite confused with the monitoring part to be honest. I'm guessing once the "legacy" option dissapears, things will be more clear to me.

Thank you very much for your suggestion.

---

<div class="post-metadata">

### Author: ![pup\_seba](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pup_seba/32/42988_2.png) [@pup\_seba](https://discuss.elastic.co/u/pup_seba)
#### Post date: [May 29, 2020, 10:45am UTC](https://discuss.elastic.co/t/standalone-cluster-for-logstash-monitoring/233898/7 "2020-05-29T10:45:31Z")

</div>

I was aware that the logstash version needed to be 7.7...I was "so sure" that the version I was using was 7.7 (all the other components are). Just when I was getting logstash version to update this post I saw that I was using 7.6. I'm gonna punish myself with a 10hs youtube video of nickelback songs for this.

Thank you very much!

---

<div class="post-metadata">

### Author: ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)
#### Post date: [June 1, 2020, 2:05pm UTC](https://discuss.elastic.co/t/standalone-cluster-for-logstash-monitoring/233898/8 "2020-06-01T14:05:33Z")

</div>

> [@pup\_seba](#):
>
> It's kind of hard for me to understand how this monitoring works really. Documentation and wizards don't always talk about the same steps to configure things and on top of that for some modules the "xpack" is needed and for others is not. Still learning though...hopefully a second lab will be better configured

This is great feedback and we very much appreciate your honesty and perspective.

You are not wrong at all. It is complex and we offer multiple ways of doing "the same thing". We will take a note to improve the documentation, but we also feel confident that this will become simpler in the future.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 29, 2020, 2:13pm UTC](https://discuss.elastic.co/t/standalone-cluster-for-logstash-monitoring/233898/9 "2020-06-29T14:13:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
