8.1.1 Cannot display Logstash pipeline in monitoring UI

Elastic Stack Kibana
1

I have migrated the Elastic stack from 7.17.1 to 8.1.1. After the upgrade everything works fine, except the Kibana monitoring logstash pipeline viewer UI.

stack monitoring > logstash > pipelines

image
image2258×529 41.2 KB

pipeline main points to https://fqdn/app/monitoring#/logstash/pipelines/main?_g=(cluster_uuid:cluster_id,refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now))

After clicking on the main pipeline, I get an empty screen.

POST request https://fqdn/api/monitoring/v1/clusters/cluster_id/logstash/pipeline/main returns payload

"pipeline":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Pipeline [main @ ] not found in the selected time range for cluster [cluster_id]."},"headers":{}}}}
2

Have you checked you pipelines, for example the ingestion pipelines after running the setup command for the various modules ?

3

Hi @zx8086, what can I check? All my Logstash pipeline config files are properly working (tested using logstash -t -f %s).

[2022-03-24T12:58:38,338][INFO ][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
4

How are you monitoring ? Metricbeat ? Logstash-xpack module ?

5

yes, correct

6

Would help to see the Logstash configuration as well as the metricbeat module configuration and logs. Metricbeat logs will show if the modules is working and data being sent not.

7

/etc/logstash/logstash.yml

path.logs: /var/log/logstash
http.host: 192.168.1.213
http.port: 9600
pipeline.ecs_compatibility: disabled
xpack.management.enabled: false
monitoring.cluster_uuid: "cluster_id"

/etc/metricbeat/modules.d/logstash-xpack.yml

- module: logstash
  xpack.enabled: true
  period: 10s
  hosts: ["http://192.168.1.213:9600"]
  username: "remote_monitoring_user"
  password: "xxxxxxxxx"
  metricsets:
    - node
    - node_stats

grep metricbeat /var/log/syslog

image

8

Configured this ?

monitoring.enabled: false

Also you can use a curl command to see if the http endpoint is working

9

sorry I left out lines below (/etc/logstash/logstash.yml)

# disable internal monitoring; replaced by metricbeat
monitoring.enabled: false

curl http://remote_monitoring_user:xxxxxxx@192.168.1.213:9600

{"host":"xxxxxx","version":"8.1.1","http_address":"192.168.1.213:9600","id":"xxxxxxxxxx","name":"xxxxxxx","ephemeral_id":"xxxxxxxxxx","status":"green","snapshot":false,"pipeline":{"workers":24,"batch_size":125,"batch_delay":50},"monitoring":{"cluster_uuid":"cluster_id"},"build_date":"2022-03-17T22:01:47Z","build_sha":"xxxxxxxxxx","build_snapshot":false}
10

Do you see the same calls reflected in the metricbeat logs ? Is metricbeat making the call to that same address, same credentials and having the same response ?

11

Also what about using the default settings? Metricbeat is running on the same node as logstash right ? So localhost should work too.

- module: logstash
  xpack.enabled: true
  period: 10s
  hosts: ["localhost:9600"]
  #username: "user"
  #password: "secret"```
12

localhost does not work as the port is not bound to all interfaces (but only to the IP adres 192.168.1.213)

13

I only see metricbeat log events like Non-zero metrics in the last 30s in /var/log/syslog.

..."logstash":{"node_stats":{"events":3,"success":3}}...
14

And if you remove the metricset ? that might be the problem

15

I have removed the metricsets and restarted the metricbeat service. Same issue. I have restored the config file and restarted the service again.

16

The logstash config you shared, was that the full one ? Hard to diagnose the issue on limited information.

17

yes, I only left out monitoring.enabled: false

18

This issue appears to be fixed in version 8.1.2

19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.