Hi all. I'm using logstash to send data from redshift to Elasticsearch. I monitor my pipelines with metricbeats, and am able to see general statistics about logstash within my monitoring cluster. What I can't see is logs. Is there a simple way to send logs from logstash into Elasticsearch?
You should use a Filebeat to read the logstash logs and send it to your monitoring cluster.
Also see filebeat module enable logstash
Thank you! I tried it out but I'm not seeing any log data in Kibana. I'm using elastic cloud. I wonder what I am doing wrong? My metricbeats works properly, and I can see that filebeat has started successfully.
Here is my configuration:
docker compose
version: "3"
services:
logstash:
image: docker.elastic.co/logstash/logstash:7.15.1
environment:
- NODE_NAME=concepts
- XPACK_MONITORING_ENABLED=false
- "LS_JAVA_OPTS=-Xmx3g -Xms3g"
env_file:
- .env
volumes:
- ./jars/:/usr/share/jars/
- ./pipeline/:/usr/share/logstash/pipeline/
- ./sql_last_value.yml:/usr/share/sql_last_value.yml
command: logstash
metricbeat:
image: docker.elastic.co/beats/metricbeat:7.15.1
depends_on:
- logstash
env_file:
- .env
volumes:
- ./metricbeat.yml:/usr/share/metricbeat/metricbeat.yml
filebeat:
image: docker.elastic.co/beats/filebeat:7.15.1
depends_on:
- logstash
env_file:
- .env
volumes:
- ./filebeat.yml:/usr/share/filebeat/filebeat.yml
filebeat.yml
filebeat.modules:
- module: logstash
log:
enabled: true
output:
elasticsearch:
hosts: ["${ES_HOST_MONITORING_PROD}"]
username: "${ES_USER_MONITORING_PROD}"
password: "${ES_PASSWORD_MONITORING_PROD}"
metricbeat.yml
metricbeat.modules:
- module: logstash
metricsets:
- node
- node_stats
period: 10s
hosts: ["logstash:9600"]
xpack.enabled: true
strict.perms: false
output:
elasticsearch:
hosts: ["${ES_HOST_MONITORING_PROD}"]
username: "${ES_USER_MONITORING_PROD}"
password: "${ES_PASSWORD_MONITORING_PROD}"
Filebeat needs to have access to the Logstash logs, you are running filebeat on a different container, this won't work.
I do not use docker, so I can't help further, but you need to look on how to configure filebeat to read docker logs.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.