Logstash send logs to monitoring cluster

Hi all. I'm using logstash to send data from redshift to Elasticsearch. I monitor my pipelines with metricbeats, and am able to see general statistics about logstash within my monitoring cluster. What I can't see is logs. Is there a simple way to send logs from logstash into Elasticsearch?

You should use a Filebeat to read the logstash logs and send it to your monitoring cluster.

Also see filebeat module enable logstash

Thank you! I tried it out but I'm not seeing any log data in Kibana. I'm using elastic cloud. I wonder what I am doing wrong? My metricbeats works properly, and I can see that filebeat has started successfully.

Here is my configuration:

docker compose

version: "3"
services:
  logstash:
    image: docker.elastic.co/logstash/logstash:7.15.1
    environment:
      - NODE_NAME=concepts
      - XPACK_MONITORING_ENABLED=false
      - "LS_JAVA_OPTS=-Xmx3g -Xms3g"
    env_file:
      - .env
    volumes:
      - ./jars/:/usr/share/jars/
      - ./pipeline/:/usr/share/logstash/pipeline/
      - ./sql_last_value.yml:/usr/share/sql_last_value.yml
    command: logstash
  metricbeat:
    image: docker.elastic.co/beats/metricbeat:7.15.1
    depends_on:
      - logstash
    env_file:
      - .env
    volumes:
      - ./metricbeat.yml:/usr/share/metricbeat/metricbeat.yml

  filebeat:
    image: docker.elastic.co/beats/filebeat:7.15.1
    depends_on:
      - logstash
    env_file:
      - .env
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml

filebeat.yml

filebeat.modules:
- module: logstash
  log:
    enabled: true

output:
  elasticsearch:
    hosts: ["${ES_HOST_MONITORING_PROD}"]
    username: "${ES_USER_MONITORING_PROD}"
    password: "${ES_PASSWORD_MONITORING_PROD}"

metricbeat.yml

metricbeat.modules:
- module: logstash
  metricsets:
    - node
    - node_stats
  period: 10s
  hosts: ["logstash:9600"]
  xpack.enabled: true
  strict.perms: false

output:
  elasticsearch:
    hosts: ["${ES_HOST_MONITORING_PROD}"]
    username: "${ES_USER_MONITORING_PROD}"
    password: "${ES_PASSWORD_MONITORING_PROD}"

Filebeat needs to have access to the Logstash logs, you are running filebeat on a different container, this won't work.

I do not use docker, so I can't help further, but you need to look on how to configure filebeat to read docker logs.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.