Filebeat is not recognized on kibana

I am running my elk stack through docker containers. The Elasticsearch and kibana are running normally. I have a filebeat receiving syslog and netflow on 2 different port. The problem is that the connection is refused when I have a telnet test and also the filebeat is not recognized by kibana on the stack monitor page.

Here is the config I am using for the filebeat:

filebeat.yml

filebeat.inputs:
- type: syslog
  format: rfc5424
  protocol.tcp:
    host: "localhost:20010"

- type: netflow
  max_message_size: 10KiB
  host: "localhost:20011"
  protocols: [ v5, v9, ipfix ]
  expiration_timeout: 30m
  queue_size: 8192
  detect_sequence_reset: true


#========================== Elasticsearch output ===============================
output.elasticsearch:
  hosts: ["${ELASTICSEARCH_HOST}:9200"]
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}

#============================== Dashboards =====================================
setup.dashboards:
  enabled: true

#============================== Kibana =========================================
setup.kibana:
  host: "${KIBANA_HOST}:5601"
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}

# ================================== General ===================================
name: lab01_ecs_log
tags: ["syslog"]

docker-compose.yml

version: '3.2'

services:
  filebeat:
    build:
      context: filebeat/
      args:
        ELK_VERSION: $ELK_VERSION
    # user: root  #To read the docker socket
    volumes:
      # - /var/run/docker.sock:/host_docker/docker.sock:ro
      # - /var/lib/docker:/host_docker/var/lib/docker:ro
      - ./filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      # - /var/log/syslog:/var/log/syslog:ro,z
    command: ["--strict.perms=false"]
    ports:
      - "20010:20010"
      - "20011:20011"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    # docker run -i
    stdin_open: true
    # docker run -t
    tty: true
    networks:
      - fbeat
    environment:
      - ELASTICSEARCH_HOST=10.40.14.33
      - KIBANA_HOST=10.40.14.33
      - ELASTICSEARCH_USERNAME=elastic
      - ELASTICSEARCH_PASSWORD=changeme
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "50"

networks:
  fbeat:
    driver: bridge

logs from filebeat:

2022-02-07T14:39:15.326Z        INFO    instance/beat.go:869    Kibana dashboards successfully loaded.
2022-02-07T14:39:15.326Z        INFO    instance/beat.go:492    filebeat start running.
2022-02-07T14:39:15.335Z        INFO    memlog/store.go:119     Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2022-02-07T14:39:15.335Z        INFO    memlog/store.go:124     Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0
2022-02-07T14:39:15.335Z        INFO    [registrar]     registrar/registrar.go:109      States Loaded from registrar: 0
2022-02-07T14:39:15.335Z        INFO    [crawler]       beater/crawler.go:71    Loading Inputs: 2
2022-02-07T14:39:15.335Z        INFO    [crawler]       beater/crawler.go:141   Starting input (ID: 9279620832289625231)
2022-02-07T14:39:15.336Z        INFO    [crawler]       beater/crawler.go:141   Starting input (ID: 5323989155346491818)
2022-02-07T14:39:15.336Z        INFO    [crawler]       beater/crawler.go:108   Loading and starting Inputs completed. Enabled inputs: 2
2022-02-07T14:39:15.336Z        INFO    [syslog]        syslog/input.go:147     Starting Syslog input   {"protocol": "tcp"}
2022-02-07T14:39:15.336Z        INFO    [netflow]       netflow/input.go:151    Starting UDP input
2022-02-07T14:39:15.336Z        INFO    [UDP]   dgram/server.go:99      Started listening for UDP connection
2022-02-07T14:39:15.336Z        INFO    [tcp]   streaming/listener.go:120       Started listening for TCP connection    {"address": "localhost:20010"}
2022-02-07T14:39:24.514Z        INFO    [monitoring]    log/log.go:184  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":294993271}},"memory":{"mem":{"usage":{"bytes":2437120}}}},"cpu":{"system":{"ticks":940,"time":{"ms":63}},"total":{"ticks":6770,"time":{"ms":255},"value":6770},"user":{"ticks":5830,"time":{"ms":192}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":14},"info":{"ephemeral_id":"2dcc8d7f-5b5c-422f-89c6-a25cf2c112be","uptime":{"ms":90123},"version":"7.16.3"},"memstats":{"gc_next":23040544,"memory_alloc":19049312,"memory_sys":262144,"memory_total":513671528,"rss":134684672},"runtime":{"goroutines":91}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":1.15,"15":1.44,"5":1.49,"norm":{"1":0.0205,"15":0.0257,"5":0.0266}}}}}}
2022-02-07T14:39:54.515Z        INFO    [monitoring]    log/log.go:184  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":9781199}},"memory":{"mem":{"usage":{"bytes":307200}}}},"cpu":{"system":{"ticks":960,"time":{"ms":23}},"total":{"ticks":6790,"time":{"ms":23},"value":6790},"user":{"ticks":5830}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":13},"info":{"ephemeral_id":"2dcc8d7f-5b5c-422f-89c6-a25cf2c112be","uptime":{"ms":120117},"version":"7.16.3"},"memstats":{"gc_next":23040544,"memory_alloc":20437664,"memory_total":515059880,"rss":135606272},"runtime":{"goroutines":89}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":1.29,"15":1.45,"5":1.49,"norm":{"1":0.023,"15":0.0259,"5":0.0266}}}}}}

I see that the filebeat is started and the dashboard is loaded successfully. But I cannot see the beat on the kibana stack monitor page and also when I open the dashboard to check the syslog information, there is nothing updating.

I would like to ask if there is any thing wrong in my set up.

Thanks.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.