Logs not being recieved by Elasticsearch/Kibana from filebeats after following quick install instructions

Elastic Stack Kibana
1

Hi All,

Filebeats is connecting with Kibana, and I assume it is also connecting with Elasticsearch too. I can't understand, however, why the logs are not being recieived. This is what I see when accessing Kibana:

Screenshot 2025-04-22 180820
Screenshot 2025-04-22 1808201918×808 171 KB

The health of the data stream from is amber and it hasn't been updated.

I have checked file permissions and all users have read access for the file path where the log file is stored. I have also added new lines saving each time to the txt log file (test.log), while running filebeats, and it still doesn't pull them through.

Details
Windows
Running locally

Versions:
filebeat-9.0.0-windows-x86_64
kibana-8.17.4-windows-x86_64
elasticsearch-8.17.4-windows-x86_64

filebeats.yml:

filebeat.inputs:
- type: filestream
  id: app1-logs-id
  enabled: true
  paths:
    - C:/CI/test_logs/*
        
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

setup.template.settings:
  index.number_of_shards: 1
setup.dashboards.enabled: true

setup.kibana:
  host: "localhost:5601"
  username: "elastic"
  password: "elastic"

output.elasticsearch:
  hosts: ["localhost:9200"]
  preset: balanced
  username: "elastic"
  password: "elastic"
  ssl.verification_mode: none


processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Commands run are:
.\filebeat setup
.\filebeat.exe -c filebeat.yml -e
This produces command line messages like the following:

*{"log.level":"info","@timestamp":"2025-04-22T18:33:36.864+0100",
"log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot","file.name":"log/log.go","file.line":192},
"message":"Non-zero metrics in the last 30s","service.name":
"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":578,"time":{"ms":16}},"total":{"ticks":1656,"time":{"ms":16},"value":1656},
"user":{"ticks":1078}},"info":{"ephemeral_id":"7dfefa69-710e-41ea-8731-a41c66779ed7","uptime":{"ms":420344},"version":"9.0.0"},
"memstats":{"gc_next":43000810,"memory_alloc":20707896,"memory_total":245233600,"rss":84660224},"runtime":{"goroutines":33}},"filebeat":{"harvester":{"open_files":0,"running":0}},
"libbeat":{"config":{"module":{"running":0},"scans":3},"output":{"events":{"active":0},
"write":{"latency":{"histogram":{"count":0,"max":0,"mean":0,"median":0,"min":0,"p75":0,"p95":0,"p99":0,"p999":0,
"stddev":0}}}},"pipeline":{"clients":0,"events":{"active":0},"queue":{"filled":{"bytes":0,"events":0,"pct":0},"max_bytes":0,"max_events":3200}}},
"registrar":{"states":{"current":0}},"system":{"handles":{"open":-3}}},"ecs.version":"1.6.0"}}*

I have been playing around with it for a few days now and not got anywhere, so any help or suggestions would be greatly appreciated!

2

in your first image you have data streams and under index there is 1. click on it.
it will go to indices. do you have document count on that?

enable verbose logging in filebeat to see if there is anything else

3

one more thing I just noticed is you running filebeat 9.0 while elastic is lower version. may be try both same version

4

The document count was 0, but I tried using the same version of filebeats as elasticsearch, as you kindly suggested, and it worked.

Thank you for your help!

1 Like