Filebeat: connection to Elasticsearch OK but no documents indexed

decibel83 · September 29, 2025, 2:03pm

Hi everyone,
I’m facing an issue with Filebeat 8.19.3 on Debian 12 (Bookworm).

Symptoms

Filebeat starts without errors and connects successfully to Elasticsearch:

Connection to backoff(elasticsearch(https://<ES_HOST>:9200)) established
doBulkRequest: 1600 events have been sent to elasticsearch in ~70ms

No Failed to index event or status code errors appear in the logs.
However, in Kibana I can’t see any documents in the filebeat-* or logs-* data streams.

Environment

OS: Debian GNU/Linux 12 (bookworm), kernel 6.1.0-38-amd64
Filebeat: 8.19.3 (installed from official Elastic package)
Output: direct to Elasticsearch 8.19.3 over HTTPS
Minimal configuration: enabled system (auth) and system (syslog) modules.
Configured paths:
```
/var/log/auth.log*
/var/log/syslog*
```

Filebeat configuration file

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

setup.kibana:
  host: localhost:5601
  space.id:

output.elasticsearch:
  hosts: ["https://<ES_HOST>:9200"]
  preset: balanced
  protocol: "https"
  api_key: ""
  username: "filebeat_setup"
  password: "<ES_PASSWORD>"

output.logstash:
  enabled: false

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

logging.level: debug
logging.selectors: ["elasticsearch"]

setup.ilm.check_exists: true

Log excerpt

{"log.level":"info","message":"filebeat start running.","service.name":"filebeat"}
{"log.level":"info","message":"Enabled modules/filesets: system (auth), system (syslog)","service.name":"filebeat"}
{"log.level":"info","message":"Configured paths: [/var/log/auth.log*]","service.name":"filebeat"}
{"log.level":"info","message":"Configured paths: [/var/log/syslog*]","service.name":"filebeat"}
{"log.level":"info","message":"Connection to backoff(elasticsearch(https://<ES_HOST>:9200)) established","service.name":"filebeat"}
{"log.level":"debug","message":"doBulkRequest: 1600 events have been sent to elasticsearch in 69ms.","service.name":"filebeat"}

What I’ve already checked

Filebeat communicates with Elasticsearch without errors.
/var/log/syslog and /var/log/auth.log contain new events.
In Kibana → Data view filebeat-* shows no documents.
ILM is enabled (lifecycle policy filebeat exists already.) and the filebeat-8.19.3 template exists.

Question

What am I missing?
If bulk requests are being successfully sent to Elasticsearch, why can’t I find any documents in the filebeat-* data streams?

Thanks a lot in advance!

AndersonQ · September 30, 2025, 7:29am

Hi @decibel83 ,

would you have the 30s metrics log to share? or the query the monitoring /inputs endpoint and share the result. It might explain what is happening with the events.

leandrojmp · September 30, 2025, 3:26pm

Do you have other data streams? Like logs-system.* ? Your filebeat shows that events were sent to Elasticsearch.

I'm not sure it will write into filebeat-* or into logs-system.*

decibel83 · October 2, 2025, 12:55pm

Yes, but logs-* is also empty:

leandrojmp · October 2, 2025, 12:58pm

What is the time frame? This is a data view (kibana feature to show data streams), to see the data streams you need to go into Stack Management > Index Management > Data Streams or run GET _cat/indices?v on Kibana Dev Tools.

Can you run GET _cat/indices?v on Kibana Dev Tools and share the result?

If the connection is ok and the modules are enabled then I would not expect any issue, something may not be configured correctly or working correctly.

decibel83 · October 2, 2025, 1:03pm

This is another log excerpt from today:

Oct  2 14:57:57 elk filebeat[313502]: {"log.level":"info","@timestamp":"2025-10-02T14:57:57.405+0200","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":138},"message":"Connecting to backoff(elasticsearch(https://elk.my.domain:9200))","service.name":"filebeat","ecs.version":"1.6.0"}
Oct  2 14:57:57 elk filebeat[313502]: {"log.level":"info","@timestamp":"2025-10-02T14:57:57.409+0200","log.logger":"elasticsearch.esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping","file.name":"eslegclient/connection.go","file.line":324},"message":"Attempting to connect to Elasticsearch version 8.19.3 (default)","service.name":"filebeat","ecs.version":"1.6.0"}
Oct  2 14:57:57 elk filebeat[313502]: {"log.level":"info","@timestamp":"2025-10-02T14:57:57.420+0200","log.logger":"index-management","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/idxmgmt.(*indexManager).Setup","file.name":"idxmgmt/index_support.go","file.line":254},"message":"Auto lifecycle enable success.","service.name":"filebeat","ecs.version":"1.6.0"}
Oct  2 14:57:57 elk filebeat[313502]: {"log.level":"info","@timestamp":"2025-10-02T14:57:57.422+0200","log.logger":"index-management.ilm","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/idxmgmt/lifecycle.(*stdManager).EnsurePolicy","file.name":"lifecycle/standard_manager.go","file.line":111},"message":"lifecycle policy filebeat exists already.","service.name":"filebeat","ecs.version":"1.6.0"}
Oct  2 14:57:57 elk filebeat[313502]: {"log.level":"info","@timestamp":"2025-10-02T14:57:57.422+0200","log.logger":"index-management","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/idxmgmt.applyLifecycleSettingsToTemplate","file.name":"idxmgmt/index_support.go","file.line":402},"message":"Set settings.index.lifecycle.name in template to filebeat as ILM is enabled.","service.name":"filebeat","ecs.version":"1.6.0"}
Oct  2 14:57:57 elk filebeat[313502]: {"log.level":"info","@timestamp":"2025-10-02T14:57:57.431+0200","log.logger":"template_loader","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/template.(*ESLoader).Load","file.name":"template/load.go","file.line":121},"message":"Template \"filebeat-8.19.3\" already exists and will not be overwritten.","service.name":"filebeat","ecs.version":"1.6.0"}
Oct  2 14:57:57 elk filebeat[313502]: {"log.level":"info","@timestamp":"2025-10-02T14:57:57.431+0200","log.logger":"index-management","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/idxmgmt.(*indexManager).Setup","file.name":"idxmgmt/index_support.go","file.line":299},"message":"Loaded index template.","service.name":"filebeat","ecs.version":"1.6.0"}
Oct  2 14:57:57 elk filebeat[313502]: {"log.level":"info","@timestamp":"2025-10-02T14:57:57.432+0200","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":146},"message":"Connection to backoff(elasticsearch(https://elk.my.domain:9200)) established","service.name":"filebeat","ecs.version":"1.6.0"}
Oct  2 14:57:57 elk filebeat[313502]: {"log.level":"debug","@timestamp":"2025-10-02T14:57:57.438+0200","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).doBulkRequest","file.name":"elasticsearch/client.go","file.line":317},"message":"doBulkRequest: 111 events have been sent to elasticsearch in 5.917839ms.","service.name":"filebeat","ecs.version":"1.6.0"}
Oct  2 14:58:07 elk filebeat[313502]: {"log.level":"warn","@timestamp":"2025-10-02T14:58:07.302+0200","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.NewClient.func3","file.name":"elasticsearch/client.go","file.line":179},"message":"Failed to index 111 events in last 10s: events were dropped! Look at the event log to view the event and cause.","service.name":"filebeat","ecs.version":"1.6.0"}
Oct  2 14:58:07 elk filebeat[313502]: {"log.level":"info","@timestamp":"2025-10-02T14:58:07.405+0200","log.logger":"input.harvester","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Harvester).Run","file.name":"log/harvester.go","file.line":313},"message":"Harvester started for paths: [/var/log/auth.log*]","service.name":"filebeat","input_id":"3802b47a-4011-43fc-a425-a5cd4eaabe88","source_file":"/var/log/auth.log","state_id":"native::96-64773","finished":false,"os_id":"96-64773","old_source":"/var/log/auth.log","old_finished":true,"old_os_id":"96-64773","harvester_id":"941c8ba5-915c-446b-8c69-75b9e189e9a4","ecs.version":"1.6.0"}
Oct  2 14:58:07 elk filebeat[313502]: {"log.level":"debug","@timestamp":"2025-10-02T14:58:07.417+0200","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).doBulkRequest","file.name":"elasticsearch/client.go","file.line":317},"message":"doBulkRequest: 64 events have been sent to elasticsearch in 7.92079ms.","service.name":"filebeat","ecs.version":"1.6.0"}
Oct  2 14:58:17 elk filebeat[313502]: {"log.level":"warn","@timestamp":"2025-10-02T14:58:17.302+0200","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.NewClient.func3","file.name":"elasticsearch/client.go","file.line":179},"message":"Failed to index 64 events in last 10s: events were dropped! Look at the event log to view the event and cause.","service.name":"filebeat","ecs.version":"1.6.0"}
Oct  2 14:58:17 elk filebeat[313502]: {"log.level":"info","@timestamp":"2025-10-02T14:58:17.305+0200","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot","file.name":"log/log.go","file.line":192},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"id":"filebeat.service"},"memory":{"id":"filebeat.service","mem":{"usage":{"bytes":183050240}}}},"cpu":{"system":{"ticks":50,"time":{"ms":50}},"total":{"ticks":190,"time":{"ms":190},"value":190},"user":{"ticks":140,"time":{"ms":140}}},"handles":{"limit":{"hard":524288,"soft":524287},"open":15},"info":{"ephemeral_id":"7d2a3b4f-6851-45fd-a877-ab585974d991","name":"filebeat","uptime":{"ms":33057},"version":"8.19.3"},"memstats":{"gc_next":47130290,"memory_alloc":35617992,"memory_sys":57169160,"memory_total":69264920,"rss":157614080},"runtime":{"goroutines":60}},"filebeat":{"events":{"active":32,"added":213,"done":181},"harvester":{"open_files":2,"running":2,"started":2}},"libbeat":{"config":{"module":{"running":1,"starts":1},"reloads":1,"scans":1},"output":{"events":{"active":0,"batches":2,"dropped":175,"total":175},"read":{"bytes":9506,"errors":2},"type":"elasticsearch","write":{"bytes":31897,"latency":{"histogram":{"count":2,"max":7,"mean":6,"median":6,"min":5,"p75":7,"p95":7,"p99":7,"p999":7,"stddev":1}}}},"pipeline":{"clients":2,"events":{"active":32,"filtered":6,"published":207,"total":213},"queue":{"acked":175,"added":{"bytes":284208,"events":207},"consumed":{"bytes":242974,"events":175},"filled":{"bytes":41234,"events":32,"pct":0.01},"max_bytes":0,"max_events":3200,"removed":{"bytes":242974,"events":175}}}},"registrar":{"states":{"current":4,"update":181},"writes":{"success":3,"total":3}},"system":{"cpu":{"cores":4},"load":{"1":5.62,"15":4.3,"5":4.72,"norm":{"1":1.405,"15":1.075,"5":1.18}}}},"ecs.version":"1.6.0"}}
Oct  2 14:58:18 elk filebeat[313502]: {"log.level":"debug","@timestamp":"2025-10-02T14:58:18.421+0200","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).doBulkRequest","file.name":"elasticsearch/client.go","file.line":317},"message":"doBulkRequest: 62 events have been sent to elasticsearch in 6.589463ms.","service.name":"filebeat","ecs.version":"1.6.0"}
Oct  2 14:58:27 elk filebeat[313502]: {"log.level":"warn","@timestamp":"2025-10-02T14:58:27.302+0200","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.NewClient.func3","file.name":"elasticsearch/client.go","file.line":179},"message":"Failed to index 62 events in last 10s: events were dropped! Look at the event log to view the event and cause.","service.name":"filebeat","ecs.version":"1.6.0"}
Oct  2 14:58:30 elk filebeat[313502]: {"log.level":"debug","@timestamp":"2025-10-02T14:58:30.426+0200","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).doBulkRequest","file.name":"elasticsearch/client.go","file.line":317},"message":"doBulkRequest: 58 events have been sent to elasticsearch in 6.599465ms.","service.name":"filebeat","ecs.version":"1.6.0"}
Oct  2 14:58:37 elk filebeat[313502]: {"log.level":"warn","@timestamp":"2025-10-02T14:58:37.302+0200","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.NewClient.func3","file.name":"elasticsearch/client.go","file.line":179},"message":"Failed to index 58 events in last 10s: events were dropped! Look at the event log to view the event and cause.","service.name":"filebeat","ecs.version":"1.6.0"}
Oct  2 14:58:42 elk filebeat[313502]: {"log.level":"debug","@timestamp":"2025-10-02T14:58:42.430+0200","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).doBulkRequest","file.name":"elasticsearch/client.go","file.line":317},"message":"doBulkRequest: 56 events have been sent to elasticsearch in 6.622743ms.","service.name":"filebeat","ecs.version":"1.6.0"}
Oct  2 14:58:47 elk filebeat[313502]: {"log.level":"warn","@timestamp":"2025-10-02T14:58:47.302+0200","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.NewClient.func3","file.name":"elasticsearch/client.go","file.line":179},"message":"Failed to index 56 events in last 10s: events were dropped! Look at the event log to view the event and cause.","service.name":"filebeat","ecs.version":"1.6.0"}
Oct  2 14:58:47 elk filebeat[313502]: {"log.level":"info","@timestamp":"2025-10-02T14:58:47.306+0200","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot","file.name":"log/log.go","file.line":192},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":183586816}}}},"cpu":{"system":{"ticks":60,"time":{"ms":10}},"total":{"ticks":220,"time":{"ms":30},"value":220},"user":{"ticks":160,"time":{"ms":20}}},"handles":{"limit":{"hard":524288,"soft":524287},"open":13},"info":{"ephemeral_id":"7d2a3b4f-6851-45fd-a877-ab585974d991","uptime":{"ms":63057},"version":"8.19.3"},"memstats":{"gc_next":47130290,"memory_alloc":39530616,"memory_total":73177544,"rss":158572544},"runtime":{"goroutines":56}},"filebeat":{"events":{"active":1,"added":145,"done":176},"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0,"batches":3,"dropped":176,"total":176},"read":{"bytes":1847,"errors":3},"write":{"bytes":28389,"latency":{"histogram":{"count":5,"max":7,"mean":6,"median":6,"min":5,"p75":6.5,"p95":7,"p99":7,"p999":7,"stddev":0.6324555320336759}}}},"pipeline":{"clients":2,"events":{"active":1,"published":145,"total":145},"queue":{"acked":176,"added":{"bytes":188362,"events":145},"consumed":{"bytes":228442,"events":176},"filled":{"bytes":1154,"events":1,"pct":0.0003125},"max_bytes":0,"max_events":3200,"removed":{"bytes":228442,"events":176}}}},"registrar":{"states":{"current":4,"update":176},"writes":{"success":3,"total":3}},"system":{"load":{"1":5.57,"15":4.34,"5":4.8,"norm":{"1":1.3925,"15":1.085,"5":1.2}}}},"ecs.version":"1.6.0"}}
Oct  2 14:58:54 elk filebeat[313502]: {"log.level":"debug","@timestamp":"2025-10-02T14:58:54.437+0200","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).doBulkRequest","file.name":"elasticsearch/client.go","file.line":317},"message":"doBulkRequest: 31 events have been sent to elasticsearch in 6.34368ms.","service.name":"filebeat","ecs.version":"1.6.0"}
Oct  2 14:58:57 elk filebeat[313502]: {"log.level":"warn","@timestamp":"2025-10-02T14:58:57.302+0200","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.NewClient.func3","file.name":"elasticsearch/client.go","file.line":179},"message":"Failed to index 31 events in last 10s: events were dropped! Look at the event log to view the event and cause.","service.name":"filebeat","ecs.version":"1.6.0"}

Below you can find the 30s metrics log:

Oct  2 15:00:17 elk filebeat[313502]: {"log.level":"info","@timestamp":"2025-10-02T15:00:17.305+0200","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot","file.name":"log/log.go","file.line":192},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":184909824}}}},"cpu":{"system":{"ticks":80,"time":{"ms":10}},"total":{"ticks":310,"time":{"ms":30},"value":310},"user":{"ticks":230,"time":{"ms":20}}},"handles":{"limit":{"hard":524288,"soft":524287},"open":13},"info":{"ephemeral_id":"7d2a3b4f-6851-45fd-a877-ab585974d991","uptime":{"ms":153057},"version":"8.19.3"},"memstats":{"gc_next":47739730,"memory_alloc":30341296,"memory_sys":262144,"memory_total":83799728,"rss":158572544},"runtime":{"goroutines":56}},"filebeat":{"events":{"active":29,"added":119,"done":146},"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0,"batches":3,"dropped":146,"total":146},"read":{"bytes":1748,"errors":3},"write":{"bytes":23786,"latency":{"histogram":{"count":13,"max":7,"mean":5.923076923076923,"median":6,"min":5,"p75":6,"p95":7,"p99":7,"p999":7,"stddev":0.6153846153846154}}}},"pipeline":{"clients":2,"events":{"active":29,"published":119,"total":119},"queue":{"acked":146,"added":{"bytes":155555,"events":119},"consumed":{"bytes":189579,"events":146},"filled":{"bytes":37131,"events":29,"pct":0.0090625},"max_bytes":0,"max_events":3200,"removed":{"bytes":189579,"events":146}}}},"registrar":{"states":{"current":4,"update":146},"writes":{"success":3,"total":3}},"system":{"load":{"1":4.07,"15":4.29,"5":4.53,"norm":{"1":1.0175,"15":1.0725,"5":1.1325}}}},"ecs.version":"1.6.0"}}

This is the /inputs endpoint result:

root@elk /e/filebeat# http 127.0.0.1:5066/inputs/
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 27
Content-Type: application/json; charset=utf-8
Date: Thu, 02 Oct 2025 13:01:25 GMT
Vary: Accept-Encoding

[]

In fact, I don’t have any input enabled in the Filebeat configuration, but I have the system module enabled:

root@elk /e/filebeat# cat /etc/filebeat/modules.d/system.yml
# Ansible managed

# Filebeat default configuration for module system on Linux

- module: system
  syslog:
    enabled: true
    var.paths: ["/var/log/syslog*"]
  auth:
    enabled: true
    var.paths: ["/var/log/auth.log*"]

Thank you!

decibel83 · October 2, 2025, 1:18pm

This is the result:

health status index                                                              uuid                   pri rep docs.count docs.deleted store.size pri.store.size dataset.size
green  open   .internal.alerts-transform.health.alerts-default-000001            sWH16blfRN2nYfqwcisluQ   1   0          0            0       249b           249b         249b
green  open   .internal.alerts-ml.anomaly-detection.alerts-default-000001        yqwbR1hOSlaWTYIvYAsaHw   1   0          0            0       249b           249b         249b
green  open   .ds-metricbeat-8.18.0-2025.08.18-000004                            xJOH4qg1SQanDaVO35gaBg   1   0   62119534            0     22.9gb         22.9gb       22.9gb
green  open   .internal.alerts-observability.slo.alerts-default-000001           KIPmb7dbSqKAsfbOyZ6Ehg   1   0          0            0       249b           249b         249b
green  open   .internal.alerts-default.alerts-default-000001                     d3f-IGrsQ_GrRdYJC2VBKA   1   0          0            0       249b           249b         249b
green  open   .internal.alerts-streams.alerts-default-000001                     jT1Ea5x1R9Sn10gGnfKK9g   1   0          0            0       249b           249b         249b
green  open   .internal.alerts-observability.apm.alerts-default-000001           p5OF5C9KQbCkPwftph8sPg   1   0          0            0       249b           249b         249b
green  open   .ds-metricbeat-8.14.1-2025.08.18-000008                            wV4Kw2d4SbiZrI9r6AP6hA   1   0   42728873            0       10gb           10gb         10gb
yellow open   .ds-metricbeat-8.14.1-2025.09.17-000012                            7V-K60ZURmqwnQtl2hnGDg   1   1   24664721            0      5.9gb          5.9gb        5.9gb
green  open   .internal.alerts-security.attack.discovery.alerts-default-000001   91_6LkPTSlW-qO0XYPjBUQ   1   0          0            0       249b           249b         249b
yellow open   .ds-metricbeat-8.19.3-2025.09.28-000002                            Iv5SfpyDR9Oz0tQO87jnQw   1   1   11845711            0      5.1gb          5.1gb        5.1gb
green  open   .internal.alerts-observability.metrics.alerts-default-000001       zshIHHb6T6ibnDbV70Y_cA   1   0          0            0       249b           249b         249b
yellow open   .ds-metricbeat-8.18.0-2025.09.17-000008                            2ydPcUJITi-Hn2OEwtQKSw   1   1   41806453            0     15.5gb         15.5gb       15.5gb
green  open   .ds-metricbeat-8.18.3-2025.08.18-000002                            3Dgqb6hRSbizlgcFgwgT8w   1   0   17527949            0        5gb            5gb          5gb
green  open   .internal.alerts-ml.anomaly-detection-health.alerts-default-000001 mb5AWlCSS1iVMtLcWSyMbQ   1   0          0            0       249b           249b         249b
green  open   .internal.alerts-security.alerts-default-000001                    f5_z3qb7RjuNmD-OHijFlA   1   0          0            0       249b           249b         249b
green  open   .internal.alerts-stack.alerts-default-000001                       gXTKuJVbQhmh32_BcjER4Q   1   0          0            0       249b           249b         249b
yellow open   .ds-metricbeat-8.17.0-2025.09.26-000038                            yXtn5nVXTROnqyncQ-SOtg   1   1   55503141            0     25.6gb         25.6gb       25.6gb
green  open   .ds-metricbeat-8.18.1-2025.08.18-000003                            voi4enygTbKiGsy3sgJkyw   1   0          0            0       249b           249b         249b
green  open   .ds-metricbeat-8.17.0-2025.08.29-000029                            l60q8TFnS4erJJjhaQ1f6g   1   0  107333766            0       50gb           50gb         50gb
green  open   .internal.alerts-observability.logs.alerts-default-000001          YdUuPea5S1qeP1sxCvUupw   1   0          0            0       249b           249b         249b
green  open   .ds-metricbeat-8.19.2-2025.08.18-000001                            5x2iAg1eSX-v_D9_AuzLTw   1   0   43755886            0     19.2gb         19.2gb       19.2gb
green  open   .internal.alerts-observability.uptime.alerts-default-000001        MOHMxRdDSVCdDRsZ_H31iQ   1   0          0            0       249b           249b         249b
green  open   .ds-metricbeat-8.17.0-2025.09.10-000030                            GyfK_Gs_RFGLdRjgDffRdQ   1   0  107847695            0     49.9gb         49.9gb       49.9gb
green  open   .ds-metricbeat-8.17.0-2025.08.18-000028                            UcJW2ZkSQYK__zHWxeg6rA   1   0  105907946            0       50gb           50gb         50gb
yellow open   .ds-metricbeat-8.18.3-2025.09.17-000004                            iIVbqWZdTviivCewUB46CA   1   1   10315184            0      2.9gb          2.9gb        2.9gb
green  open   .kibana-observability-ai-assistant-conversations-000001            eB35q-YiTLerLR4Sav3oYQ   1   0          0            0       249b           249b         249b
yellow open   .ds-winlogbeat-8.14.1-2025.09.18-000001                            YPxzJRDDQButZhYRUdV4Tw   1   1   36433713            0     18.3gb         18.3gb       18.3gb
green  open   .ds-metricbeat-8.19.3-2025.08.29-000001                            bBdU7qCYQVqF-Cdz0uhscA   1   0   48895428            0       21gb           21gb         21gb
green  open   .internal.alerts-observability.threshold.alerts-default-000001     sb2k-X2DQ5W7TtLsZ2TGYw   1   0          0            0       249b           249b         249b
yellow open   .ds-filebeat-8.19.3-2025.09.25-000001                              Y_-CPWDbT_CD9VPfJtToNg   1   1          0            0       249b           249b         249b
green  open   .internal.alerts-dataset.quality.alerts-default-000001             qQx2K-L8Sp2uOJacEeQa1g   1   0          0            0       249b           249b         249b
green  open   .kibana-observability-ai-assistant-kb-000001                       5xTfI1D9RzKBb7HUlwyEhA   1   0          0            0       249b           249b         249b
yellow open   .ds-metricbeat-8.19.2-2025.09.17-000002                            rkm3gGXRQ9mt8p52XWG76A   1   1          0            0       249b           249b         249b

stephenb · October 2, 2025, 2:04pm

Hi @decibel83

"output":{"events":{"active":0,"batches":2,"dropped":175,"total":175},
"output":{"events":{"active":0,"batches":3,"dropped":146,"total":146}

Looks like you are perhaps dropping events.

Did you run

./filebeat setup -e

Can you run

GET _index_template/filebeat-8.19.3

Under the logs directory is there and events directory if so check in there.

decibel83 · October 2, 2025, 2:31pm

Yes, I ran it and ran it again right now.

Yes, I receive a 28000+ rows JSON output.

Ooooookay, maybe I got it:

{"log.level":"warn","@timestamp":"2025-10-02T16:24:50.001+0200","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).applyItemStatus","file.name":"elasticsearch/client.go","file.line":533},"message":"Cannot index event '{\"@timestamp\":\"2025-10-02T14:24:39.994Z\",\"event\":{\"dataset\":\"system.syslog\",\"module\":\"system\",\"timezone\":\"+02:00\"},\"fileset\":{\"name\":\"syslog\"},\"input\":{\"type\":\"log\"},\"ecs\":{\"version\":\"1.12.0\"},\"log\":{\"offset\":11806925633,\"file\":{\"path\":\"/var/log/syslog\"}},\"service\":{\"type\":\"system\"},\"host\":{\"os\":{\"platform\":\"debian\",\"version\":\"12 (bookworm)\",\"family\":\"debian\",\"name\":\"Debian GNU/Linux\",\"kernel\":\"6.1.0-38-amd64\",\"codename\":\"bookworm\",\"type\":\"linux\"},\"id\":\"3680414d6afd4c7a8b11cadfd31dfaac\",\"containerized\":false,\"ip\":[\"192.168.110.140\",\"fe80::8c2d:e9ff:fe57:aa37\",\"172.18.0.1\",\"fe80::8cb2:22ff:feb5:d201\",\"172.19.0.1\",\"fe80::b49b:efff:fef9:7a63\",\"172.240.0.1\",\"fe80::dcf5:4eff:fe77:9d0c\",\"fe80::9c13:1dff:fefe:e801\"],\"mac\":[\"8E-2D-E9-57-AA-37\",\"8E-B2-22-B5-D2-01\",\"9E-13-1D-FE-E8-01\",\"B6-9B-EF-F9-7A-63\",\"D6-DC-3A-7C-F6-22\",\"DE-F5-4E-77-9D-0C\"],\"name\":\"elk\",\"hostname\":\"elk\",\"architecture\":\"x86_64\"},\"agent\":{\"name\":\"elk\",\"type\":\"filebeat\",\"version\":\"8.19.3\",\"ephemeral_id\":\"ec7287ab-e926-4684-81d0-7805fd6fcad1\",\"id\":\"15b0aca0-3127-4275-94f9-a93f239d8696\"},\"message\":\"Oct  2 16:24:39 elk logstash[409844]: [2025-10-02T16:24:39,240][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]\"}\n, Meta: {\"pipeline\":\"filebeat-8.19.3-system-syslog-entrypoint\"}' (status=403): {\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/bulk[s]] is unauthorized for user [filebeat_setup] with effective roles [filebeat_setup,filebeat_writer,ingest_admin,kibana_admin] on indices [filebeat-8.19.3], this action is granted by the index privileges [create_doc,create,delete,index,write,all]\"}, dropping event!","service.name":"filebeat","log.type":"event","ecs.version":"1.6.0"}

I mistakenly created the filebeat_writer role: I fixed the permissions adding the create_doc permission to filebeat-*.

Thank you!

stephenb · October 2, 2025, 2:57pm

@decibel83 Cool glad you found it.

Yeah and the bummer about an error like that the filebeat test output will still connect but it doesn't actually try to write a document... so you need to check the event

Topic		Replies	Views
Filebeat - It is not creating index on Elasticsearch Beats filebeat	4	4533	July 5, 2017
Filebeat is not shipping the log data to Elasticsearch Beats filebeat	19	28524	April 10, 2018
Filebeat cannot able to index into elasticsearch Beats filebeat	27	6497	May 4, 2017
Filebeat is running on two different systems with two different modules but no documents appear in filebeat indx Beats	2	491	March 6, 2020
Elastic Search is receiving data from beat, but no documents appear Elasticsearch	15	1595	March 9, 2020