# Loss of Logs occurred while migrating from log input to filestream input
**URL:** https://discuss.elastic.co/t/loss-of-logs-occurred-while-migrating-from-log-input-to-filestream-input/370532
**Category:** Beats
**Tags:** docker, filebeat
**Created:** [November 14, 2024, 9:31am UTC](https://discuss.elastic.co/t/loss-of-logs-occurred-while-migrating-from-log-input-to-filestream-input/370532 "2024-11-14T09:31:33Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@koti903281](https://discuss.elastic.co/u/koti903281)
#### Post date: [November 14, 2024, 9:31am UTC](https://discuss.elastic.co/t/loss-of-logs-occurred-while-migrating-from-log-input-to-filestream-input/370532/1 "2024-11-14T09:31:33Z")
Hi All
1.We are trying to migrate from existing log input to filestream input in filebeat for Autodiscover disable case.
2.We are using the filebeat,logstash and opensearch as 3pp in our micro-services.
3.This is the template file generated at the time log input
```auto
data:
filebeat.yml: |
filebeat.inputs:
- type: log
paths:
- /var/lib/docker.log
- /var/log/pods.log
fields:
logplane: "adp-app-logs"
fields_under_root: true
close_timeout: "5m"
processors:
target_prefix: "kubernetes"
ignore_failure: true
- drop_fields:
fields:
- "kubernetes.log.file.name"
ignore_missing: true
output.logstash:
hosts: "lt:1234"
ssl.certificate_authorities: "ca.crt"
ssl.certificate: "{CERT}"
ssl.key: "${KEY}"
ssl.verification_mode: "full"
ssl.renegotiation: "freely"
ssl.supported_protocols: ["TLSv1.2", "TLSv1.3"]
ssl.cipher_suites: []
bulk_max_size: 2048
worker: 1
pipelining: 0
ttl: 30
queue.mem:
flush.timeout: 1s
filebeat.registry.flush: 5s
logging.level: "info"
logging.metrics.enabled: false
http.enabled: true
http.host: localhost
http.port: 1234
```
4.This is the template file generated at the time filestream input
```auto
data:
filebeat.yml: |
filebeat.inputs:
- type: filestream
paths:
- /var/lib/docker.log
- /var/log/pods.log
fields:
logplane: "adp-app-logs"
id: my_ids_1
take_over: true
enabled: true
fields_under_root: true
close_timeout: "5m"
processors:
target_prefix: "kubernetes"
ignore_failure: true
- drop_fields:
fields:
- "kubernetes.log.file.name"
ignore_missing: true
output.logstash:
hosts: "lt:1234"
ssl.certificate_authorities: "ca.crt"
ssl.certificate: "${CERT}"
ssl.key: "${KEY}"
ssl.verification_mode: "full"
ssl.renegotiation: "freely"
ssl.supported_protocols: ["TLSv1.2", "TLSv1.3"]
ssl.cipher_suites: []
bulk_max_size: 2048
worker: 1
pipelining: 0
ttl: 30
queue.mem:
flush.timeout: 1s
filebeat.registry.flush: 5s
logging.level: "info"
logging.metrics.enabled: false
http.enabled: true
http.host: localhost
http.port: 1234
```
I have tested many scenarios like sending logs to opensearch and please find the test results below for the better understanding
| input Type | Total logs sending from log producer | logs per sec | total duartion of sending logs | Logs Missing (Yes/No) | Loss of Logs Count |
| --- | --- | --- | --- | --- | --- |
| filestream | 10 | 1 | 10s | No | 0 |
| filestream | 100 | 1 | 100s | No | 0 |
| filestream | 1,000 | 10 | 100s | No | 0 |
| filestream | 36,000 | 100 | 6 mins | No | 0 |
| filestream | 90,000 | 100 | 15 mins | No | 0 |
| filestream | 1,80,000 | 100 | 30 mins | No | 0 |
| filestream | 3,60,000 | 200 | 30 mins | Yes | 2,08,640 |
| filestream | 3,60,000 | 200 | 30 mins | Yes | 50,390 |
| filestream | 2,70,000 | 150 | 30 mins | Yes | 75,984 |
We would like to know why the loss of logs are happening like so many of the logs are not reaching to opensearch via logstash?
If you need any stats I am open to provide or any other info you need 🙂
---
### Author:  [@sasikiranvaddi](https://discuss.elastic.co/u/sasikiranvaddi)
#### Post date: [April 3, 2025, 10:04am UTC](https://discuss.elastic.co/t/loss-of-logs-occurred-while-migrating-from-log-input-to-filestream-input/370532/2 "2025-04-03T10:04:47Z")
A bug has been created.
We could we there is performance bottleneck in harvesting logs of filebeat with input type as filestream compared to input type log.
More detailed information has been updated in the below bug reference.
>
>
> We are use filebeat version 8.16.3, performing tests with 1500 logs/second for a…bout 15 minutes. We could observe some of the logs that are not getting harvested.
> i.e., 1350000 supposed to be sent to respective destination but around 1250000 to 1320000 on repeated tests.
>
> When we analyze the metrics, the logs that are harvested by input type filestream are sent to destination(i.e., logstash). We supsect there is some issue in harvesting the rest of logs.
> Similar tests are performed with input type as log and we did not see any loss of logs.
>
>
> filebeat.yml
> \`filebeat.inputs:
> \- type: filestream
> paths:
> - \
> fields:
> logplane: "app-logs"
> id: my\_filestream\_id\_1
> ignore\_inactive: since\_last\_start
> close.reader.after\_interval: "5m"
> prospector.scanner.check\_interval: 1s
> fields\_under\_root: true
> processors:
> - dissect:
> when:
> regexp:
> log.file.path: '^/var/log/pods/\[^/\]+\_\[^/\]+\_\[^/\]+/\[^/\]+/\[^/\]+$'
> tokenizer: "/var/log/pods/%{namespace}\_%{pod.name}\_%{pod.uid}/%{container.name}/%{log.file.name}"
> field: "log.file.path"
> target\_prefix: "kubernetes"
> ignore\_failure: true
> - drop\_fields:
> fields:
> - "kubernetes.log.file.name"
> ignore\_missing: true
> output.logstash:
> hosts: "logstash:5044"
> ssl.certificate\_authorities: \
> ssl.certificate: \
> ssl.key: \
> ssl.verification\_mode: "full"
> ssl.renegotiation: "freely"
> ssl.supported\_protocols: \["TLSv1.2", "TLSv1.3"\]
> ssl.cipher\_suites: \[\]
> worker: 2
> ttl: 30
> queue.mem:
> flush.timeout: 1s
> filebeat.registry.flush: 5s
> logging.level: "debug"
> logging.selectors: \["harvester", "registrar"\]
> logging.metrics.enabled: true
> http.enabled: true
> http.host: localhost
> http.port: 5066
> \`
> For confirmed bugs, please report:
> \- Version: 8.16.3
> \- Operating System: K8S, SUSE based container.
> \- Discuss Forum URL: https://discuss.elastic.co/t/loss-of-logs-occurred-while-migrating-from-log-input-to-filestream-input/370532
> \- Steps to Reproduce: