Hi, I have configured filebeat to send a file to elastic but it is not happening, for some reason which I can not discover is not reading the file.
This is my config:
logging.level: debug
logging.metrics.period: 1m
logging.to_files: true
logging.files:
rotateeverybytes: 16777216
keepfiles: 7
permissions: 0600
filebeat.inputs:
- type: log
enabled: true
ignore_older: 48h
paths:
- "/var/log/loyalty/batch_caducidad.log"
output.elasticsearch:
hosts: ["elastic:9200"]
index: "log_stage_loyalty_7p-%{+yyyy.MM.dd}"
setup.ilm.enabled: false
setup.template.name: "log_stage_loyalty_7p"
setup.template.pattern: "log_stage_loyalty_7p-*"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 3
index.number_of_replicas: 1
Here the log:
2022-05-24T23:41:22.746Z INFO instance/beat.go:304 Setup Beat: filebeat; Version: 7.11.0
2022-05-24T23:41:22.746Z DEBUG [beat] instance/beat.go:330 Initializing output plugins
2022-05-24T23:41:22.746Z INFO eslegclient/connection.go:99 elasticsearch url: https://elb-elastic.prod:9202
2022-05-24T23:41:22.746Z WARN [tls] tlscommon/tls_config.go:93 SSL/TLS verifications disabled.
2022-05-24T23:41:22.747Z DEBUG [publisher] pipeline/consumer.go:148 start pipeline event consumer
2022-05-24T23:41:22.747Z INFO [publisher] pipeline/module.go:113 Beat name: ip-172-31-29-43
2022-05-24T23:41:22.747Z INFO instance/beat.go:468 filebeat start running.
2022-05-24T23:41:22.747Z INFO [monitoring] log/log.go:117 Starting metrics logging every 1m0s
2022-05-24T23:41:22.747Z DEBUG [test] registrar/migrate.go:304 isFile(/var/lib/filebeat/registry) -> false
2022-05-24T23:41:22.747Z DEBUG [test] registrar/migrate.go:304 isFile() -> false
2022-05-24T23:41:22.747Z DEBUG [test] registrar/migrate.go:297 isDir(/var/lib/filebeat/registry/filebeat) -> true
2022-05-24T23:41:22.747Z DEBUG [test] registrar/migrate.go:304 isFile(/var/lib/filebeat/registry/filebeat/meta.json) -> true
2022-05-24T23:41:22.747Z DEBUG [registrar] registrar/migrate.go:84 Registry type '1' found
2022-05-24T23:41:22.747Z INFO memlog/store.go:119 Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transac
tion id=68635
2022-05-24T23:41:22.842Z INFO memlog/store.go:124 Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Act
ive transaction id=75673
2022-05-24T23:41:22.842Z DEBUG [httpjson.transforms] v2/transform_registry.go:75 Register transform request:append
2022-05-24T23:41:22.842Z DEBUG [httpjson.transforms] v2/transform_registry.go:75 Register transform request:delete
2022-05-24T23:41:22.842Z DEBUG [httpjson.transforms] v2/transform_registry.go:75 Register transform request:set
2022-05-24T23:41:22.842Z DEBUG [httpjson.transforms] v2/transform_registry.go:75 Register transform response:append
2022-05-24T23:41:22.842Z DEBUG [httpjson.transforms] v2/transform_registry.go:75 Register transform response:delete
2022-05-24T23:41:22.842Z DEBUG [httpjson.transforms] v2/transform_registry.go:75 Register transform response:set
2022-05-24T23:41:22.842Z DEBUG [httpjson.transforms] v2/transform_registry.go:75 Register transform pagination:append
2022-05-24T23:41:22.842Z DEBUG [httpjson.transforms] v2/transform_registry.go:75 Register transform pagination:delete
2022-05-24T23:41:22.842Z DEBUG [httpjson.transforms] v2/transform_registry.go:75 Register transform pagination:set
2022-05-24T23:41:22.843Z INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 17
2022-05-24T23:41:22.843Z INFO [crawler] beater/crawler.go:71 Loading Inputs: 1
2022-05-24T23:41:22.843Z DEBUG [input] log/config.go:207 recursive glob enabled
2022-05-24T23:41:22.843Z DEBUG [registrar] registrar/registrar.go:140 Starting Registrar
2022-05-24T23:41:22.843Z DEBUG [input] log/input.go:169 exclude_files: []. Number of states: 17
2022-05-24T23:41:22.843Z DEBUG [input] file/states.go:68 New state added for /var/log/loyalty/batch_caducidad.log
2022-05-24T23:41:22.843Z DEBUG [acker] beater/acker.go:59 stateful ack {"count": 1}
2022-05-24T23:41:22.843Z DEBUG [publisher] pipeline/client.go:231 Pipeline client receives callback 'onFilteredOut' for event: {Timest
amp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::393338-51714 PrevId: Finished:true Fileinfo:<nil> Source:/var/log/loyalt
y/batch_caducidad.log Offset:112319 Timestamp:2022-05-24 23:30:57.945518309 +0000 UTC TTL:-1ns Type:log Meta:map[] FileStateOS:393338-51714 Identifi
erName:native} TimeSeries:false}
2022-05-24T23:41:22.843Z DEBUG [input] log/input.go:199 input with previous states loaded: 1
2022-05-24T23:41:22.843Z INFO log/input.go:157 Configured paths: [/var/log/loyalty/batch_caducidad.log]
2022-05-24T23:41:22.843Z INFO [crawler] beater/crawler.go:141 Starting input (ID: 4795600949698172230)
2022-05-24T23:41:22.843Z INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 1
2022-05-24T23:41:22.843Z DEBUG [registrar] registrar/registrar.go:264 Processing 1 events
2022-05-24T23:41:22.843Z DEBUG [registrar] registrar/registrar.go:231 Registrar state updates processed. Count: 1
2022-05-24T23:41:22.843Z DEBUG [registrar] registrar/registrar.go:254 Registrar states cleaned up. Before: 17, After: 17, Pending:
0
2022-05-24T23:41:22.843Z DEBUG [input] log/input.go:205 Start next scan
2022-05-24T23:41:22.843Z DEBUG [registrar] registrar/registrar.go:201 Registry file updated. 17 active states.
2022-05-24T23:41:22.843Z DEBUG [input] log/input.go:439 Check file for harvesting: /var/log/loyalty/batch_caducidad.log
2022-05-24T23:41:22.843Z DEBUG [input] log/input.go:530 Update existing file for harvesting: /var/log/loyalty/batch_caducidad.log, o
ffset: 112319
2022-05-24T23:41:22.843Z DEBUG [input] log/input.go:585 File didn't change: /var/log/loyalty/batch_caducidad.log
2022-05-24T23:41:22.843Z DEBUG [input] log/input.go:226 input states cleaned up. Before: 1, After: 1, Pending: 0
I tried using the ignore_older parameter but it does not read the file either.
Any ideas?