Short description:
I have a log input set to scan every 5 seconds. With full debug logs turned on I
see
2020-05-20T19:29:06.606Z DEBUG [input] log/input.go:191 Start next scan
Which discovers some files, but then I never see it again
ubuntu@ip-192-168-16-196:~$ docker logs f233 2>&1 | grep "scan"
2020-05-20T19:29:06.606Z DEBUG [input] log/input.go:191 Start next scan
Furthermore I never see the line "Run input" which should be printed here https://github.com/elastic/beats/blob/52fa26598cef8a90c8981970414c780e4fd31dbf/filebeat/input/input.go#L141
Indicating the loop isnt running
Not sure if I'm missing something but it seems like maybe theres a bug where the scan is stuck and not returning, so it never runs again.
More details:
All input logs
ubuntu@ip-192-168-16-196:~$ date
Wed May 20 19:43:37 UTC 2020
ubuntu@ip-192-168-16-196:~$ docker logs f233 2>&1 | grep "input"
2020-05-20T19:29:06.476Z DEBUG [input] log/config.go:204 recursive glob enabled
2020-05-20T19:29:06.476Z DEBUG [input] log/input.go:164 exclude_files: []. Number of states: 99
2020-05-20T19:29:06.476Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/10f8f0f3-a83f-955f-b68d-a90d6e222961/alloc/logs/browser.stdout.0
2020-05-20T19:29:06.476Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/88c2a7b3-cf94-be86-f259-6de0c244aa78/alloc/logs/filebeat.stderr.0
2020-05-20T19:29:06.476Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/94d697ef-ee5e-548b-a274-c2b852550694/alloc/logs/browser.stderr.0
2020-05-20T19:29:06.480Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/41d4d191-713a-d653-aff7-66a1c1855f22/alloc/logs/browser.stderr.0
2020-05-20T19:29:06.480Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/58d622fa-196c-3255-3bde-951fca64b15f/alloc/logs/filebeat.stderr.0
2020-05-20T19:29:06.483Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/c6d5b648-d40c-df30-a45e-b91d2e367e3a/alloc/logs/browser.stderr.0
2020-05-20T19:29:06.486Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/85ce842c-037a-48cf-7150-3b218d6c7e4e/alloc/logs/browser.stderr.0
2020-05-20T19:29:06.490Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/47d90b32-410a-e367-069b-5f1126651159/alloc/logs/browser.stdout.0
2020-05-20T19:29:06.493Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/4e47f814-df09-ce2d-4c53-2432ec235afc/alloc/logs/filebeat.stderr.0
2020-05-20T19:29:06.498Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/47af4bf0-557b-5ddf-148d-d0969900dfd2/alloc/logs/orchestrator.stdout.0
2020-05-20T19:29:06.502Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/55e0f41a-3258-2d07-02b5-6e234eb4bdfd/alloc/logs/browser.stderr.0
2020-05-20T19:29:06.505Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/f30588e4-096a-c7cb-d51f-6cdc76a217a1/alloc/logs/browser.stderr.0
2020-05-20T19:29:06.509Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/b5374f8f-9da5-2896-16a1-8bc1a0a9ac1c/alloc/logs/browser.stderr.0
2020-05-20T19:29:06.513Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/d01919b0-f3e5-31b0-0945-e11eea70a833/alloc/logs/browser.stderr.0
2020-05-20T19:29:06.516Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/0f5bc890-9f74-4595-5236-fa050692845e/alloc/logs/browser.stdout.0
2020-05-20T19:29:06.521Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/924c33b5-329d-5eb4-7487-1753a339d967/alloc/logs/browser.stdout.0
2020-05-20T19:29:06.525Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/309ca6f5-53f4-3572-28c8-f4cc46a3746a/alloc/logs/browser.stdout.0
2020-05-20T19:29:06.529Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/5d722831-7783-07bc-da5e-5ebc10398adc/alloc/logs/browser.stdout.0
2020-05-20T19:29:06.533Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/3d5e13ff-73d4-1901-2c00-43d64fd9e2c8/alloc/logs/filebeat.stdout.0
2020-05-20T19:29:06.538Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/1a6576c3-ae7d-cc97-f959-dfac7e1a3e1c/alloc/logs/orchestrator.stdout.0
2020-05-20T19:29:06.543Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/441383f3-611d-d574-1037-c0006b99c598/alloc/logs/browser.stdout.0
2020-05-20T19:29:06.548Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/8d1275c0-362e-fe06-7be2-58ff6a52a37a/alloc/logs/browser.stdout.0
2020-05-20T19:29:06.553Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/9efdb723-3770-cedf-dc3f-f14c4f9b30c7/alloc/logs/browser.stderr.0
2020-05-20T19:29:06.558Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/0d03482e-0246-8586-b6dc-2f1bcf1ecadc/alloc/logs/browser.stdout.0
2020-05-20T19:29:06.562Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/e7a7ee76-8c5f-a880-039c-8f7f3e4d263e/alloc/logs/browser.stdout.0
2020-05-20T19:29:06.566Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/23453bc0-84c0-571b-c4e1-ae52348cc207/alloc/logs/orchestrator.stdout.0
2020-05-20T19:29:06.571Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/5d722831-7783-07bc-da5e-5ebc10398adc/alloc/logs/browser.stderr.0
2020-05-20T19:29:06.575Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/810917a8-28b8-db76-5a49-b312ec653141/alloc/logs/browser.stdout.0
2020-05-20T19:29:06.580Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/1098a232-9abd-fcc9-8607-04c4da3855b2/alloc/logs/orchestrator.stdout.0
2020-05-20T19:29:06.585Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/700e2d2c-0fd4-841f-08ff-e27fd335790b/alloc/logs/orchestrator.stdout.0
2020-05-20T19:29:06.589Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/95f34a81-9283-0e67-9228-ba9ff4dd3048/alloc/logs/test2.stderr.0
2020-05-20T19:29:06.594Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/9674274e-db84-caa7-8dc9-c86a4bae9ce2/alloc/logs/filebeat.stdout.0
2020-05-20T19:29:06.598Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/0687b2a0-2846-4c06-6cb0-865c66bfba2f/alloc/logs/browser.stdout.0
2020-05-20T19:29:06.602Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/5b7d4852-c981-5904-9428-e86e591cfc66/alloc/logs/browser.stdout.0
2020-05-20T19:29:06.606Z DEBUG [input] log/input.go:185 input with previous states loaded: 34
2020-05-20T19:29:06.606Z INFO log/input.go:152 Configured paths: [/opt/nomad/data/alloc/*/alloc/logs/*]
2020-05-20T19:29:06.606Z INFO input/input.go:114 Starting input of type: log; ID: 6359703886156017179
2020-05-20T19:29:06.606Z INFO beater/crawler.go:105 Loading and starting Inputs completed. Enabled inputs: 1
2020-05-20T19:29:06.606Z DEBUG [input] log/input.go:191 Start next scan
2020-05-20T19:29:06.610Z DEBUG [input] log/input.go:421 Check file for harvesting: /opt/nomad/data/alloc/673b8a8e-4d96-ce56-3979-ae34831c6d2c/alloc/logs/filebeat.stdout.0
2020-05-20T19:29:06.610Z DEBUG [input] log/input.go:494 Start harvester for new file: /opt/nomad/data/alloc/673b8a8e-4d96-ce56-3979-ae34831c6d2c/alloc/logs/filebeat.stdout.0
2020-05-20T19:29:06.613Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/673b8a8e-4d96-ce56-3979-ae34831c6d2c/alloc/logs/filebeat.stdout.0
2020-05-20T19:29:06.613Z DEBUG [input] log/input.go:421 Check file for harvesting: /opt/nomad/data/alloc/95f34a81-9283-0e67-9228-ba9ff4dd3048/alloc/logs/test2.stderr.0
2020-05-20T19:29:06.613Z DEBUG [input] log/input.go:511 Update existing file for harvesting: /opt/nomad/data/alloc/95f34a81-9283-0e67-9228-ba9ff4dd3048/alloc/logs/test2.stderr.0, offset: 2
2020-05-20T19:29:06.613Z DEBUG [input] log/input.go:565 File didn't change: /opt/nomad/data/alloc/95f34a81-9283-0e67-9228-ba9ff4dd3048/alloc/logs/test2.stderr.0
2020-05-20T19:29:06.613Z DEBUG [input] log/input.go:421 Check file for harvesting: /opt/nomad/data/alloc/bff55475-edc1-2eee-9495-3aa4687d90a5/alloc/logs/.browser.stderr.fifo
2020-05-20T19:29:06.613Z DEBUG [input] log/input.go:494 Start harvester for new file: /opt/nomad/data/alloc/bff55475-edc1-2eee-9495-3aa4687d90a5/alloc/logs/.browser.stderr.fifo
2020-05-20T19:29:06.626Z DEBUG [input] file/states.go:68 New state added for /opt/nomad/data/alloc/673b8a8e-4d96-ce56-3979-ae34831c6d2c/alloc/logs/filebeat.stdout.0
My config:
ubuntu@ip-192-168-16-196:~$ docker exec f233a cat /local/config/filebeat.yml
setup.ilm:
# TODO: Enable once bug is fixed; in the meantime policy must be created manually
# https://github.com/elastic/beats/issues/11866
enabled: false
policy_name: filebeat-2
setup.template:
enabled: true
overwrite: false
json:
enabled: true
name: "filebeat-2"
name: filebeat-2
pattern: "filebeat-2-*"
logging.level: debug
filebeat.inputs:
- type: log
fields:
source: nomad-task
paths:
- /opt/nomad/data/alloc/*/alloc/logs/*
scan_frequency: 5s
processors:
- add_cloud_metadata: ~
processors:
- drop_fields:
fields:
[
"agent.id",
"agent.ephemeral_id",
"agent.type",
"ecs.version",
"agent.hostname",
"docker.container.labels.com_amazonaws_ecs_task-definition-family",
"docker.container.labels.com_amazonaws_ecs_task-arn",
"docker.container.labels.org_label-schema_build-date"
]
- add_fields:
target: ""
fields:
environment: "bbuzbee"
region: "us-west-2"
nomad_pool: "browsers"
output.elasticsearch:
hosts:
[
"<REDACTED>",
]
index: "<REDACTED>"
username: "<REDACTED>"
password: ""