Hi All,
I am using ELK stack 7.6.2. The log does not get pushed from Filebeat to Logstash for some reason. The log /opt/ttcmkts/work/bcdcommoncache/bcd.log is rolling very fast.
My logstash conf file looks like below:
filebeat.inputs:
- type: log
paths:
- /opt/ttcmkts/work/bcdcommoncache/bcd.log
fields:
type: pd_ttc_cva
fields_under_root: true
filebeat.registry.path: /opt/ttcmkts/elkagent/applications/filebeat/data/registry
setup.dashboards.enabled: true
# Change to true to enable this input configuration.
enabled: true
multiline.pattern: '^\[[0-9]{4} [a-zA-Z]{3} [0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}\]'
multiline.negate: true
multiline.match: after
output.logstash:
hosts: ["xxxxxxx:5045","xxxxxxxx:5045","xxxxxxxx:5045","xxxxxxxx:5045","xxxxxxxx:5045"]
loadbalance: true
index: filebeat
logging.level: debug
logging.to_files: true
logging.files:
path: /opt/ttcmkts/elkagent/log/
name: filebeat_pd_debug.log
keepfiles: 70
permissions: 0644
Contents of the log are as follows. Please note offset 0 :
2021-05-08T22:53:03.533+0100 INFO instance/beat.go:622 Home path: [/opt/ttcmkts/elkagent/applications/filebeat] Config path: [/opt/ttcmkts/elkagent/applications/filebeat] Data path: [/opt/ttcmkts/elkagent/applications/filebeat/data] Logs path: [/opt/ttcmkts/elkagent/applications/filebeat/logs]
2021-05-08T22:53:03.533+0100 DEBUG [beat] instance/beat.go:674 Beat metadata path: /opt/ttcmkts/elkagent/applications/filebeat/data/meta.json
2021-05-08T22:53:03.535+0100 INFO instance/beat.go:630 Beat ID: 7dc47baf-399b-4199-99b3-5b1ba180890a
2021-05-08T22:53:03.535+0100 INFO [seccomp] seccomp/seccomp.go:101 Syscall filter could not be installed because the kernel does not support seccomp
2021-05-08T22:53:03.535+0100 INFO [beat] instance/beat.go:958 Beat info {"system_info": {"beat": {"path": {"config": "/opt/ttcmkts/elkagent/applications/filebeat", "data": "/opt/ttcmkts/elkagent/applications/filebeat/data", "home": "/opt/ttcmkts/elkagent/applications/filebeat", "logs": "/opt/ttcmkts/elkagent/applications/filebeat/logs"}, "type": "filebeat", "uuid": "7dc47baf-399b-4199-99b3-5b1ba180890a"}}}
2021-05-08T22:53:03.535+0100 INFO [beat] instance/beat.go:967 Build info {"system_info": {"build": {"commit": "d57bcf8684602e15000d65b75afcd110e2b12b59", "libbeat": "7.6.2", "time": "2020-03-26T05:23:38.000Z", "version": "7.6.2"}}}
2021-05-08T22:53:03.535+0100 INFO [beat] instance/beat.go:970 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":24,"version":"go1.13.8"}}}
2021-05-08T22:53:03.538+0100 INFO [beat] instance/beat.go:974 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-05-02T00:06:14+01:00","containerized":false,"name":"becda2c","ip":["127.0.0.1/8","33.44.44.55/24"],"kernel_version":"2.6.32-754.36.1.el6.x86_64","mac":["3c:4a:92:73:29:08","3c:4a:92:73:29:0a","3c:4a:92:73:29:08","3c:4a:92:73:29:0e","3c:4a:92:73:29:08"],"os":{"family":"redhat","platform":"redhat","name":"Red","version":"6.10 (Santiago)","major":6,"minor":10,"patch":0,"codename":"Santiago"},"timezone":"BST","timezone_offset_sec":3600,"id":"083fb79105c1d1506ce8ddcd00000012"}}}
2021-05-08T22:53:03.539+0100 INFO [beat] instance/beat.go:1003 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63"],"ambient":null}, "cwd": "/opt/ttcmkts/elkagent/applications/filebeat-7.6.2-linux-x86_64", "exe": "/opt/ttcmkts/elkagent/applications/filebeat-7.6.2-linux-x86_64/filebeat", "name": "filebeat", "pid": 8042, "ppid": 1, "seccomp": {"mode":""}, "start_time": "2021-05-08T22:53:02.880+0100"}}}
2021-05-08T22:53:03.539+0100 INFO instance/beat.go:298 Setup Beat: filebeat; Version: 7.6.2
2021-05-08T22:53:03.539+0100 DEBUG [beat] instance/beat.go:324 Initializing output plugins
2021-05-08T22:53:03.540+0100 DEBUG [publisher] pipeline/consumer.go:137 start pipeline event consumer
2021-05-08T22:53:03.540+0100 INFO [publisher] pipeline/module.go:110 Beat name: becda2c
2021-05-08T22:53:03.541+0100 WARN beater/filebeat.go:152 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-05-08T22:53:03.542+0100 INFO instance/beat.go:439 filebeat start running.
2021-05-08T22:53:03.542+0100 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2021-05-08T22:53:03.542+0100 DEBUG [test] registrar/migrate.go:159 isFile(/opt/ttcmkts/elkagent/applications/filebeat/data/registry) -> false
2021-05-08T22:53:03.542+0100 DEBUG [test] registrar/migrate.go:159 isFile() -> false
2021-05-08T22:53:03.542+0100 DEBUG [test] registrar/migrate.go:152 isDir(/opt/ttcmkts/elkagent/applications/filebeat/data/registry/filebeat) -> false
2021-05-08T22:53:03.542+0100 DEBUG [registrar] registrar/migrate.go:51 Registry type '' found
2021-05-08T22:53:03.542+0100 DEBUG [test] registrar/migrate.go:159 isFile(.bak) -> false
2021-05-08T22:53:03.542+0100 DEBUG [test] registrar/migrate.go:152 isDir(/opt/ttcmkts/elkagent/applications/filebeat/data/registry/filebeat) -> false
2021-05-08T22:53:03.542+0100 INFO registrar/migrate.go:104 No registry home found. Create: /opt/ttcmkts/elkagent/applications/filebeat/data/registry/filebeat
2021-05-08T22:53:03.542+0100 DEBUG [test] registrar/migrate.go:159 isFile(/opt/ttcmkts/elkagent/applications/filebeat/data/registry/filebeat/meta.json) -> false
2021-05-08T22:53:03.542+0100 INFO registrar/migrate.go:112 Initialize registry meta file
2021-05-08T22:53:03.543+0100 INFO registrar/registrar.go:108 No registry file found under: /opt/ttcmkts/elkagent/applications/filebeat/data/registry/filebeat/data.json. Creating a new registry file.
2021-05-08T22:53:03.543+0100 DEBUG [registrar] registrar/registrar.go:411 Write registry file: /opt/ttcmkts/elkagent/applications/filebeat/data/registry/filebeat/data.json (0)
2021-05-08T22:53:03.544+0100 DEBUG [registrar] registrar/registrar.go:404 Registry file updated. 0 states written.
2021-05-08T22:53:03.544+0100 INFO registrar/registrar.go:145 Loading registrar data from /opt/ttcmkts/elkagent/applications/filebeat/data/registry/filebeat/data.json
2021-05-08T22:53:03.544+0100 INFO registrar/registrar.go:152 States Loaded from registrar: 0
2021-05-08T22:53:03.544+0100 WARN beater/filebeat.go:368 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-05-08T22:53:03.544+0100 INFO crawler/crawler.go:72 Loading Inputs: 1
2021-05-08T22:53:03.544+0100 DEBUG [registrar] registrar/registrar.go:278 Starting Registrar
2021-05-08T22:53:03.545+0100 DEBUG [input] log/config.go:204 recursive glob enabled
2021-05-08T22:53:03.545+0100 DEBUG [input] log/input.go:164 exclude_files: []. Number of stats: 0
2021-05-08T22:53:03.545+0100 DEBUG [input] log/input.go:185 input with previous states loaded: 0
2021-05-08T22:53:03.546+0100 INFO log/input.go:152 Configured paths: [/opt/ttcmkts/work/bcdcommoncache/bcd.log]
2021-05-08T22:53:03.546+0100 INFO input/input.go:114 Starting input of type: log; ID: 14857729624588492233
2021-05-08T22:53:03.546+0100 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2021-05-08T22:53:03.546+0100 DEBUG [input] log/input.go:191 Start next scan
2021-05-08T22:53:03.546+0100 DEBUG [input] log/input.go:421 Check file for harvesting: /opt/ttcmkts/work/bcdcommoncache/bcd.log
2021-05-08T22:53:03.546+0100 DEBUG [input] log/input.go:494 Start harvester for new file: /opt/ttcmkts/work/bcdcommoncache/bcd.log
2021-05-08T22:53:03.546+0100 DEBUG [harvester] log/harvester.go:573 Setting offset for file based on seek: /opt/ttcmkts/work/bcdcommoncache/bcd.log
2021-05-08T22:53:03.546+0100 DEBUG [harvester] log/harvester.go:559 Setting offset for file: /opt/ttcmkts/work/bcdcommoncache/bcd.log. Offset: 0
2021-05-08T22:53:03.547+0100 DEBUG [harvester] log/harvester.go:205 Harvester setup successful. Line terminator: 1
2021-05-08T22:53:03.547+0100 DEBUG [publisher] pipeline/client.go:220 Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id: Finished:false Fileinfo:0xc0006e6680 Source:/opt/ttcmkts/work/bcdcommoncache/bcd.log Offset:0 Timestamp:2021-05-08 22:53:03.546474215 +0100 BST m=+0.061908536 TTL:-1ns Type:log Meta:map[] FileStateOS:13763972-64785} TimeSeries:false}
2021-05-08T22:53:03.547+0100 DEBUG [acker] beater/acker.go:64 stateful ack {"count": 1}
2021-05-08T22:53:03.547+0100 DEBUG [harvester] log/harvester.go:478 Update state: /opt/ttcmkts/work/bcdcommoncache/bcd.log, offset: 0
2021-05-08T22:53:03.547+0100 DEBUG [registrar] registrar/registrar.go:356 Processing 1 events
2021-05-08T22:53:03.547+0100 DEBUG [input] file/states.go:68 New state added for /opt/ttcmkts/work/bcdcommoncache/bcd.log
2021-05-08T22:53:03.547+0100 DEBUG [input] file/states.go:68 New state added for /opt/ttcmkts/work/bcdcommoncache/bcd.log
2021-05-08T22:53:03.547+0100 DEBUG [registrar] registrar/registrar.go:326 Registrar state updates processed. Count: 1
2021-05-08T22:53:03.547+0100 DEBUG [registrar] registrar/registrar.go:411 Write registry file: /opt/ttcmkts/elkagent/applications/filebeat/data/registry/filebeat/data.json (1)
2021-05-08T22:53:03.547+0100 INFO log/harvester.go:297 Harvester started for file: /opt/ttcmkts/work/bcdcommoncache/bcd.log
2021-05-08T22:53:03.547+0100 DEBUG [input] log/input.go:212 input states cleaned up. Before: 1, After: 1, Pending: 0
2021-05-08T22:53:03.563+0100 DEBUG [registrar] registrar/registrar.go:404 Registry file updated. 1 states written.
2021-05-08T22:53:04.587+0100 DEBUG [harvester] log/log.go:107 End of file reached: /opt/ttcmkts/work/bcdcommoncache/bcd.log; Backoff now.
2021-05-08T22:53:10.590+0100 DEBUG [harvester] log/log.go:107 End of file reached: /opt/ttcmkts/work/bcdcommoncache/bcd.log; Backoff now.
2021-05-08T22:53:11.590+0100 DEBUG [harvester] log/log.go:107 End of file reached: /opt/ttcmkts/work/bcdcommoncache/bcd.log; Backoff now.
2021-05-08T22:53:12.591+0100 DEBUG [harvester] log/log.go:107 End of file reached: /opt/ttcmkts/work/bcdcommoncache/bcd.log; Backoff now.
2021-05-08T22:53:13.547+0100 DEBUG [input] input/input.go:152 Run input
2021-05-08T22:53:13.547+0100 DEBUG [input] log/input.go:191 Start next scan
2021-05-08T22:53:13.548+0100 DEBUG [input] log/input.go:421 Check file for harvesting: /opt/ttcmkts/work/bcdcommoncache/bcd.log
2021-05-08T22:53:13.548+0100 DEBUG [input] log/input.go:511 Update existing file for harvesting: /opt/ttcmkts/work/bcdcommoncache/bcd.log, offset: 0
2021-05-08T22:53:13.548+0100 DEBUG [input] log/input.go:563 Harvester for file is still running: /opt/ttcmkts/work/bcdcommoncache/bcd.log
2021-05-08T22:53:13.548+0100 DEBUG [input] log/input.go:212 input states cleaned up. Before: 1, After: 1, Pending: 0
2021-05-08T22:53:13.591+0100 DEBUG [harvester] log/log.go:107 End of file reached: /opt/ttcmkts/work/bcdcommoncache/bcd.log; Backoff now.
2021-05-08T22:53:22.596+0100 DEBUG [harvester] log/log.go:107 End of file reached: /opt/ttcmkts/work/bcdcommoncache/bcd.log; Backoff now.
Thanks. Please guide