Hi all.
I have a problem with beats (winlogbeat, filebeat) on Windows Server 2016. I use filebeat to collect IIS logs.
Server information:
- OS Name: Microsoft Windows Server 2016 Standard
- OS Version: 10.0.14393 N/A Build 14393
Issue:
- When I run service Filebeat or command: "filebeat.exe -e", Filebeat does ship log to Logstash. After some time (30s, ~1m), Filebeat has hung.
I tried to enable DEBUG but couldn't find any useful information. - Winlogbeat has the same issue
Beats version:
- 7.17.1
My config modules IIS:
- module: iis
access:
enabled: true
input:
ignore_older: 24h
error:
enabled: true
input:
ignore_older: 24h
Some debug info:
2022-03-25T10:17:49.938+0700 DEBUG [input] input/input.go:139 Run input
2022-03-25T10:17:49.938+0700 DEBUG [input] log/input.go:222 Start next scan {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.113+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC2\u_ex200409.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.113+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC2\u_ex200409.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC2\\u_ex200409.log", "state_id": "native::3932160-279913-3099028754", "finished": false, "os_id": "3932160-279913-3099028754"}
2022-03-25T10:17:50.119+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC2\u_ex201116.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.120+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC2\u_ex201116.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC2\\u_ex201116.log", "state_id": "native::11927552-151835-3099028754", "finished": false, "os_id": "11927552-151835-3099028754"}
2022-03-25T10:17:50.120+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC4\u_ex191215.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.120+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC4\u_ex191215.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC4\\u_ex191215.log", "state_id": "native::1638400-126525-3099028754", "finished": false, "os_id": "1638400-126525-3099028754"}
2022-03-25T10:17:50.120+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC4\u_ex200514.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.120+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC4\u_ex200514.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC4\\u_ex200514.log", "state_id": "native::6029312-115391-3099028754", "finished": false, "os_id": "6029312-115391-3099028754"}
2022-03-25T10:17:50.121+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC3\u_ex201001.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.121+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC3\u_ex201001.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC3\\u_ex201001.log", "state_id": "native::524288-24919-3099028754", "finished": false, "os_id": "524288-24919-3099028754"}
2022-03-25T10:17:50.121+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC3\u_ex211104.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.121+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC3\u_ex211104.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC3\\u_ex211104.log", "state_id": "native::17760256-241851-3099028754", "finished": false, "os_id": "17760256-241851-3099028754"}
2022-03-25T10:17:50.121+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC2\u_ex200208.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.121+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC2\u_ex200208.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC2\\u_ex200208.log", "state_id": "native::2621440-180627-3099028754", "finished": false, "os_id": "2621440-180627-3099028754"}
2022-03-25T10:17:50.122+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC2\u_ex200621.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.122+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC2\u_ex200621.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC2\\u_ex200621.log", "state_id": "native::1900544-140188-3099028754", "finished": false, "os_id": "1900544-140188-3099028754"}
2022-03-25T10:17:50.122+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC2\u_ex210831.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.122+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC2\u_ex210831.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC2\\u_ex210831.log", "state_id": "native::18939904-141421-3099028754", "finished": false, "os_id": "18939904-141421-3099028754"}
2022-03-25T10:17:50.122+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC2\u_ex211001.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.122+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC2\u_ex211001.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC2\\u_ex211001.log", "state_id": "native::13303808-52289-3099028754", "finished": false, "os_id": "13303808-52289-3099028754"}
2022-03-25T10:17:50.123+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC3\u_ex201026.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.123+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC3\u_ex201026.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC3\\u_ex201026.log", "state_id": "native::11534336-142452-3099028754", "finished": false, "os_id": "11534336-142452-3099028754"}
2022-03-25T10:17:50.123+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC3\u_ex201126.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.123+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC3\u_ex201126.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC3\\u_ex201126.log", "state_id": "native::2162688-153053-3099028754", "finished": false, "os_id": "2162688-153053-3099028754"}
2022-03-25T10:17:50.123+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC3\u_ex210610.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.123+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC3\u_ex210610.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC3\\u_ex210610.log", "state_id": "native::5832704-135631-3099028754", "finished": false, "os_id": "5832704-135631-3099028754"}
2022-03-25T10:17:50.124+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC4\u_ex200112.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.124+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC4\u_ex200112.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC4\\u_ex200112.log", "state_id": "native::2162688-110600-3099028754", "finished": false, "os_id": "2162688-110600-3099028754"}
2022-03-25T10:17:50.124+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC4\u_ex200412.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.124+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC4\u_ex200412.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC4\\u_ex200412.log", "state_id": "native::5570560-267597-3099028754", "finished": false, "os_id": "5570560-267597-3099028754"}
2022-03-25T10:17:50.124+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC3\u_ex210614.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.124+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC3\u_ex210614.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC3\\u_ex210614.log", "state_id": "native::17367040-167089-3099028754", "finished": false, "os_id": "17367040-167089-3099028754"}
2022-03-25T10:17:50.124+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC3\u_ex220129.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.125+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC3\u_ex220129.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC3\\u_ex220129.log", "state_id": "native::45023232-165303-3099028754", "finished": false, "os_id": "45023232-165303-3099028754"}
2022-03-25T10:17:50.125+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC4\u_ex200223.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.125+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC4\u_ex200223.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC4\\u_ex200223.log", "state_id": "native::7274496-275169-3099028754", "finished": false, "os_id": "7274496-275169-3099028754"}
2022-03-25T10:17:50.125+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC4\u_ex200315.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.125+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC4\u_ex200315.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC4\\u_ex200315.log", "state_id": "native::3801088-269270-3099028754", "finished": false, "os_id": "3801088-269270-3099028754"}
2022-03-25T10:17:50.125+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC4\u_ex200325.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.126+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC4\u_ex200325.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC4\\u_ex200325.log", "state_id": "native::1441792-22875-3099028754", "finished": false, "os_id": "1441792-22875-3099028754"}
2022-03-25T10:17:50.126+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC4\u_ex200420.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.126+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC4\u_ex200420.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC4\\u_ex200420.log", "state_id": "native::4784128-301129-3099028754", "finished": false, "os_id": "4784128-301129-3099028754"}
2022-03-25T10:17:50.126+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC4\u_ex211019.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.126+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC4\u_ex211019.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC4\\u_ex211019.log", "state_id": "native::524288-204290-3099028754", "finished": false, "os_id": "524288-204290-3099028754"}
2022-03-25T10:17:50.126+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC2\u_ex191104.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.127+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC2\u_ex191104.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC2\\u_ex191104.log", "state_id": "native::458752-240790-3099028754", "finished": false, "os_id": "458752-240790-3099028754"}
2022-03-25T10:17:50.127+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC2\u_ex200415.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.127+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC2\u_ex200415.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC2\\u_ex200415.log", "state_id": "native::4849664-299107-3099028754", "finished": false, "os_id": "4849664-299107-3099028754"}
2022-03-25T10:17:50.127+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC2\u_ex200524.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.127+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC2\u_ex200524.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC2\\u_ex200524.log", "state_id": "native::1900544-34068-3099028754", "finished": false, "os_id": "1900544-34068-3099028754"}
2022-03-25T10:17:50.127+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC2\u_ex210223.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.127+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC2\u_ex210223.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC2\\u_ex210223.log", "state_id": "native::43581440-153177-3099028754", "finished": false, "os_id": "43581440-153177-3099028754"}
2022-03-25T10:17:50.146+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC3\u_ex201225.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.147+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC3\u_ex201225.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC3\\u_ex201225.log", "state_id": "native::12976128-115773-3099028754", "finished": false, "os_id": "12976128-115773-3099028754"}
2022-03-25T10:17:50.147+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC3\u_ex210322.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.147+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC3\u_ex210322.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC3\\u_ex210322.log", "state_id": "native::3997696-165866-3099028754", "finished": false, "os_id": "3997696-165866-3099028754"}
2022-03-25T10:17:50.147+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC2\u_ex200507.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}
2022-03-25T10:17:50.148+0700 DEBUG [input] log/input.go:632 Ignore file because ignore_older reached: C:\inetpub\logs\LogFiles\W3SVC2\u_ex200507.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47", "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC2\\u_ex200507.log", "state_id": "native::3604480-304346-3099028754", "finished": false, "os_id": "3604480-304346-3099028754"}
2022-03-25T10:17:50.148+0700 DEBUG [input] log/input.go:472 Check file for harvesting: C:\inetpub\logs\LogFiles\W3SVC3\u_ex200924.log {"input_id": "dca1c752-9db7-473a-9db3-aea796c48b47"}