Filebeat as webjob in an Azure App Service

Hello,
Has anyone got Filebeat to work in an Azure App Service? I'm interested in how I can send the IIS logs using the IIS module from Filebeat running as a webjob to our Elastic cloud cluster.

Please let me know,
Thank you!

For those that are wondering - I eventually got this to work by using Filebeat version 6.3.0. From what I could tell, the code in the later version causes filebeat to throw the error "error initializing processors" for "GetFileVersionInfoSize"

For whatever reason this seems to be broken when testing with newer versions - maybe due to the enrollment features added.

Interesting. Can you share some logs or stack traces?

Also for others, what did you do to get it running?

Which configuation file are you using. I wonder if some of the default processors for collecting host/cloud meta-data might be the cause.

Hello,
Here is the error:

D:\home\site\wwwroot\App_Data\jobs\triggered\filebeat\filebeat-6.5.4-windows-x86_64>filebeat -e -d *
2019-01-29T15:09:55.378Z	INFO	instance/beat.go:592	Home path: [D:\home\site\wwwroot\App_Data\jobs\triggered\filebeat\filebeat-6.5.4-windows-x86_64] Config path: [D:\home\site\wwwroot\App_Data\jobs\triggered\filebeat\filebeat-6.5.4-windows-x86_64] Data path: [D:\home\site\wwwroot\App_Data\jobs\triggered\filebeat\filebeat-6.5.4-windows-x86_64\data] Logs path: [D:\home\site\wwwroot\App_Data\jobs\triggered\filebeat\filebeat-6.5.4-windows-x86_64\logs]
2019-01-29T15:09:55.768Z	DEBUG	[beat]	instance/beat.go:629	Beat metadata path: D:\home\site\wwwroot\App_Data\jobs\triggered\filebeat\filebeat-6.5.4-windows-x86_64\data\meta.json
2019-01-29T15:09:55.768Z	INFO	instance/beat.go:599	Beat UUID: 5e898cd7-768d-4cb8-8d17-b71ad4b56281
2019-01-29T15:09:55.768Z	DEBUG	[seccomp]	seccomp/seccomp.go:88	Syscall filtering is only supported on Linux
2019-01-29T15:09:55.815Z	INFO	[beat]	instance/beat.go:825	Beat info	{"system_info": {"beat": {"path": {"config": "D:\\home\\site\\wwwroot\\App_Data\\jobs\\triggered\\filebeat\\filebeat-6.5.4-windows-x86_64", "data": "D:\\home\\site\\wwwroot\\App_Data\\jobs\\triggered\\filebeat\\filebeat-6.5.4-windows-x86_64\\data", "home": "D:\\home\\site\\wwwroot\\App_Data\\jobs\\triggered\\filebeat\\filebeat-6.5.4-windows-x86_64", "logs": "D:\\home\\site\\wwwroot\\App_Data\\jobs\\triggered\\filebeat\\filebeat-6.5.4-windows-x86_64\\logs"}, "type": "filebeat", "uuid": "5e898cd7-768d-4cb8-8d17-b71ad4b56281"}}}
2019-01-29T15:09:55.815Z	INFO	[beat]	instance/beat.go:834	Build info	{"system_info": {"build": {"commit": "bd8922f1c7e93d12b07e0b3f7d349e17107f7826", "libbeat": "6.5.4", "time": "2018-12-17T20:21:46.000Z", "version": "6.5.4"}}}
2019-01-29T15:09:55.815Z	INFO	[beat]	instance/beat.go:837	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.10.6"}}}
2019-01-29T15:09:55.815Z	INFO	[beat]	instance/beat.go:870	Process info	{"system_info": {"process": {"cwd": "D:\\home\\site\\wwwroot\\App_Data\\jobs\\triggered\\filebeat\\filebeat-6.5.4-windows-x86_64", "exe": "\\\\10.0.176.6\\volume-47-default\\a16fde6cb5bb7142537e\\1836fb860726498da5dd83a1977d2f7e\\site\\wwwroot\\App_Data\\jobs\\triggered\\filebeat\\filebeat-6.5.4-windows-x86_64\\filebeat.exe", "name": "filebeat.exe", "pid": 17720, "ppid": 23696, "start_time": "2019-01-29T15:09:54.182Z"}}}
2019-01-29T15:09:55.815Z	INFO	instance/beat.go:278	Setup Beat: filebeat; Version: 6.5.4
2019-01-29T15:09:55.815Z	DEBUG	[beat]	instance/beat.go:299	Initializing output plugins
2019-01-29T15:09:55.815Z	INFO	instance/beat.go:357	filebeat stopped.

2019-01-29T15:09:55.831Z	ERROR	instance/beat.go:800	Exiting: error initializing publisher: error initializing processors: 1 error: GetFileVersionInfoSize failed: The system cannot find the file specified.
Exiting: error initializing publisher: error initializing processors: 1 error: GetFileVersionInfoSize failed: The system cannot find the file specified.
D:\home\site\wwwroot\App_Data\jobs\triggered\filebeat\filebeat-6.5.4-windows-x86_64> 

Here is the config:

D:\home\site\wwwroot\App_Data\jobs\triggered\filebeat\filebeat-6.5.4-windows-x86_64>filebeat export config
cloud:
  auth: elastic:redacted
  id: 
filebeat:
  config:
    modules:
      path: D:\home\site\wwwroot\App_Data\jobs\triggered\filebeat\filebeat-6.5.4-windows-x86_64/modules.d/*.yml
      reload:
        enabled: false
  inputs:
  - enabled: false
    paths:
    - /var/log/*.log
    type: log
output:
  elasticsearch:
    hosts:
    - https://redacted.gcp.cloud.es.io:443
    password: redacted
    username: elastic
path:
  config: D:\home\site\wwwroot\App_Data\jobs\triggered\filebeat\filebeat-6.5.4-windows-x86_64
  data: D:\home\site\wwwroot\App_Data\jobs\triggered\filebeat\filebeat-6.5.4-windows-x86_64\data
  home: D:\home\site\wwwroot\App_Data\jobs\triggered\filebeat\filebeat-6.5.4-windows-x86_64
  logs: D:\home\site\wwwroot\App_Data\jobs\triggered\filebeat\filebeat-6.5.4-windows-x86_64\logs
processors:
- add_host_metadata: null
- add_cloud_metadata: null
setup:
  kibana:
    host: https://redacted.us-central1.gcp.cloud.es.io:443
  template:
    settings:
      index:
        number_of_shards: 3

This works fine in 6.4.0 as well.

Which modules have you enabled? Might be a module configuration accessing a file with wrong credentials?

Hello - This was with the IIS module enabled.
Based on the versions, I think it may be related to the new ILM or beat enrollment features added in newer releases.

ILM and beats enrollment are disabled by default.

The call that failed is used to get the file size. Seems to be indirectly called by the stdlib when we check if a file is actually present. Unfortunately we're missing the actual file name. Makes me wonder if something is wrong with the file locations in the 6.5.4 release.

Can you start filebeat with some strace like windows alternative, so we can check the actual parameters that made the call fail?
E.g.

Also run filebeat in debug mode with -d *.

Unfortunately logging swallows some context here. If possible I'd love to fix/improve the logging, so to give users a chance to debug. And maybe find and fix the root cause. It would be a shame if the error persists in future releases.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.