Filebeat: Configuring input from URL

[Conor_Hennessy]

Hi All,

I have used filebeat to parse and send an xml to elasticsearch and it has worked nicely. The xml file was downloaded from a url (something like example.com/example.xml) and I then included the path to the downloaded copy in filebeat.yml like

filebeat.inputs:
- type: log
  paths:
    - /path/to/example.xml

I am wondering if it is possible to use the url as an input in filebeat directly, or is there any workaround which would allow for retrieving the xml from a url?

Thank you in advance

[mtojek (Marcin Tojek)]

Did you try to pair it with HTTP input and maybe an ingest pipeline? Docs: HTTP Endpoint input | Filebeat Reference [7.13] | Elastic

[Conor_Hennessy]

Thanks a lot for you reply Marcin!

Since the HTTP endpoint input waits for a HTTP POST request, and my input is just a static url, I'm not sure how I can use the HTTP endpoint as input. Would you have any first steps that I can try?

[legoguy1000 (Alex)]

Your can use a GET request with the httpjson input. Whether it will work in ur case will depend on whether the URL returns text data or binary data. I'd just an xml text response, that will work and can be parsed. If it's binary file then that will not work and you'd have to use something else to download the file and the point filebeat to it.