Option for transferring log info from Filebeat to Logstash using https

Elastic Stack Beats
1

Hi,
I am looking into an option where i can transfer the log information from FileBeat to Logstash using http(s).Currently the problem i am facing is that filebeat uses beat/lumberjack protocol which is being blocked by our company's infrastructure.

Please advice.

Thanks,
RK

1 Like
2

Hi Ramkchirra,

There are a few options to achieve this but none of them are perfect at this point in time.

SOCKS5 proxy support was added to the logstash output in the recent Beats v5.0.0 ALPHA1 release:
https://www.elastic.co/guide/en/beats/libbeat/master/release-notes-5.0.0-alpha1.html

See this pull request to see what the config would look like:

But this is SOCKS5 not HTTP proxy mode, so depends what your proxy is capable of or allows.

Another possibility is to ship events from beats directly to Elasticsearch as the current version of beats has elasticsearch output with a http proxy option:
https://www.elastic.co/guide/en/beats/filebeat/1.2/elasticsearch-output.html#_proxy_url

But this probably won't help you with filebeat as you need logstash to do some transform stuff typically when dealing with log files.

Other than the above options if you can run a logstash on the same side of the proxy as the hosts forwarding logs with beats shippers, then you could use the elasticsearch output in logstash to ship events with HTTP proxy configuration.

See https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-proxy

Perhaps test out 5.0.0 alpha2 of filebeat which should include the socks5 proxy and see if it works for you.
The 5.0 stable release isn't too far off.

Let me know what you think about the suggested options.

Thanks

1 Like
3

Elasticsearch 5.0 will add so called 'ingest node' supporting some logstash like event processing (e.g. grok filters). See Elasticsearch 5.0 alpha2 docs: https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest.html

2 Likes
4

thanks for the quick reply !! Will look into the options and get back to you.:slight_smile:

1 Like
5

Thanks for your reply. That's very good to hear !!

1 Like
6

I forgot about the new ingest node, this could be a great solution for you if you can perform all the required logstash transform work directly inside the elasticsearch ingest node and also use HTTP proxying to ship your events.

1 Like