Logstash-input-azureblob Fails to Install with Error

Elastic Stack Logstash
1

$ sudo /usr/share/logstash/bin/logstash-plugin install logstash-input-azureblob
Using bundled JDK: /usr/share/logstash/jdk
Validating logstash-input-azureblob
Resolving mixin dependencies
WARN: Unresolved or ambiguous specs during Gem::Specification.reset:
date (>= 0)
Available/installed versions of this gem:
- 3.3.4
- 3.3.3
WARN: Clearing out unresolved specs. Try 'gem cleanup '
Please report a bug if this causes problems.
Installing logstash-input-azureblob
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "faraday":
In Gemfile:
faraday (~> 1)

logstash-input-azureblob was resolved to 0.9.7, which depends on
  azure (~> 0.7.1) was resolved to 0.7.10, which depends on
    faraday (~> 0.9)

$

2

0.9.7 dates to Feb 2017. If I am reading things correctly it uses an API that predates Blobs being GA, and the input was completely rewritten using a different Azure API a few months later. There have been minor updates to the code since, but it has never been re-released. It's not supported by Elastic and does not appear to be supported by anyone else either. Don't use it.

There is an issue about this, and the solution may be Filebeat.

3

Thank you for the response. I am trying to get Azure Flow Logs, which are stored in Azure Blob Storage, into Grafana Loki. Microsoft recommends using Logstash with Azure Blob as an input and Loki as an output. Loki Plugin installs (very old there too), but Azure Blob does not.

I'm trying to paste the link here, but getting an error trying to do that. The Microsoft Documentation recommending this setup is called " Manage and analyze network security group flow logs using Network Watcher and Grafana" 05/31/24

4

That tells you to install logstash 5.2.0, which also dates to early 2017, and it would not surprise me if the azureblob input worked with that. (Although it may not, I don't know if Azure still supports the old API that the input uses.)

However, if you run logstash 5.2, which is years beyond EOL, you will also need to run EOL versions of the rest of the elastic stack. I'm not saying it cannot be done, but I do not recommend it.

5

Right, I get it, I just don't think Microsoft does. I have no idea why they would recommend ancient software.

Conclusion:

  • logstash-input-azureblob is way EOL, do not use it
  • Microsoft documentation is inaccurate
  • There is no known way to export Flow Logs from Azure Blob Storage without using expensive MS tools
6

Probably it was created a long time ago and never really updated.

The pipeline itself probably work, you just need the data from the files.

You could try to build a data processing flow using filebeat to read the files and send to logstash, and then use logstash to process it.

A few extra filters to parse the message may be required, but I see no reason for it to not work in this way.

Something like this:

Blob Storage -> Filebeat input azure-blob-storage -> Filebeat output logstash -> Logstash beats input -> parsing filters -> Logstash elasticsearch output.

1 Like
7

I was able to get a successful workflow going with Azure FlowLogs->Azure Blob Storage->Filebeat->Logstash->Loki->Grafana.

Thank you for the suggestions.