Azureblob input plugin - error

[Kevin_f]

Hello,

Hoping that someone within the community has come across this issue we are facing and may have managed to resolve it or have any idea?

Basically we have installed the azureblob plugin (0.9.13) to ingest logs from azure blob. Our logstash is on version 6.8.6-1

Our input and output config is as follows:

input {
    azureblob {
        storage_account_name => "storage_name"
        storage_access_key => "storage_key"
        container => "storage_container_name"
        add_field => { "service" => "blob" }
        tags => [ 'azure' ]
        codec => "json"
    }
}

output {
   if [service] == "blob" and "azure" in [tags] {
     elasticsearch {
        user => "elastic_user"
        password => "password"
        hosts => ["elasticsearch:9200"]
        cacert => '/etc/logstash/certs/ca.crt'
        index => "logstash-blob-v2-%{+xxxx.ww}"
        ssl => true
        ssl_certificate_verification => false
     }
  }
}

The error we are seeing in the logstash logs are:

[ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<NoMethodError: undefined method ` version' for nil:NilClass>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:135:in  `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:259:in ` register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270:in  `block in register_plugins'", "org/jruby/RubyArray.java:1792:in ` each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270:in  `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:413:in ` start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:311:in  `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:217:in ` run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:176:in  `block in start'"], :thread=>"#<Thread:0x86fc3c5 run>"}

Your help would be much appreciated. Many thanks.

Regards,
Kevin

[Badger]

The code is here. I have no suggestions as to why Gem.latest_spec_for('logstash-input-azureblob') would return nil

[Kevin_f]

HI Badger,

Thank you for coming back to me. I have no idea myself.

What would you suggest that I can do to work around this?

[Badger]

I really know nothing about gems. I know just enough ruby to write a few filters.

[Kevin_f]

Thank you Badger!

[Kevin_f]

okay so this is what I have done so far to fix the issue I was initially facing.

Add the 2 lines below in the jvm.options file at the path /etc/logstash/

-Dhttp.proxyHost=x.x.x.x.
-Dhttp.proxyPort=3128

Now I am facing another error shown below:

Oh My, An error occurred. LeaseNotPresentWithLeaseOperation (409): There is currently no lease on the blob.
RequestId:5ad6a382-701e-0114-0861-1d455d000000
Time:2020-04-28T13:31:15.0955857Z: ["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-core-0.1.15/lib/azure/core/http/retry_policy.rb:58:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-core-0.1.15/lib/azure/core/http/http_request.rb:110:in `block in with_filter'", "org/jruby/RubyMethod.java:116:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-core-0.1.15/lib/azure/core/http/signer_filter.rb:28:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-core-0.1.15/lib/azure/core/http/http_request.rb:110:in `block in with_filter'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-core-0.1.15/lib/azure/core/service.rb:36:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-core-0.1.15/lib/azure/core/filtered_service.rb:34:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-core-0.1.15/lib/azure/core/signed_service.rb:41:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-storage-0.15.0.preview/lib/azure/storage/service/storage_service.rb:62:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-storage-0.15.0.preview/lib/azure/storage/blob/blob_service.rb:62:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-storage-0.15.0.preview/lib/azure/storage/blob/blob_service.rb:445:in `break_lease'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/azure-storage-0.15.0.preview/lib/azure/storage/blob/blob.rb:627:in `break_blob_lease'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:337:in `acquire_lease'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:360:in `register_for_read'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:165:in `process'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azureblob-0.9.13-java/lib/logstash/inputs/azureblob.rb:151:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:426:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:420:in `block in start_input'"] {:exception=>#<Azure::Core::Http::HTTPError:2014 @status_code: 409, @http_response: #<Azure::Core::Http::HttpResponse:0x72d082ba @http_response=#<Faraday::Response:0x1aa1339c @on_complete_callbacks=[], @env=#<Faraday::Env @method=:put @body="\xEF\xBB\xBF<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>LeaseNotPresentWithLeaseOperation</Code><Message>There is currently no lease on the blob.\nRequestId:5ad6a382-701e-0114-0861-1d455d000000\nTime:2020-04-28T13:31:15.0955857Z</Message></Error>" @url=#<URI::HTTPS https://<storagename>.blob.core.windows.net/insights-logs-auditlogs/data/registry?comp=lease> @request=#<Faraday::RequestOptions open_timeout=60> @request_headers={"User-Agent"=>"logstash-input-azureblob/0.9.11; Azure-Storage/0.15.0-preview (Ruby 2.5.3-p0; Linux linux)", "x-ms-date"=>"Tue, 28 Apr 2020 13:31:14 GMT", "x-ms-version"=>"2016-05-31", "DataServiceVersion"=>"1.0;NetFx", "MaxDataServiceVersion"=>"3.0;NetFx", "Content-Type"=>"application/atom+xml; charset=utf-8", "x-ms-lease-action"=>"break", "x-ms-lease-break-period"=>"30", "Content-Length"=>"0", "Authorization"=>"SharedKey iotglobalaadauditlogs:Y6XtJc1bzX0ZlwIn63/l4Uj2Lfd9ith/VJ5T2obyBos="} @ssl=#<Faraday::SSLOptions verify=true> @response=#<Faraday::Response:0x1aa1339c ...> @response_headers={"content-length"=>"242", "content-type"=>"application/xml", "server"=>"Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0", "x-ms-request-id"=>"5ad6a382-701e-0114-0861-1d455d000000", "x-ms-version"=>"2016-05-31", "date"=>"Tue, 28 Apr 2020 13:31:14 GMT", "connection"=>"close"} @status=409>>,

Anyone with any ideas? Is there something on Azure that needs to be configured for leasing?