Getting logs from azure blog storage and pushing to ES through Logstash?

[Yaswanth]

Hi Team,

I am sending my txt file which is stored in azureblobstorage to ES directly with the help of logstash with this plugin.

My config is like this:

input
{
    azureblob
    {
        storage_account_name => "mystorageaccount"
        storage_access_key => "VGhpcyBpcyBhIGZha2Uga2V5Lg=="
        container => "mycontainer"
    }
}
output {
  elasticsearch { hosts => ["localhost:9200"] 
}
} 

I am getting error like this in logstash

Caught exception while locking {:exception=>#<Azure::Core::Http::HTTPError:15752 @status_code: 400, @http_response: #<Azure::Core::Http::HttpResponse:0x70b417e1 @http_response=#<Faraday::Response:0x5dbac6c4 @on_complete_callbacks=[], @env=#<Faraday::Env @method=:put @body="\xEF\xBB\xBF<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>BlobTypeNotSupported</Code><Message>Page blob is not supported for this account type.\nRequestId:a1be676a-0001-0050-3fb9-002df1000000\nTime:2017-07-19T18:06:18.1830830Z</Message></Error>" @url=#<URI::HTTPS:0x37eca579 URL:https://azureexplorerblob.blob.core.windows.net/eslogs/picaso.txt.lock> @request=#<Faraday::RequestOptions open_timeout=60> @request_headers={"User-Agent"=>"Azure-Core/0.1.7", "x-ms-date"=>"Wed, 19 Jul 2017 18:06:17 GMT", "x-ms-version"=>"2014-02-14", "DataServiceVersion"=>"1.0;NetFx", "MaxDataServiceVersion"=>"2.0;NetFx", "Content-Type"=>"application/atom+xml; charset=utf-8", "x-ms-blob-type"=>"PageBlob", "Content-Length"=>"0", "x-ms-blob-content-length"=>"512", "x-ms-sequence-number"=>"0", "Authorization"=>"SharedKey azureexplorerblob:FXkteDw28TnQ7LuQRdRy+DmmLxzQZtHrce9E2lthLoI="} @ssl=#<Faraday::SSLOptions verify=true> @response=#<Faraday::Response:0x5dbac6c4 ...> @response_headers={"content-length"=>"238", "content-type"=>"application/xml", "server"=>"Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0", "x-ms-request-id"=>"a1be676a-0001-0050-3fb9-002df1000000", "x-ms-version"=>"2014-02-14", "date"=>"Wed, 19 Jul 2017 18:06:17 GMT", "connection"=>"close"} @status=400>>, @uri=#<URI::HTTPS:0x37eca579 URL:https://azureexplorerblob.blob.core.windows.net/eslogs/picaso.txt.lock>>, @uri: #<URI::HTTPS:0x37eca579 URL:https://azureexplorerblob.blob.core.windows.net/eslogs/picaso.txt.lock>, @description: "Page blob is not supported for this account type.\nRequestId:a1be676a-0001-0050-3fb9-002df1000000\nTime:2017-07-19T18:06:18.1830830Z", @type: "BlobTypeNotSupported">, :level=>:error}

But when i changed the account type from blob to general storage when i executed the first time the i can see the output send to ES . When i did the same again it is not sending any output to ES and also didnt showing any error in logstash too(i.e.Even i kept stdout { codec => rubydebug } in output ).

How can i resolve this and send data to ES ?

Thanks

[jsvd (João Duarte)]

I am not familiar with the azure blog storage, but the error is coming from the azure api and seems clear:

<Error><Code>BlobTypeNotSupported</Code><Message>Page blob is not supported for this account type.\nRequestId:a1be676a-0001-0050-3fb9-002df1000000

Googling for that error I see some hits like:

Page Blob is not supported for this account type – Azure Resource Manager

https://social.msdn.microsoft.com/Forums/en-US/e946458c-4cf5-4e1d-b842-860fc77ef1d9/azure-blob-storage-issue-with-cta-sp3-emc-cloud-tiering-appliance?forum=windowsazuredata