Logstash plugin for Azure storage

Elastic Stack Logstash
1

Hi Team , I am trying to access and retrieve files from Azure storage. I am getting the following error when I run the config file. Any help is appreciated.

< [2022-02-21T12:07:37,177][ERROR][logstash.inputs.azureblobstorage] Unknown setting 'storage_account_name' for azure_blob_storage
[2022-02-21T12:07:37,178][ERROR][logstash.inputs.azureblobstorage] Unknown setting 'storage_access_key' for azure_blob_storage
[2022-02-21T12:07:37,188][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org
[2022-02-21T12:07:37,202][ERROR][logstash.agent           ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`"}
[2022-02-21T12:07:37,250][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "C:/Users/logstash-7.17.0-windows-x86_64/logstash-7.17.0/logstash-core/lib/logstash/agent.rb:396:in `block in converge_state'"]}
[2022-02-21T12:07:37,262][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
2

Please find below my Config file for reference:

input
{
    azure_blob_storage
    {
        storage_account_name => "***"
        storage_access_key => "***"
        container => "***"
    }
}

filter{
  json {
        source => "message"
    }
    mutate {
        add_field => { "environment" => "test-env" }
        remove_field => [ "message" ]
    }
    date {
        match => ["unixtimestamp", "UNIX"]
    }
}

output {
  stdout { }
}

output {
    file {
        path => 'C:\Users\**'
        codec => rubydebug
    }
    stdout { 
        codec => rubydebug
3

You have the wrong names for those options. Check the code for the right names.

You may have been reading the documentation for another third party input plugin that reads data from Azure blob storage. (This versus this.)

4

Thank You @Badger . I changed the names and ran the config.
Below is the error I am getting now: Please advice.

C:\Users\...\config>logstash -f logstash-azure.conf
Using JAVA_HOME defined java: C:\Program Files\Java\jdk1.8.0_321
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.
.............
[2022-02-21T23:03:40,097][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-02-21T23:03:41,453][INFO ][org.reflections.Reflections] Reflections took 89 ms to scan 1 urls, producing 119 keys and 417 values
[2022-02-21T23:03:45,565][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["C:/Users/.../config/logstash-azure.conf"], :thread=>"#<Thread:0x683eae23 run>"}
[2022-02-21T23:03:46,773][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.2}
[2022-02-21T23:03:46,803][INFO ][logstash.inputs.azureblobstorage][main] === azure_blob_storage 0.12.3 / main / 0b8639 / ruby 2.5.8p0 ===
[2022-02-21T23:03:46,804][INFO ][logstash.inputs.azureblobstorage][main] If this plugin doesn't work, please raise an issue in https://github.com/janmg/logstash-input-azure_blob_storage
[2022-02-21T23:03:46,812][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-02-21T23:03:46,890][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-02-21T23:03:51,050][ERROR][logstash.inputs.azureblobstorage][main][0b8639dd2a856a7d9b1264745734af6a47df3e073cc6c77673484cb3b1cecdf4] caught: InvalidResourceName (400): The specifed resource name contains invalid characters.
RequestId:863be9a2-501e-00b4-3149-2793c3000000
Time:2022-02-21T17:33:50.2807442Z
[2022-02-21T23:03:51,052][ERROR][logstash.inputs.azureblobstorage][main][0b8639dd2a856a7d9b1264745734af6a47df3e073cc6c77673484cb3b1cecdf4] loading registry failed for attempt 1 of 3
[2022-02-21T23:03:51,320][ERROR][logstash.inputs.azureblobstorage][main][0b8639dd2a856a7d9b1264745734af6a47df3e073cc6c77673484cb3b1cecdf4] caught: InvalidResourceName (400): The specifed resource name contains invalid characters.
RequestId:863bea61-501e-00b4-6349-2793c3000000
Time:2022-02-21T17:33:51.1911894Z
[2022-02-21T23:03:51,321][ERROR][logstash.inputs.azureblobstorage][main][0b8639dd2a856a7d9b1264745734af6a47df3e073cc6c77673484cb3b1cecdf4] loading registry failed for attempt 2 of 3
[2022-02-21T23:03:51,496][ERROR][logstash.inputs.azureblobstorage][main][0b8639dd2a856a7d9b1264745734af6a47df3e073cc6c77673484cb3b1cecdf4] caught: InvalidResourceName (400): The specifed resource name contains invalid characters.
RequestId:863bea93-501e-00b4-0c49-2793c3000000
Time:2022-02-21T17:33:51.3980625Z
[2022-02-21T23:03:51,496][ERROR][logstash.inputs.azureblobstorage][main][0b8639dd2a856a7d9b1264745734af6a47df3e073cc6c77673484cb3b1cecdf4] loading registry failed for attempt 3 of 3
[2022-02-21T23:03:51,508][INFO ][logstash.inputs.azureblobstorage][main][0b8639dd2a856a7d9b1264745734af6a47df3e073cc6c77673484cb3b1cecdf4] learn_encapsulation, this can be skipped by setting skip_learning => true. Or set both head_file and tail_file
[2022-02-21T23:03:51,526][INFO ][logstash.inputs.azureblobstorage][main][0b8639dd2a856a7d9b1264745734af6a47df3e073cc6c77673484cb3b1cecdf4] learn json header and footer failed because bad URI(is not URI?): https://storageaccount.blob.core.windows.net/containers
[2022-02-21T23:03:51,528][INFO ][logstash.inputs.azureblobstorage][main][0b8639dd2a856a7d9b1264745734af6a47df3e073cc6c77673484cb3b1cecdf4] head will be: {"records":[ and tail is set to ]}
[2022-02-21T23:03:51,544][ERROR][logstash.inputs.azureblobstorage][main][0b8639dd2a856a7d9b1264745734af6a47df3e073cc6c77673484cb3b1cecdf4] caught: bad URI(is not URI?): https://storageaccount.blob.core.windows.net/containers for list_blobs retries left 3
[2022-02-21T23:03:51,546][ERROR][logstash.inputs.azureblobstorage][main][0b8639dd2a856a7d9b1264745734af6a47df3e073cc6c77673484cb3b1cecdf4] caught: bad URI(is not URI?): https://storageaccount.blob.core.windows.net/containers for list_blobs retries left 2
[2022-02-21T23:03:51,548][ERROR][logstash.inputs.azureblobstorage][main][0b8639dd2a856a7d9b1264745734af6a47df3e073cc6c77673484cb3b1cecdf4] caught: bad URI(is not URI?): https://storageaccount.blob.core.windows.net/containers for list_blobs retries left 1
[2022-02-21T23:03:51,643][ERROR][logstash.javapipeline    ][main][0b8639dd2a856a7d9b1264745734af6a47df3e073cc6c77673484cb3b1cecdf4] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::AzureBlobStorage container=>"containers", id=>"0b8639dd2a856a7d9b1264745734af6a47df3e073cc6c77673484cb3b1cecdf4", storageaccount=>"storageaccount", access_key=><password>, enable_metric=>true, codec=><LogStash::Codecs::JSON id=>"json_cc53cd37-2cf3-458a-8a78-365ef844c863", enable_metric=>true, charset=>"UTF-8">, logtype=>"raw", dns_suffix=>"core.windows.net", registry_path=>"data/registry.dat", registry_create_policy=>"resume", interval=>60, addfilename=>false, debug_until=>0, debug_timer=>false, skip_learning=>false, file_head=>"{\"records\":[", file_tail=>"]}", path_filters=>["**/*"]>
  Error: undefined method `each' for nil:NilClass
  Exception: NoMethodError
  Stack: C:/Users/.../logstash-7.17.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-azure_blob_storage-0.12.3/lib/logstash/inputs/azure_blob_storage.rb:199:in `run'
C:/Users/.../logstash-7.17.0/logstash-core/lib/logstash/java_pipeline.rb:409:in `inputworker'
C:/Users/.../logstash-7.17.0/logstash-core/lib/logstash/java_pipeline.rb:400:in `block in start_input'
5

And also, the error messages keep on coming up without stopping.

6

It fails whilst starting the plugin, for this error it will retry indefinitely.

I suspect the container name is the problem. Allowed names are specified here.

7

Thank You @Badger. I fixed the container name and re ran the config.
The script stops abruptly and nothing happens. Can you pls look into & assist.

C:\Users\...\config>logstash -f logstash-azure.conf
Using JAVA_HOME defined java: C:\Program Files\Java\jdk1.8.0_321
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.
Sending Logstash logs to C:/Users/.../logstash-7.17.0/logs which is now configured via log4j2.properties
[2022-02-22T13:56:31,984][INFO ][logstash.runner          ] Log4j configuration path used is: C:\Users\...\logstash-7.17.0\config\log4j2.properties
[2022-02-22T13:56:32,002][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.17.0", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 Java HotSpot(TM) 64-Bit Server VM 25.321-b07 on 1.8.0_321-b07 +indy +jit [mswin32-x86_64]"}
[2022-02-22T13:56:32,003][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[2022-02-22T13:56:32,116][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-02-22T13:56:34,953][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-02-22T13:56:36,387][INFO ][org.reflections.Reflections] Reflections took 104 ms to scan 1 urls, producing 119 keys and 417 values
[2022-02-22T13:56:40,246][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["C:/Users/.../logstash-7.17.0/config/logstash-azure.conf"], :thread=>"#<Thread:0x2ffd69e5 run>"}
[2022-02-22T13:56:41,613][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.36}
[2022-02-22T13:56:41,641][INFO ][logstash.inputs.azureblobstorage][main] === azure_blob_storage 0.12.3 / main / 44bb18 / ruby 2.5.8p0 ===
[2022-02-22T13:56:41,642][INFO ][logstash.inputs.azureblobstorage][main] If this plugin doesn't work, please raise an issue in https://github.com/janmg/logstash-input-azure_blob_storage
[2022-02-22T13:56:41,651][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-02-22T13:56:41,735][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-02-22T13:56:45,192][INFO ][logstash.inputs.azureblobstorage][main][44bb18b3ceef8a838bfedb112df6d8af74b3824044a5ba2a2c1df878bdff5e58] resuming from remote registry data/registry.dat
[2022-02-22T13:56:45,199][INFO ][logstash.inputs.azureblobstorage][main][44bb18b3ceef8a838bfedb112df6d8af74b3824044a5ba2a2c1df878bdff5e58] learn_encapsulation, this can be skipped by setting skip_learning => true. Or set both head_file and tail_file
[2022-02-22T13:56:46,614][INFO ][logstash.inputs.azureblobstorage][main][44bb18b3ceef8a838bfedb112df6d8af74b3824044a5ba2a2c1df878bdff5e58] learn json one of the attempts failed undefined method `name' for nil:NilClass
[2022-02-22T13:56:46,801][INFO ][logstash.inputs.azureblobstorage][main][44bb18b3ceef8a838bfedb112df6d8af74b3824044a5ba2a2c1df878bdff5e58] learn json one of the attempts failed undefined method `name' for nil:NilClass
[2022-02-22T13:56:46,803][INFO ][logstash.inputs.azureblobstorage][main][44bb18b3ceef8a838bfedb112df6d8af74b3824044a5ba2a2c1df878bdff5e58] head will be: 0 and tail is set to 0
8

Also tried to include the below, but nothing happens

file_head => 0
file_tail => 0
9

Looking at the code, I would guess that the container is empty.

10

@Badger - The container has 29 files and I am able to see that Logstash has saved the 29 blobs. I am unable to see them at my specified output location, but a data folder is getting created inside the Azure container where again all the 29 blobs are present.

[2022-02-23T17:28:36,604][INFO ][logstash.inputs.azureblobstorage][main][8437e2e837d318d43641fd4e9aed1cd1a691639b9e5756911d5aea70a1a73c9a] **processed 2 events, saving 29 blobs** and offsets to remote registry data/registry.dat
[2022-02-23T17:28:37,596][INFO ][logstash.outputs.file    ][main][8e0b14ee3af49a625b5809e2ba0a1c0a6ef3aa80b4da6d4530d41b80459fe4b5] Opening file {:path=>"C:/Users/.../Labs/Localisation"}
[2022-02-23T17:28:37,598][ERROR][logstash.javapipeline    ][main] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"main", :error=>"(EACCES) Permission denied - C:/Users/.../Labs/Localisation", :exception=>Java::OrgJrubyExceptions::SystemCallError, :backtrace=>["org.jruby.RubyIO.sysopen(org/jruby/RubyIO.java:1237)", "org.jruby.RubyFile.initialize(org/jruby/RubyFile.java:365)", "org.jruby.RubyIO.new(org/jruby/RubyIO.java:876)", "C_3a_.Users.PratiekAKP.Pratiek.Apache.logstash_minus_7_dot_17_dot_0_minus_windows_minus_x86_64.logstash_minus_7_dot_17_dot_0.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.open(C:/Users/PratiekAKP/Pratiek/Apache/logstash-7.17.0-windows-x86_64/logstash-7.17.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:276)", "C_3a_.Users.PratiekAKP.Pratiek.Apache.logstash_minus_7_dot_17_dot_0_minus_windows_minus_x86_64.logstash_minus_7_dot_17_dot_0.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(C:/Users/PratiekAKP/Pratiek/Apache/logstash-7.17.0-windows-x86_64/logstash-7.17.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:119)", "org.jruby.RubyHash.each(org/jruby/RubyHash.java:1415)", "C_3a_.Users.PratiekAKP.Pratiek.Apache.logstash_minus_7_dot_17_dot_0_minus_windows_minus_x86_64.logstash_minus_7_dot_17_dot_0.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(C:/Users/PratiekAKP/Pratiek/Apache/logstash-7.17.0-windows-x86_64/logstash-7.17.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:118)", "org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:164)", "C_3a_.Users.PratiekAKP.Pratiek.Apache.logstash_minus_7_dot_17_dot_0_minus_windows_minus_x86_64.logstash_minus_7_dot_17_dot_0.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(C:/Users/PratiekAKP/Pratiek/Apache/logstash-7.17.0-windows-x86_64/logstash-7.17.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:117)", "C_3a_.Users.PratiekAKP.Pratiek.Apache.logstash_minus_7_dot_17_dot_0_minus_windows_minus_x86_64.logstash_minus_7_dot_17_dot_0.logstash_minus_core.lib.logstash.outputs.base.multi_receive(C:/Users/PratiekAKP/Pratiek/Apache/logstash-7.17.0-windows-x86_64/logstash-7.17.0/logstash-core/lib/logstash/outputs/base.rb:103)", "org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.multi_receive(org/logstash/config/ir/compiler/OutputStrategyExt.java:143)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.multi_receive(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121)", "C_3a_.Users.PratiekAKP.Pratiek.Apache.logstash_minus_7_dot_17_dot_0_minus_windows_minus_x86_64.logstash_minus_7_dot_17_dot_0.logstash_minus_core.lib.logstash.java_pipeline.start_workers(C:/Users/.../logstash-7.17.0/logstash-core/lib/logstash/java_pipeline.rb:299)"], :thread=>"#<Thread:0x1608db54 sleep>"}
[2022-02-23T17:28:39,132][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2022-02-23T17:28:39,326][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2022-02-23T17:28:39,362][INFO ][logstash.runner          ] Logstash shut down.
11

And if I try to access the same container as the above which had 29 files, I get the below message:

[2022-02-23T17:44:36,504][INFO ][logstash.inputs.azureblobstorage][main][2a48cb792d94ad10ae1aaa52200b1e0249a552495d75b1277929ece9a8773fcf] learn json one of the attempts failed undefined method `name' for nil:NilClass
[2022-02-23T17:44:36,706][INFO ][logstash.inputs.azureblobstorage][main][2a48cb792d94ad10ae1aaa52200b1e0249a552495d75b1277929ece9a8773fcf] learn json one of the attempts failed undefined method `name' for nil:NilClass
[2022-02-23T17:44:36,708][INFO ][logstash.inputs.azureblobstorage][main][2a48cb792d94ad10ae1aaa52200b1e0249a552495d75b1277929ece9a8773fcf] head will be: {"records":[ and tail is set to ]}
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.