# Logstash is not processing files from input plugin Azure Storage blob, even though new files are available to process **URL:** \ **Category:** Logstash\ **Created:** [November 14, 2016, 3:24pm UTC](https://discuss.elastic.co/t/logstash-is-not-processing-files-from-input-plugin-azure-storage-blob-even-though-new-files-are-available-to-process/65999 "2016-11-14T15:24:28Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![nethis](https://avatars.discourse-cdn.com/v4/letter/n/d2c977/32.png) [@nethis](https://discuss.elastic.co/u/nethis)\ **Post date:** [November 14, 2016, 3:24pm UTC](https://discuss.elastic.co/t/logstash-is-not-processing-files-from-input-plugin-azure-storage-blob-even-though-new-files-are-available-to-process/65999/1 "2016-11-14T15:24:28Z")
I have been using Azure Storage Input Plugin , for json files. After an improper shutdown of pipeline, the restart of the logstash process is not picking up the files from Azure Storage blob, even there are so many pending files to process. Logstash config is as below input { azureblob { storage\_account\_name =\> "aggregatesdata" storage\_access\_key =\> "key" container =\> "custommetric" } } output { elasticsearch { action =\> "index" hosts =\> ["10.158.36.199"] codec =\> json index =\> "it\_custommetric" document\_type =\> "dailyaggregate" document\_id =\> "%{dataid}" } } ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 15, 2016, 1:08am UTC](https://discuss.elastic.co/t/logstash-is-not-processing-files-from-input-plugin-azure-storage-blob-even-though-new-files-are-available-to-process/65999/2 "2016-11-15T01:08:30Z")
That doesn't look like a standard input plugin, where did you find it? ---
**Author:** ![nethis](https://avatars.discourse-cdn.com/v4/letter/n/d2c977/32.png) [@nethis](https://discuss.elastic.co/u/nethis)\ **Post date:** [November 18, 2016, 8:54am UTC](https://discuss.elastic.co/t/logstash-is-not-processing-files-from-input-plugin-azure-storage-blob-even-though-new-files-are-available-to-process/65999/3 "2016-11-18T08:54:47Z")
Got from the link below > **[Azure/azure-diagnostics-tools](https://github.com/Azure/azure-diagnostics-tools/tree/master/Logstash/logstash-input-azureblob)** > > azure-diagnostics-tools - Plugins and tools for collecting, processing, managing, and visualizing diagnostics data and configuration Actually the problem is not with the plug-in. Logstash is showing the following message continuously. Can you please help to resolve this issue. This seems to be something related to improper shutdown of earlier process? If yes, how can I resolve this. This message are showing up, even after the system restart. {:timestamp=\>"2016-11-16T11:30:48.160000+0000", :message=\>#\0, :events\_consumed=\>0, :worker\_count=\>2, :inflight\_count=\>0, :worker\_states=\>[{:status=\>"sleep", :alive=\>true, :index=\>0, :inflight\_count=\>0}, {:status=\>"sleep", :alive=\>true, :index=\>1, :inflight\_count=\>0}], :output\_info=\>[{:type=\>"elasticsearch", :config=\>{"action"=\>"index", "hosts"=\>["10.158.36.220"], "codec"=\>"json", "index"=\>"it\_customevent", "document\_type"=\>"dailyaggregate", "document\_id"=\>"%{dataid}", " **ALLOW\_ENV**"=\>false}, :is\_multi\_worker=\>false, :events\_received=\>0, :workers=\>\"index", hosts=\>["10.158.36.220"], codec=\>\"UTF-8"\>, index=\>"it\_customevent", document\_type=\>"dailyaggregate", document\_id=\>"%{dataid}", workers=\>1, manage\_template=\>true, template\_name=\>"logstash", template\_overwrite=\>false, flush\_size=\>500, idle\_flush\_time=\>1, doc\_as\_upsert=\>false, max\_retries=\>3, script\_type=\>"inline", script\_var\_name=\>"event", scripted\_upsert=\>false, retry\_max\_interval=\>2, retry\_max\_items=\>500, retry\_on\_conflict=\>1, ssl\_certificate\_verification=\>true, sniffing=\>false, sniffing\_delay=\>5\>]\>, :busy\_workers=\>0}], :thread\_info=\>[{"thread\_id"=\>19, "name"=\>"[main]\nil, "backtrace"=\>............ms/logstash-input-azureblob-0.9.5/lib/logstash/inputs/azureblob.rb:39:in `list_blob_names'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-input-azureblob-0.9.5/lib/logstash/inputs/azureblob.rb:37:in`loop'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-input-azureblob-0.9.5/lib/logstash/inputs/azureblob.rb:37:in `list_blob_names'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-input-azureblob-0.9.5/lib/logstash/inputs/azureblob.rb:73:in`process'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-input-azureblob-0.9.5/lib/logstash/inputs/azureblob.rb:89:in `run'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:342:in`inputworker'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:336:in `start_input'"], "blocked_on"=>nil, "status"=>"run", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/nokogiri-1.6.8-java/lib/nokogiri/xml/searchable.rb:165:in`evaluate'"}, {"thread\_id"=\>23, "name"=\>"[main]\>worker0", "plugin"=\>nil, "backtrace"=\>["[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:309:in `synchronize'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:309:in`inflight\_batches\_synchronize'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:234:in `worker_loop'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:201:in`start\_workers'"], "blocked\_on"=\>nil, "status"=\>"sleep", "current\_call"=\>"[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:309:in `synchronize'"}, {"thread_id"=>24, "name"=>"[main]>worker1", "plugin"=>nil, "backtrace"=>["[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:309:in`synchronize'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:309:in `inflight_batches_synchronize'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:234:in`worker\_loop'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:201:in `start_workers'"], "blocked_on"=>nil, "status"=>"sleep", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:309:in`synchronize'"}], :stalling\_threads\_info=\>[{"thread\_id"=\>19, "name"=\>"[main]\nil, "current\_call"=\>"[...]/vendor/bundle/jruby/1.9/gems/nokogiri-1.6.8-java/lib/nokogiri/xml/searchable.rb:165:in `evaluate'"}, {"thread_id"=>23, "name"=>"[main]>worker0", "plugin"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:309:in`synchronize'"}, {"thread\_id"=\>24, "name"=\>"[main]\>worker1", "plugin"=\>nil, "current\_call"=\>"[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:309:in `synchronize'"}]}\>, :level=\>:warn} {:timestamp=\>"2016-11-16T11:30:48.160000+0000", :message=\>"Forcefully quitting logstash..", :level=\>:fatal} ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 16, 2016, 8:55am UTC](https://discuss.elastic.co/t/logstash-is-not-processing-files-from-input-plugin-azure-storage-blob-even-though-new-files-are-available-to-process/65999/4 "2016-12-16T08:55:06Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.