# Configure logstash to read local files recursively and send to elastic cloud

**URL:** <https://discuss.elastic.co/t/configure-logstash-to-read-local-files-recursively-and-send-to-elastic-cloud/125440>\
**Category:** Logstash\
**Created:** [March 24, 2018, 1:41pm UTC](https://discuss.elastic.co/t/configure-logstash-to-read-local-files-recursively-and-send-to-elastic-cloud/125440 "2018-03-24T13:41:20Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![elastic\_bees](https://avatars.discourse-cdn.com/v4/letter/e/779978/32.png) [@elastic\_bees](https://discuss.elastic.co/u/elastic_bees)\
**Post date:** [March 24, 2018, 1:41pm UTC](https://discuss.elastic.co/t/configure-logstash-to-read-local-files-recursively-and-send-to-elastic-cloud/125440/1 "2018-03-24T13:41:21Z")

</div>

Hi,

I'm trying to configure logstash to read local files recursively and send the output to elastic cloud.

I put my CloudID in /etc/logstash/logstash.yml  
Should I specify something in the logstatsh conf file?

Do you guys may have an conf file exampe for me?

Thanks

---

<div class="post-metadata">

**Author:** ![zqc0512](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zqc0512/32/32141_2.png) [@zqc0512](https://discuss.elastic.co/u/zqc0512)\
**Post date:** [March 26, 2018, 6:33am UTC](https://discuss.elastic.co/t/configure-logstash-to-read-local-files-recursively-and-send-to-elastic-cloud/125440/2 "2018-03-26T06:33:21Z")

</div>

as i know in logstash doc. have a example.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 26, 2018, 6:46am UTC](https://discuss.elastic.co/t/configure-logstash-to-read-local-files-recursively-and-send-to-elastic-cloud/125440/3 "2018-03-26T06:46:04Z")

</div>

You need a pipeline configuration file with, at a minimum, a file input and an elasticsearch output. This is a standard setup so there should be plenty of examples to find.

---

<div class="post-metadata">

**Author:** ![elastic\_bees](https://avatars.discourse-cdn.com/v4/letter/e/779978/32.png) [@elastic\_bees](https://discuss.elastic.co/u/elastic_bees)\
**Post date:** [March 26, 2018, 7:13am UTC](https://discuss.elastic.co/t/configure-logstash-to-read-local-files-recursively-and-send-to-elastic-cloud/125440/4 "2018-03-26T07:13:13Z")

</div>

Dear all,

Thanks for your quick replies. Much appreciated.  
I've tried to gather something based on what I've found but I can't make it working.  
A little bit of context:  
I have loads of subdirectories under /home/user/logs. They all contains numerous log files that need to be injected in my elastic cloud database.

Here's what I've setup in a logstatsh-conf.conf

input {  
file {  
path =\> "/home/user/logs/\*"  
}  
}

output {  
elasticsearch { }

}

Unfortunately elastic cloud does not get populated.

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 26, 2018, 7:17am UTC](https://discuss.elastic.co/t/configure-logstash-to-read-local-files-recursively-and-send-to-elastic-cloud/125440/5 "2018-03-26T07:17:25Z")

</div>

Read the file input documentation carefully, especially what's said about `start_position` and sincedb.

---

<div class="post-metadata">

**Author:** ![elastic\_bees](https://avatars.discourse-cdn.com/v4/letter/e/779978/32.png) [@elastic\_bees](https://discuss.elastic.co/u/elastic_bees)\
**Post date:** [March 26, 2018, 7:28am UTC](https://discuss.elastic.co/t/configure-logstash-to-read-local-files-recursively-and-send-to-elastic-cloud/125440/6 "2018-03-26T07:28:24Z")

</div>

I'm actually more doubtful about that part:

> [@elastic\_bees](#):
>
> output {
> 
> elasticsearch { }
> 
> }

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 26, 2018, 8:27am UTC](https://discuss.elastic.co/t/configure-logstash-to-read-local-files-recursively-and-send-to-elastic-cloud/125440/7 "2018-03-26T08:27:54Z")

</div>

Oh, that was literally what you had in your configuration? That won't work. See [https://www.elastic.co/guide/en/cloud/current/logstash.html](https://www.elastic.co/guide/en/cloud/current/logstash.html).

---

<div class="post-metadata">

**Author:** ![elastic\_bees](https://avatars.discourse-cdn.com/v4/letter/e/779978/32.png) [@elastic\_bees](https://discuss.elastic.co/u/elastic_bees)\
**Post date:** [March 26, 2018, 9:14am UTC](https://discuss.elastic.co/t/configure-logstash-to-read-local-files-recursively-and-send-to-elastic-cloud/125440/8 "2018-03-26T09:14:26Z")

</div>

Thanks Magnus,

Got it configured as close as I thought it should be but now I get this nasty error.....I've checked possible syntax errors for hours..... maybe you would know what's happening? Thanks

[2018-03-26T10:52:31,671][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>2, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50}  
[2018-03-26T10:52:31,705][DEBUG][logstash.outputs.elasticsearch] Normalizing http path {:path=\>nil, :normalized=\>nil}  
[2018-03-26T10:52:31,997][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>[[https://elastic:xxxxxx@bae0d0d5360f193f8d3584ef04595612.eu-central-1.aws.cloud.es.io:9243/](https://elastic:xxxxxx@bae0d0d5360f193f8d3584ef04595612.eu-central-1.aws.cloud.es.io:9243/)]}}  
[2018-03-26T10:52:31,999][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[https://elastic:xxxxxx@bae0d0d5360f193f8d3584ef04595612.eu-central-1.aws.cloud.es.io:9243/](https://elastic:xxxxxx@bae0d0d5360f193f8d3584ef04595612.eu-central-1.aws.cloud.es.io:9243/), :path=\>"/"}  
[2018-03-26T10:52:32,908][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[https://elastic:xxxxxx@bae0d0d5360f193f8d3584ef04595612.eu-central-1.aws.cloud.es.io:9243/](https://elastic:xxxxxx@bae0d0d5360f193f8d3584ef04595612.eu-central-1.aws.cloud.es.io:9243/)"}  
[2018-03-26T10:52:33,444][ERROR][logstash.pipeline] Error registering plugin {:pipeline\_id=\>"main", :plugin=\>"#\<LogStash::OutputDelegator:0x144646 @namespaced\_metric=#\<LogStash::Instrument::NamespacedMetric:0x1b85dc6 @metric=#\<LogStash::Instrument::Metric:0x1b41dee @collector=#\<LogStash::Instrument::Collector:0x8ef65d @agent=nil, @metric\_store=#\<LogStash::Instrument::MetricStore:0x6cbbfc @store=#\<Concurrent:🗺0x00000000000fb0 entries=4 default\_proc=nil\>, @structured\_lookup\_mutex=#Mutex:0x18d855a, @fast\_lookup=#\<Concurrent:🗺0x00000000000fb4 entries=63 default\_proc=nil\>\>\>\>, @namespace\_name=[:stats, :pipelines, :main, :plugins, :outputs, :"0094fcbe61344a8e74698cf70b12dbcb996aed4044c2e103744ffe2e73fb90cf"]\>, @metric=#\<LogStash::Instrument::NamespacedMetric:0x1de8464 @metric=#\<LogStash::Instrument::Metric:0x1b41dee @collector=#\<LogStash::Instrument::Collector:0x8ef65d @agent=nil, @metric\_store=#\<LogStash::Instrument::MetricStore:0x6cbbfc @store=#\<Concurrent:🗺0x00000000000fb0 entries=4 default\_proc=nil\>, @structured\_lookup\_mutex=#Mutex:0x18d855a, @fast\_lookup=#\<Concurrent:🗺0x00000000000fb4 entries=63 default\_proc=nil\>\>\>\>, @namespace\_name=[:stats, :pipelines, :main, :plugins, :outputs]\>, @out\_counter=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: out value:0, @strategy=#\<LogStash::OutputDelegatorStrategies::Shared:0x1749773 @output=\<LogStash::Outputs::ElasticSearch hosts=\>[[https://bae0d0d5360f193f8d3584ef04595612.eu-central-1.aws.cloud.es.io:9243](https://bae0d0d5360f193f8d3584ef04595612.eu-central-1.aws.cloud.es.io:9243)], user=\>"elastic", password=\>, index=\>"myindex", document\_type=\>"psalogs", id=\>"0094fcbe61344a8e74698cf70b12dbcb996aed4044c2e103744ffe2e73fb90cf", enable\_metric=\>true, codec=\>\<LogStash::Codecs::Plain id=\>"plain\_ec1e9f2e-7aa2-4a8f-afee-ac3f68fb1253", enable\_metric=\>true, charset=\>"UTF-8"\>, workers=\>1, manage\_template=\>true, template\_name=\>"logstash", template\_overwrite=\>false, doc\_as\_upsert=\>false, script\_type=\>"inline", script\_lang=\>"painless", script\_var\_name=\>"event", scripted\_upsert=\>false, retry\_initial\_interval=\>2, retry\_max\_interval=\>64, retry\_on\_conflict=\>1, action=\>"index", ssl\_certificate\_verification=\>true, sniffing=\>false, sniffing\_delay=\>5, timeout=\>60, pool\_max=\>1000, pool\_max\_per\_route=\>100, resurrect\_delay=\>5, validate\_after\_inactivity=\>10000, http\_compression=\>false, ssl=\>true\>\>, @in\_counter=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: in value:0, @id="0094fcbe61344a8e74698cf70b12dbcb996aed4044c2e103744ffe2e73fb90cf", @time\_metric=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: duration\_in\_millis value:0, @metric\_events=#\<LogStash::Instrument::NamespacedMetric:0xf10d15 @metric=#\<LogStash::Instrument::Metric:0x1b41dee @collector=#\<LogStash::Instrument::Collector:0x8ef65d @agent=nil, @metric\_store=#\<LogStash::Instrument::MetricStore:0x6cbbfc @store=#\<Concurrent:🗺0x00000000000fb0 entries=4 default\_proc=nil\>, @structured\_lookup\_mutex=#Mutex:0x18d855a, @fast\_lookup=#\<Concurrent:🗺0x00000000000fb4 entries=63 default\_proc=nil\>\>\>\>, @namespace\_name=[:stats, :pipelines, :main, :plugins, :outputs, :"0094fcbe61344a8e74698cf70b12dbcb996aed4044c2e103744ffe2e73fb90cf", :events]\>, @output\_class=LogStash::Outputs::ElasticSearch\>", :error=\>"Unexpected character ('\<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')\n at [Source: (byte[])""; line: 1, column: 2]", :thread=\>"#\<Thread:0x5b0b20@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 run\>"}  
[2018-03-26T10:52:33,448][ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\<LogStash::Json::ParserError: Unexpected character ('\<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')  
at [Source: (byte[])"var hashRoute = '/app/kibana';  
var defaultRoute = '/app/kibana';

---

<div class="post-metadata">

**Author:** ![elastic\_bees](https://avatars.discourse-cdn.com/v4/letter/e/779978/32.png) [@elastic\_bees](https://discuss.elastic.co/u/elastic_bees)\
**Post date:** [March 26, 2018, 9:15am UTC](https://discuss.elastic.co/t/configure-logstash-to-read-local-files-recursively-and-send-to-elastic-cloud/125440/9 "2018-03-26T09:15:52Z")

</div>

var hash = window.location.hash;  
if (hash.length) {  
window.location = hashRoute + hash;  
} else {  
window.location = defaultRoute;  
}"; line: 1, column: 2]\>, :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/json.rb:17:in `jruby_load'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:446:in`get\_es\_version'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http\_client/pool.rb:248:in `block in healthcheck!'", "org/jruby/RubyHash.java:1343:in`each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http\_client/pool.rb:240:in `healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:340:in`update\_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http\_client/pool.rb:70:in `start'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:292:in`build\_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http\_client.rb:60:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:101:in`create\_http\_client'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http\_client\_builder.rb:97:in `build'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch.rb:230:in`build\_client'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:24:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:in`register'", "/usr/share/logstash/logstash-core/lib/logstash/output\_delegator.rb:42:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:in`register\_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:352:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in`each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:352:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:735:in`maybe\_setup\_out\_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:362:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:289:in`run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:249:in `block in start'"], :thread=\>"#\<Thread:0x5b0b20@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 run\>"}

[2018-03-26T10:52:33,467][ERROR][logstash.agent] Failed to execute action {:id=\>:main, :action\_type=\>LogStash::ConvergeResult::FailedAction, :message=\>"Could not execute action: LogStash::PipelineAction::Create/pipeline\_id:main, action\_result: false", :backtrace=\>nil}  
[2018-03-26T10:52:33,471][DEBUG][logstash.instrument.periodicpoller.os] Stopping  
[2018-03-26T10:52:33,472][DEBUG][logstash.instrument.periodicpoller.jvm] Stopping  
[2018-03-26T10:52:33,472][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Stopping  
[2018-03-26T10:52:33,472][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Stopping  
[2018-03-26T10:52:33,478][DEBUG][logstash.agent] Shutting down all pipelines {:pipelines\_count=\>0}  
[2018-03-26T10:52:33,479][DEBUG][logstash.agent] Converging pipelines state {:actions\_count=\>0}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 23, 2018, 10:48am UTC](https://discuss.elastic.co/t/configure-logstash-to-read-local-files-recursively-and-send-to-elastic-cloud/125440/13 "2018-04-23T10:48:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.