Configure logstash to read local files recursively and send to elastic cloud

Elastic Stack Logstash
1

Hi,

I'm trying to configure logstash to read local files recursively and send the output to elastic cloud.

I put my CloudID in /etc/logstash/logstash.yml
Should I specify something in the logstatsh conf file?

Do you guys may have an conf file exampe for me?

Thanks

2

as i know in logstash doc. have a example.

3

You need a pipeline configuration file with, at a minimum, a file input and an elasticsearch output. This is a standard setup so there should be plenty of examples to find.

4

Dear all,

Thanks for your quick replies. Much appreciated.
I've tried to gather something based on what I've found but I can't make it working.
A little bit of context:
I have loads of subdirectories under /home/user/logs. They all contains numerous log files that need to be injected in my elastic cloud database.

Here's what I've setup in a logstatsh-conf.conf

input {
file {
path => "/home/user/logs/*"
}
}

output {
elasticsearch { }

}

Unfortunately elastic cloud does not get populated.

Thanks

5

Read the file input documentation carefully, especially what's said about start_position and sincedb.

6

I'm actually more doubtful about that part:

7

Oh, that was literally what you had in your configuration? That won't work. See https://www.elastic.co/guide/en/cloud/current/logstash.html.

8

Thanks Magnus,

Got it configured as close as I thought it should be but now I get this nasty error.....I've checked possible syntax errors for hours..... maybe you would know what's happening? Thanks

[2018-03-26T10:52:31,671][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-03-26T10:52:31,705][DEBUG][logstash.outputs.elasticsearch] Normalizing http path {:path=>nil, :normalized=>nil}
[2018-03-26T10:52:31,997][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@bae0d0d5360f193f8d3584ef04595612.eu-central-1.aws.cloud.es.io:9243/]}}
[2018-03-26T10:52:31,999][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://elastic:xxxxxx@bae0d0d5360f193f8d3584ef04595612.eu-central-1.aws.cloud.es.io:9243/, :path=>"/"}
[2018-03-26T10:52:32,908][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://elastic:xxxxxx@bae0d0d5360f193f8d3584ef04595612.eu-central-1.aws.cloud.es.io:9243/"}
[2018-03-26T10:52:33,444][ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"#<LogStash::OutputDelegator:0x144646 @namespaced_metric=#<LogStash::Instrument::NamespacedMetric:0x1b85dc6 @metric=#<LogStash::Instrument::Metric:0x1b41dee @collector=#<LogStash::Instrument::Collector:0x8ef65d @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x6cbbfc @store=#<Concurrent::map:0x00000000000fb0 entries=4 default_proc=nil>, @structured_lookup_mutex=#Mutex:0x18d855a, @fast_lookup=#<Concurrent::map:0x00000000000fb4 entries=63 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"0094fcbe61344a8e74698cf70b12dbcb996aed4044c2e103744ffe2e73fb90cf"]>, @metric=#<LogStash::Instrument::NamespacedMetric:0x1de8464 @metric=#<LogStash::Instrument::Metric:0x1b41dee @collector=#<LogStash::Instrument::Collector:0x8ef65d @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x6cbbfc @store=#<Concurrent::map:0x00000000000fb0 entries=4 default_proc=nil>, @structured_lookup_mutex=#Mutex:0x18d855a, @fast_lookup=#<Concurrent::map:0x00000000000fb4 entries=63 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs]>, @out_counter=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: out value:0, @strategy=#<LogStash::OutputDelegatorStrategies::Shared:0x1749773 @output=<LogStash::Outputs::ElasticSearch hosts=>[https://bae0d0d5360f193f8d3584ef04595612.eu-central-1.aws.cloud.es.io:9243], user=>"elastic", password=>, index=>"myindex", document_type=>"psalogs", id=>"0094fcbe61344a8e74698cf70b12dbcb996aed4044c2e103744ffe2e73fb90cf", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_ec1e9f2e-7aa2-4a8f-afee-ac3f68fb1253", enable_metric=>true, charset=>"UTF-8">, workers=>1, manage_template=>true, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, action=>"index", ssl_certificate_verification=>true, sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false, ssl=>true>>, @in_counter=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: in value:0, @id="0094fcbe61344a8e74698cf70b12dbcb996aed4044c2e103744ffe2e73fb90cf", @time_metric=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: duration_in_millis value:0, @metric_events=#<LogStash::Instrument::NamespacedMetric:0xf10d15 @metric=#<LogStash::Instrument::Metric:0x1b41dee @collector=#<LogStash::Instrument::Collector:0x8ef65d @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x6cbbfc @store=#<Concurrent::map:0x00000000000fb0 entries=4 default_proc=nil>, @structured_lookup_mutex=#Mutex:0x18d855a, @fast_lookup=#<Concurrent::map:0x00000000000fb4 entries=63 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"0094fcbe61344a8e74698cf70b12dbcb996aed4044c2e103744ffe2e73fb90cf", :events]>, @output_class=LogStash::Outputs::ElasticSearch>", :error=>"Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')\n at [Source: (byte[])""; line: 1, column: 2]", :thread=>"#<Thread:0x5b0b20@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 run>"}
[2018-03-26T10:52:33,448][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<LogStash::Json::ParserError: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source: (byte[])"var hashRoute = '/app/kibana';
var defaultRoute = '/app/kibana';

9

var hash = window.location.hash;
if (hash.length) {
window.location = hashRoute + hash;
} else {
window.location = defaultRoute;
}"; line: 1, column: 2]>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/json.rb:17:in jruby_load'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:446:inget_es_version'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:248:in block in healthcheck!'", "org/jruby/RubyHash.java:1343:ineach'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:240:in healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:340:inupdate_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:70:in start'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:292:inbuild_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:60:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:101:increate_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:97:in build'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch.rb:230:inbuild_client'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:24:in register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:inregister'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:42:in register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:inregister_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:352:in block in register_plugins'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:352:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:735:inmaybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:362:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:289:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:249:in `block in start'"], :thread=>"#<Thread:0x5b0b20@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 run>"}

[2018-03-26T10:52:33,467][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: LogStash::PipelineAction::Create/pipeline_id:main, action_result: false", :backtrace=>nil}
[2018-03-26T10:52:33,471][DEBUG][logstash.instrument.periodicpoller.os] Stopping
[2018-03-26T10:52:33,472][DEBUG][logstash.instrument.periodicpoller.jvm] Stopping
[2018-03-26T10:52:33,472][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Stopping
[2018-03-26T10:52:33,472][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Stopping
[2018-03-26T10:52:33,478][DEBUG][logstash.agent ] Shutting down all pipelines {:pipelines_count=>0}
[2018-03-26T10:52:33,479][DEBUG][logstash.agent ] Converging pipelines state {:actions_count=>0}

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.