Logstash 5.6.3 setup with Filebeat and Kibana on CentOS 7

[Jens_van_der_Wateren]

I've been trying to experiment with Elasticsearch and the Elastic Cloud because I might want to use it for the company I currently work for. I have a 2 week trial, but I'm afraid I've not yet come very far because I cannot manage to get Logstash running without issues. This is my logstash.yml (mostly just default settings):

Pastebin logstash.yml

I want to use Filebeat to send apache logs realtime to the Elastic Cloud so I can analyze them. My logstash.conf looks like this:

Summary

input {
beats {
host => "127.0.0.1"
port => 5044
}
}

The filter part of this file is commented out to indicate that it is

optional.

filter {

}

output {
elasticsearch {
hosts => ["https://749ef3120c86cdbf58c4c002f2344b86.eu-west-1.aws.found.io:9243"]
user => "Jens"
password => "pass"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

The pipeline is never actually created. Below is the error which I think is causing all this trouble, but I have honestly no idea how to fix it (or what the cause is).

Summary

[2017-10-19T15:05:35,247][ERROR][logstash.pipeline ] Error registering plugin {:plugin=>"#<LogStash::OutputDelegator:0x2d46745c @namespaced_metric=#<LogStash::Instrument::NamespacedMetric:0x3fb113ac @metric=#<LogStash::Instrument::Metric:0x39c77192 @collector=#<LogStash::Instrument::Collector:0x5a4ad3ea @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x4430a86b @store=#<Concurrent:0x000000000644a8 entries=3 default_proc=nil>, @structured_lookup_mutex=#Mutex:0x732bef32, @fast_lookup=#<Concurrent:0x000000000644ac entries=61 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"8f2791f16b92257ba930f259763a80cdb7570108-2"]>, @metric=#<LogStash::Instrument::NamespacedMetric:0x7dbb1361 @metric=#<LogStash::Instrument::Metric:0x39c77192 @collector=#<LogStash::Instrument::Collector:0x5a4ad3ea @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x4430a86b @store=#<Concurrent:0x000000000644a8 entries=3 default_proc=nil>, @structured_lookup_mutex=#Mutex:0x732bef32, @fast_lookup=#<Concurrent:0x000000000644ac entries=61 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs]>, @logger=#<LogStash::Logging::Logger:0x1077b591 @logger=#Java::OrgApacheLoggingLog4jCore::Logger:0x24b8860e>, @out_counter=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"8f2791f16b92257ba930f259763a80cdb7570108-2", :events] key: out value: 0, @in_counter=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"8f2791f16b92257ba930f259763a80cdb7570108-2", :events] key: in value: 0, @strategy=#<LogStash::OutputDelegatorStrategies::Shared:0x26295cb0 @output=<LogStash::Outputs::ElasticSearch hosts=>[https://749ef3120c86cdbf58c4c002f2344b86.eu-west-1.aws.found.io:9243], user=>"Jens", password=>, manage_template=>false, index=>"%{[@metadata][beat]}-%{+YYYY.MM.dd}", document_type=>"%{[@metadata][type]}", id=>"8f2791f16b92257ba930f259763a80cdb7570108-2", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_ac1228ed-ffe4-4565-aa96-c38800fd7d43", enable_metric=>true, charset=>"UTF-8">, workers=>1, template_name=>"logstash", template_overwrite=>false, idle_flush_time=>1, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, action=>"index", ssl_certificate_verification=>true, sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false, ssl=>true>>, @id="8f2791f16b92257ba930f259763a80cdb7570108-2", @time_metric=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"8f2791f16b92257ba930f259763a80cdb7570108-2", :events] key: duration_in_millis value: 0, @metric_events=#<LogStash::Instrument::NamespacedMetric:0xb5fbed9 @metric=#<LogStash::Instrument::Metric:0x39c77192 @collector=#<LogStash::Instrument::Collector:0x5a4ad3ea @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x4430a86b @store=#<Concurrent:0x000000000644a8 entries=3 default_proc=nil>, @structured_lookup_mutex=#Mutex:0x732bef32, @fast_lookup=#<Concurrent:0x000000000644ac entries=61 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"8f2791f16b92257ba930f259763a80cdb7570108-2", :events]>, @output_class=LogStash::Outputs::ElasticSearch>", :error=>"Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')\n at [Source: (byte[])""; line: 1, column: 2]"}
[2017-10-19T15:05:35,254][ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>#<LogStash::Json::ParserError: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')

This line comes from my logstash-plain.log - the following is what happens over and over again because Logstash is trying to start again after it has stopped.

The entire log file can be visited here: logstash-plain.log

I'm entirely out of ideas. I've looked at the documentation, Googled the error messages... I have no clue why it's not working.

Thanks in advance

Jens