Hey guys,
My environment - Dev
- Master nodes (Elasticsearch & Logstash are installed) x 2
- Kibana node (Only Kibana) x 1
- All m servers are on CentOS7
Before you ask
- I can reach my master nodes
- Both nodes see eachother
- I can access Kibana
What I am trying to achieve
Having syslog from Elasticsearch servers sent to my ELK. Why ? To understand how it works and the output from my configuration.
That being said, launching logstash results in a fail so is my filebeat.
Config files
Logstash conf
Path: /etc/logstash/conf.d/logstash-syslog.conf
input {
beats { port => "5044" }
}filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}}
output {
elasticsearch {
hosts => ["192.168.131.134:9200","192.168.131.136:9200"]
}
stdout {
codec => rubydebug
}
}
Pipeline conf
pipeline.id: main
path.config: "/etc/logstash/conf.d/*.conf"pipeline.id: filebeat
queue.type: persisted
path.config: "/etc/logstash/conf.d/logstash-syslog.conf"
Filebeat conf
filebeat.prospectors:
- input_type: log
enabled: true
paths:
- /var/log/messages
- /var/log/*.log
- /var/log/syslog
setup.kibana:
host: "192.168.131.135:5601"output.logstash:
hosts: ["192.168.131.134:5044", "192.168.131.136:5044"]
Error facing
> -- The start-up result is done.
> Sep 05 11:19:48 elk-prod-1 logstash[4986]: Thread.exclusive is deprecated, use Thread::Mutex
> Sep 05 11:19:52 elk-prod-1 logstash[4986]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
> Sep 05 11:19:54 elk-prod-1 logstash[4986]: [2019-09-05T11:19:54,927][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.3.1"}
> Sep 05 11:19:56 elk-prod-1 logstash[4986]: [2019-09-05T11:19:56,880][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 11, column 1 (byte 125) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}
> Sep 05 11:19:56 elk-prod-1 logstash[4986]: [2019-09-05T11:19:56,875][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:filebeat, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 11, column 1 (byte 125) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}
> Sep 05 11:19:57 elk-prod-1 logstash[4986]: [2019-09-05T11:19:57,354][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
> Sep 05 11:20:02 elk-prod-1 logstash[4986]: [2019-09-05T11:20:02,066][INFO ][logstash.runner ] Logstash shut down.
> Sep 05 11:20:02 elk-prod-1 systemd[1]: logstash.service: main process exited, code=exited, status=1/FAILURE
> Sep 05 11:20:02 elk-prod-1 systemd[1]: Unit logstash.service entered failed state.
> Sep 05 11:20:02 elk-prod-1 systemd[1]: logstash.service failed.
> Sep 05 11:20:02 elk-prod-1 systemd[1]: logstash.service holdoff time over, scheduling restart.
> Sep 05 11:20:02 elk-prod-1 systemd[1]: Stopped logstash.
> -- Subject: Unit logstash.service has finished shutting down
> -- Defined-By: systemd
> -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
> --
> -- Unit logstash.service has finished shutting down.