Team,
I'm not sure what I'm missing or I have broken here. I have an Elastic Server, Kibana Server, and Logstash Server. Elastic and Kibana are working. I'v setup one cisco switch to send logs to logstash and setup logstash to point to elastic. When I start Logstash I see the below error.
[INFO ] 2022-09-07 17:11:52.351 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9601, :ssl_enabled=>false}
[INFO ] 2022-09-07 17:11:53.266 [Converge PipelineAction::Create<main>] Reflections - Reflections took 106 ms to scan 1 urls, producing 125 keys and 434 values
[INFO ] 2022-09-07 17:11:54.048 [Converge PipelineAction::Create<main>] javapipeline - Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[INFO ] 2022-09-07 17:11:54.094 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://10.141.0.170:9200"]}
[INFO ] 2022-09-07 17:11:54.336 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@10.141.0.170:9200/]}}
[INFO ] 2022-09-07 17:11:54.630 [[main]-pipeline-manager] elasticsearch - Failed to perform request {:message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}
[WARN ] 2022-09-07 17:11:54.639 [[main]-pipeline-manager] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@10.141.0.170:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://10.141.0.170:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
[INFO ] 2022-09-07 17:11:54.701 [[main]-pipeline-manager] elasticsearch - Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[WARN ] 2022-09-07 17:11:54.703 [[main]-pipeline-manager] elasticsearch - Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[WARN ] 2022-09-07 17:11:54.713 [[main]-pipeline-manager] grok - ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[INFO ] 2022-09-07 17:11:54.996 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/syslog.conf"], :thread=>"#<Thread:0x5a26475 run>"}
[INFO ] 2022-09-07 17:11:55.821 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.82}
[INFO ] 2022-09-07 17:11:55.843 [[main]-pipeline-manager] beats - Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2022-09-07 17:11:55.857 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2022-09-07 17:11:55.940 [[main]<udp] udp - Starting UDP listener {:address=>"0.0.0.0:5051"}
[INFO ] 2022-09-07 17:11:55.986 [[main]<beats] Server - Starting server on port: 5044
[INFO ] 2022-09-07 17:11:55.988 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[ERROR] 2022-09-07 17:11:56.029 [[main]<udp] udp - UDP listener died {:exception=>#<Errno::EADDRINUSE: Address already in use - bind(2) for "0.0.0.0" port 5051>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:200:in `bind'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-udp-3.5.0/lib/logstash/inputs/udp.rb:129:in `udp_listener'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-udp-3.5.0/lib/logstash/inputs/udp.rb:81:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:410:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:401:in `block in start_input'"]}
[INFO ] 2022-09-07 17:11:59.717 [Ruby-0-Thread-9: :1] elasticsearch - Failed to perform request {:message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}
[WARN ] 2022-09-07 17:11:59.718 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@10.141.0.170:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://10.141.0.170:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
[INFO ] 2022-09-07 17:12:01.034 [[main]<udp] udp - Starting UDP listener {:address=>"0.0.0.0:5051"}
[ERROR] 2022-09-07 17:12:01.037 [[main]<udp] udp - UDP listener died {:exception=>#<Errno::EADDRINUSE: Address already in use - bind(2) for "0.0.0.0" port 5051>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:200:in `bind'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-udp-3.5.0/lib/logstash/inputs/udp.rb:129:in `udp_listener'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-udp-3.5.0/lib/logstash/inputs/udp.rb:81:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:410:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:401:in `block in start_input'"]}
[ERROR] 2022-09-07 17:12:04.733 [[main]<beats] javapipeline - A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Beats port=>5044, id=>"cc5b8bfb340abc5fa25fabb054b9c73e7523d8a2727a579e0dc920013efed4b0", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_22f118d0-4c1f-47ee-99c0-7582d75a804b", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", ssl=>false, ssl_verify_mode=>"none", ssl_peer_metadata=>false, include_codec_tag=>true, ssl_handshake_timeout=>10000, ssl_cipher_suites=>["TLS_AES_256_GCM_SHA384", "TLS_AES_128_GCM_SHA256", "TLS_CHACHA20_POLY1305_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256", "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], ssl_supported_protocols=>["TLSv1.2", "TLSv1.3"], client_inactivity_timeout=>60, executor_threads=>4, add_hostname=>false, tls_min_version=>1, tls_max_version=>1.3>
Error: Address already in use
Exception: Java::JavaNet::BindException
Stack: sun.nio.ch.Net.bind0(Native Method)
sun.nio.ch.Net.bind(sun/nio/ch/Net.java:555)
sun.nio.ch.ServerSocketChannelImpl.netBind(sun/nio/ch/ServerSocketChannelImpl.java:337)
sun.nio.ch.ServerSocketChannelImpl.bind(sun/nio/ch/ServerSocketChannelImpl.java:294)
io.netty.channel.socket.nio.NioServerSocketChannel.doBind(io/netty/channel/socket/nio/NioServerSocketChannel.java:134)
io.netty.channel.AbstractChannel$AbstractUnsafe.bind(io/netty/channel/AbstractChannel.java:562)
io.netty.channel.DefaultChannelPipeline$HeadContext.bind(io/netty/channel/DefaultChannelPipeline.java:1334)
io.netty.channel.AbstractChannelHandlerContext.invokeBind(io/netty/channel/AbstractChannelHandlerContext.java:506)
io.netty.channel.AbstractChannelHandlerContext.bind(io/netty/channel/AbstractChannelHandlerContext.java:491)
io.netty.channel.DefaultChannelPipeline.bind(io/netty/channel/DefaultChannelPipeline.java:973)
io.netty.channel.AbstractChannel.bind(io/netty/channel/AbstractChannel.java:260)
io.netty.bootstrap.AbstractBootstrap$2.run(io/netty/bootstrap/AbstractBootstrap.java:356)
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(io/netty/util/concurrent/AbstractEventExecutor.java:164)
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(io/netty/util/concurrent/SingleThreadEventExecutor.java:472)
io.netty.channel.nio.NioEventLoop.run(io/netty/channel/nio/NioEventLoop.java:500)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(io/netty/util/concurrent/SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(io/netty/util/internal/ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(io/netty/util/concurrent/FastThreadLocalRunnable.java:30)
java.lang.Thread.run(java/lang/Thread.java:833)
Here is the syslog.conf in logstash:
input {
# tcp {
# port => 5050
# type => syslo
# }
udp {
port => 5051
type => syslog
}
beats {
port => 5044
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch { hosts => ["https://10.141.0.170:9200"]
user => ["elastic"]
password => ["password"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}
Cisco is basically send all syslogs to log stash IP 10.141.0.172 port 5051.
I setup metricbeats on the the elastic server and here is the yml file:
# ------------------------------ Logstash Output -------------------------------
output.logstash:
# The Logstash hosts
hosts: ["http://10.141.0.172:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
Thoughts on what I'm missing?