I have been asking tons of questions and have been slowly making my way through the final part of my ELK SIEM installation. I am running a small environment with Elasticsearch, Logstash and Kibana installed on seperate RHEL servers. I have opened the ports so that each system can talk to one another and send information. Where I am stuck at the moment, and have been for some time is getting syslogs to be passed to Elasticsearch so they can be viewed and assessed in Kibana.
My network team has formatted the internal firewall and an ntp server to send their syslogs directly to the Logstash server over UDP port 514. I have forwarded that 514 port to a higher port as I am a SUDO user and not ROOT. I have tested using a tcpdump -n -vv host HOST_IP_ADD and I am seeing that the syslogs are being sent and received on my Logstash server.
If I go to either my Elasticsearch or Kibana server and run an echo 'hello world' | nc Logstash_IP port I can see the hello world in my Kibana under the syslog index I created in my logstash.conf file. Below is my logstash.conf file, which is located in the /etc/logstash directory.
input {
udp {
port => port#
type => syslog
}
tcp {
port => port#
type => syslog
}
}
output {
elasticsearch {
hosts => ["es_ip_addresss:port"]
user => "elastic_user"
password => "elastic_password"]
index => "syslog"-%{+YYYY.MM.dd}"
}
}
If I run a test of my logstash.conf file from /usr/share/logstash and running
bin/logstash --config.test_and_exit -f /etc/logstash/logstash.conf: It spits out that it is OK.
I am not sure what else I can do to make sure that it will do what I intend it to do. Which is read the syslogs that it is being sent and send them to Elasticsearch and the I can access them in kibana.
Also if I look in Kibana under Security --> Hosts --> filter down to source.ip and the ip of the ntp server I see that traffic is flowing into my Elasticsearch via the use of packetbeat.
Thank you for any and all assistance you can provide.
I have read through a lot of the other peoples questions regarding logstash.conf as well and I notice that when they add their output they tend to use Localhost:9200 instead of the Elasticsearch IP address. Is that correct?
As I am running seperate systems for the three layers of ELK, would I not want to specify the output to Elasticsearch?
when the Logstash server is operating properly again, and I make that change to my output in the logstash.conf file, what would I be looking for in terms of messages?
You would be looking for an output straight from your input.
The reason behind doing this is right now you have an input and an output. By what you are saying I can't determine which is the problem. Setting the output to stdout and if you see messages then we know the input is good and can move to the output.
The issue is still with your input and I would focus on that until you see messages using stdout. When you start Logstash you should see all the info, warn, error messages. Can you post that? That should contain a clue on what's going wrong.
As an aside, I just sent a hello world from my kibana server to the Logstash server on the udp syslog port and it went through and I see it in my Kibana console under the Discover panel.
And More snippets of log after the change in the output.
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_0_dot_10_minus_java.lib.logstash.inputs.tcp.RUBY$method$register$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.10-java/lib/logstash/inputs/tcp.rb:154) [jruby-complete-9.2.16.0.jar:?]
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_0_dot_10_minus_java.lib.logstash.inputs.tcp.RUBY$method$register$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.10-java/lib/logstash/inputs/tcp.rb) [jruby-complete-9.2.16.0.jar:?]
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80) [jruby-complete-9.2.16.0.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70) [jruby-complete-9.2.16.0.jar:?]
at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207) [jruby-complete-9.2.16.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$register_plugins$1(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228) [jruby-complete-9.2.16.0.jar:?]
at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:148) [jruby-complete-9.2.16.0.jar:?]
at org.jruby.runtime.BlockBody.yield(BlockBody.java:106) [jruby-complete-9.2.16.0.jar:?]
at org.jruby.runtime.Block.yield(Block.java:184) [jruby-complete-9.2.16.0.jar:?]
at org.jruby.RubyArray.each(RubyArray.java:1809) [jruby-complete-9.2.16.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$register_plugins$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227) [jruby-complete-9.2.16.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$register_plugins$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb) [jruby-complete-9.2.16.0.jar:?]
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80) [jruby-complete-9.2.16.0.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70) [jruby-complete-9.2.16.0.jar:?]
at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207) [jruby-complete-9.2.16.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$start_inputs$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:386) [jruby-complete-9.2.16.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$start_inputs$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb) [jruby-complete-9.2.16.0.jar:?]
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80) [jruby-complete-9.2.16.0.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70) [jruby-complete-9.2.16.0.jar:?]
at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207) [jruby-complete-9.2.16.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$start_workers$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:311) [jruby-complete-9.2.16.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$start_workers$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb) [jruby-complete-9.2.16.0.jar:?]
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80) [jruby-complete-9.2.16.0.jar:?]
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.