Our situation is as follows: We have a Kubernetes Cluster with a lot of logs piling up. We would like to get a centralized place for the logs to query through. This centralized place is a local VM, which has Elasticsearch, Kibana, Logstash and Filebeat installed in one place.
The kubernetes cluster (with IP 192.168.253.160) sends the logs to the IP of the local VM (192.168.253.150) and the port of local Elasticsearch (which is 9200). This all works fine and I can see the logs being send to Elasticsearch using Kibana.
Only the problem here is that these logs don't go through Logstash and/or Filebeat, because if I try to put a filter on the Logstash configuration, nothing happens to the logs. Even if I stop Logstash and Filebeat completely, the logs still get send to Elasticsearch no problem.
I'm pretty sure the logs don't need to be send to Elasticsearch port, but when I try to send them to the ports 9600 (which is Logstash) or 5044 (which is Filebeat) the logs don't go through and I can't see them come into Kibana anymore. Maybe I need the use of the HTTP API?
Below I'll provide the changes I made in the configs of Elasticsearch, Logstash and Filebeat:
Elasticsearch:
nano /etc/elasticsearch/elasticsearch.yml
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
Filebeat:
nano /etc/filebeat/filebeat.yml
# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
# Array of hosts to connect to.
#hosts: ["localhost:9200"]
# Protocol - either `http` (default) or `https`.
#protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
#username: "elastic"
#password: "changeme"
# ------------------------------ Logstash Output -------------------------------
output.logstash:
# The Logstash hosts
hosts: ["0.0.0.0:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
Logstash:
nano /etc/logstash/conf.d/02-beats-input.conf
input {
beats {
port => 5044
}
}
output {
stdout { codec => rubydebug }
}
Also, I'm very new to all of this and IT in general, so please make it a bit noob friendly. Thanks!