Hi all,
Newbie here.
I have kubernetes cluster in a location with patchy internet connection. I have metric beats on the cluster and I want to send metrics to ES which runs in cloud. I thought that when I put logstash between beats and ES, it will buffer events and retransmit them when ES is reachable again. However only metrics from the first 15 minutes are buffered (i.e. if connection drops at 10AM and restores at 11AM, only metrics between 10:15 and 11:00 are missing). Interestingly enough, the same happens when beats send metrics directly to ES. What can be the problem?
This is my logstash configuration:
apiVersion: v1
kind: ConfigMap
metadata:
name: logstash-configmap
data:
logstash.yml: |
http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline
logstash.conf: |
input {
beats {
port => 5044
}
}
filter {
}
output {
elasticsearch {
index => "metrics-%{[@metadata][beat]}"
hosts => [ "https://myesincloud.com:9200/" ]
user => "elastic"
password => "1234567890"
ssl => true
ssl_certificate_verification => false
cacert => "/etc/logstash/certificates/ca.crt"
}
}
This is how the pod is deployed:
apiVersion: v1
kind: Pod
metadata:
labels:
app: logstash
name: logstash
spec:
containers:
- image: docker.elastic.co/logstash/logstash:7.12.0
name: logstash
ports:
- containerPort: 25826
- containerPort: 5044
env:
- name: ES_HOSTS
value: "https://myesincloud.com:9200/"
- name: ES_USER
value: "elastic"
- name: ES_PASSWORD
value: "1234567890"
resources: {}
volumeMounts:
- name: config-volume
mountPath: /usr/share/logstash/config
- name: logstash-pipeline-volume
mountPath: /usr/share/logstash/pipeline
- name: cert-ca
mountPath: "/etc/logstash/certificates"
readOnly: true
restartPolicy: OnFailure
volumes:
- name: config-volume
configMap:
name: logstash-configmap
items:
- key: logstash.yml
path: logstash.yml
- name: logstash-pipeline-volume
configMap:
name: logstash-configmap
items:
- key: logstash.conf
path: logstash.conf
- name: cert-ca
secret:
secretName: myesincloud-es-http-certs-public
These are the only messages in logstash log
[ERROR] 2021-05-05 09:51:45.370 [[main]>worker0] elasticsearch - Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>64}
[WARN ] 2021-05-05 09:51:47.126 [Ruby-0-Thread-5: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:241] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@myesincloud.com:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@myesincloud.com:9200/][Manticore::ClientProtocolException] SSL peer shut down incorrectly"}
. . .
[WARN ] 2021-05-05 09:52:28.170 [Ruby-0-Thread-5: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:241] elasticsearch - Restored connection to ES instance {:url=>"https://elastic:xxxxxx@myesincloud.com:9200/"}
I appreciate your help.
Thank you.