Hello,
I am running Logstash 6.4.3 on Openshift with only 1 Pod atm.
I actually added quite a few servers within the last couple of days which all send logs via Filebeat to my Logstash instance. Today I realized that my Logstash can't keep up any longer. It ran out of memory and also some logs weren't being processed in time. So I tried scaling Logstash by creating another Pod, but that second Pod doesn't seem to process any workload whatsoever, the metrics show very low resource consumption on one instance whereas the other one runs out of em.
As far as I know, a service should be able to load balance the traffic to each pod evenly
Am I missing anything here?
Deployment Config
apiVersion: v1
kind: Template
metadata:
name: logstash
parameters:
- name: ENVIRONMENT
description: The name of the Environment eg. qa1, feature-XY-dev, dev5,
value: dev #default value
required: true
- name: IMAGE_TAG
description: The tag of the image to deploy
value: dev
required: true
# The following values are usually not set as parameter, they are only changed at this position
- name: APPLICATION_NAME
description: The name of the application. All Resources are prefixed with this name
value: logstash
required: false
objects:
- apiVersion: v1
kind: ImageStream
metadata:
name: ${APPLICATION_NAME}-${ENVIRONMENT}
labels:
app: ${APPLICATION_NAME}-${ENVIRONMENT}
spec:
tags:
- name: ${IMAGE_TAG}
from:
kind: DockerImage
name: custom-logstash:${IMAGE_TAG}
importPolicy: {}
referencePolicy:
type: Source
- apiVersion: v1
kind: DeploymentConfig
metadata:
name: ${APPLICATION_NAME}-${ENVIRONMENT}
labels:
app: ${APPLICATION_NAME}-${ENVIRONMENT}
spec:
replicas: 1
strategy:
activeDeadlineSeconds: 21600
type: Rolling
triggers:
- imageChangeParams:
automatic: true
containerNames:
- ${APPLICATION_NAME}-${ENVIRONMENT}
from:
kind: ImageStreamTag
name: ${APPLICATION_NAME}-${ENVIRONMENT}:${IMAGE_TAG}
type: ImageChange
selector:
app: ${APPLICATION_NAME}-${ENVIRONMENT}
deploymentconfig: ${APPLICATION_NAME}-${ENVIRONMENT}
template:
metadata:
labels:
app: ${APPLICATION_NAME}-${ENVIRONMENT}
deploymentconfig: ${APPLICATION_NAME}-${ENVIRONMENT}
spec:
restartPolicy: Always
containers:
- name: ${APPLICATION_NAME}-${ENVIRONMENT}
# Wenn es ImageChangeTrigger gibt dann muss der Image name leer sein sonst werden unabsichtlich deployments ausgelöst
image:
imagePullPolicy: Always
ports:
- name: monitoringapi
containerPort: 9600
protocol: TCP
- name: filebeat
containerPort: 5044
protocol: TCP
env:
readinessProbe:
httpGet:
path: /
port: 9600
scheme: HTTP
initialDelaySeconds: 140
periodSeconds: 10
failureThreshold: 3
successThreshold: 1
timeoutSeconds: 5
livenessProbe:
httpGet:
path: /
port: 9600
scheme: HTTP
initialDelaySeconds: 140
periodSeconds: 10
failureThreshold: 3
successThreshold: 1
timeoutSeconds: 5
resources:
limits:
cpu: 700m
memory: 1500Mi
requests:
cpu: 100m
memory: 500Mi
- apiVersion: v1
kind: Service
metadata:
labels:
app: ${APPLICATION_NAME}-${ENVIRONMENT}
name: ${APPLICATION_NAME}-${ENVIRONMENT}-monitoringapi
spec:
ports:
- name: monitoringapi-9600
port: 9600
protocol: TCP
targetPort: 9600
selector:
deploymentconfig: ${APPLICATION_NAME}-${ENVIRONMENT}
type: ClusterIP
- apiVersion: v1
kind: Service
metadata:
labels:
app: ${APPLICATION_NAME}-${ENVIRONMENT}
name: ${APPLICATION_NAME}-${ENVIRONMENT}-filebeat
spec:
ports:
- name: filebeat-443
port: 443
protocol: TCP
targetPort: 5044
selector:
deploymentconfig: ${APPLICATION_NAME}-${ENVIRONMENT}
type: ClusterIP
- apiVersion: v1
kind: Route
metadata:
name: ${APPLICATION_NAME}-${ENVIRONMENT}-monitoringapi
labels:
app: ${APPLICATION_NAME}-${ENVIRONMENT}
spec:
host: ${APPLICATION_NAME}-${ENVIRONMENT}-monitoringapi.int.ocp.cloud
port:
targetPort: 9600
tls:
termination: edge
to:
kind: Service
name: ${APPLICATION_NAME}-${ENVIRONMENT}-monitoringapi
- apiVersion: v1
kind: Route
metadata:
name: ${APPLICATION_NAME}-${ENVIRONMENT}-filebeat
labels:
app: ${APPLICATION_NAME}-${ENVIRONMENT}
spec:
host: ${APPLICATION_NAME}-${ENVIRONMENT}-filebeat.int.ocp.cloud
port:
targetPort: 5044
tls:
termination: passthrough
to:
kind: Service
name: ${APPLICATION_NAME}-${ENVIRONMENT}-filebeat