Scale Logstash on Openshift

Hello,

I am running Logstash 6.4.3 on Openshift with only 1 Pod atm.
I actually added quite a few servers within the last couple of days which all send logs via Filebeat to my Logstash instance. Today I realized that my Logstash can't keep up any longer. It ran out of memory and also some logs weren't being processed in time. So I tried scaling Logstash by creating another Pod, but that second Pod doesn't seem to process any workload whatsoever, the metrics show very low resource consumption on one instance whereas the other one runs out of em.

As far as I know, a service should be able to load balance the traffic to each pod evenly
Am I missing anything here?

Deployment Config

apiVersion: v1
kind: Template
metadata:
  name: logstash
parameters:
- name: ENVIRONMENT
  description: The name of the Environment eg. qa1, feature-XY-dev, dev5,
  value: dev #default value
  required: true

- name: IMAGE_TAG
  description: The tag of the image to deploy
  value: dev
  required: true

# The following values are usually not set as parameter, they are only changed at this position
- name: APPLICATION_NAME
  description: The name of the application. All Resources are prefixed with this name
  value: logstash
  required: false

objects:
- apiVersion: v1
  kind: ImageStream
  metadata:
	name: ${APPLICATION_NAME}-${ENVIRONMENT}
	labels:
	  app: ${APPLICATION_NAME}-${ENVIRONMENT}
  spec:
	tags:
	- name: ${IMAGE_TAG}
	  from:
		kind: DockerImage
		name: custom-logstash:${IMAGE_TAG}
	  importPolicy: {}
	  referencePolicy:
		type: Source

- apiVersion: v1
  kind: DeploymentConfig
  metadata:
	name: ${APPLICATION_NAME}-${ENVIRONMENT}
	labels:
	  app: ${APPLICATION_NAME}-${ENVIRONMENT}
  spec:
	replicas: 1
	strategy:
	  activeDeadlineSeconds: 21600
	  type: Rolling
	triggers:
	- imageChangeParams:
		automatic: true
		containerNames:
		- ${APPLICATION_NAME}-${ENVIRONMENT}
		from:
		  kind: ImageStreamTag
		  name: ${APPLICATION_NAME}-${ENVIRONMENT}:${IMAGE_TAG}
	  type: ImageChange
	selector:
	  app: ${APPLICATION_NAME}-${ENVIRONMENT}
	  deploymentconfig: ${APPLICATION_NAME}-${ENVIRONMENT}
	template:
	  metadata:
		labels:
		  app: ${APPLICATION_NAME}-${ENVIRONMENT}
		  deploymentconfig: ${APPLICATION_NAME}-${ENVIRONMENT}
	  spec:
		restartPolicy: Always
		containers: 
		- name: ${APPLICATION_NAME}-${ENVIRONMENT}
		  # Wenn es ImageChangeTrigger gibt dann muss der Image name leer sein sonst werden unabsichtlich deployments ausgelöst
		  image:
		  imagePullPolicy: Always
		  ports:
		  - name: monitoringapi
			containerPort: 9600
			protocol: TCP
		  - name: filebeat
			containerPort: 5044
			protocol: TCP
		  env:
		  readinessProbe:
			httpGet:
			  path: /
			  port: 9600
			  scheme: HTTP
			initialDelaySeconds: 140
			periodSeconds: 10
			failureThreshold: 3
			successThreshold: 1
			timeoutSeconds: 5
		  livenessProbe:
			httpGet:
			  path: /
			  port: 9600
			  scheme: HTTP
			initialDelaySeconds: 140
			periodSeconds: 10
			failureThreshold: 3
			successThreshold: 1
			timeoutSeconds: 5
		  resources:
			limits:
			  cpu: 700m
			  memory: 1500Mi
			requests:
			  cpu: 100m
			  memory: 500Mi


- apiVersion: v1
  kind: Service
  metadata:
	labels:
	  app: ${APPLICATION_NAME}-${ENVIRONMENT}
	name: ${APPLICATION_NAME}-${ENVIRONMENT}-monitoringapi
  spec:
	ports:
	- name: monitoringapi-9600
	  port: 9600
	  protocol: TCP
	  targetPort: 9600
	selector:
	  deploymentconfig: ${APPLICATION_NAME}-${ENVIRONMENT}
	type: ClusterIP

- apiVersion: v1
  kind: Service
  metadata:
	labels:
	  app: ${APPLICATION_NAME}-${ENVIRONMENT}
	name: ${APPLICATION_NAME}-${ENVIRONMENT}-filebeat
  spec:
	ports:
	- name: filebeat-443
	  port: 443
	  protocol: TCP
	  targetPort: 5044
	selector:
	  deploymentconfig: ${APPLICATION_NAME}-${ENVIRONMENT}
	type: ClusterIP

- apiVersion: v1
  kind: Route
  metadata:
	name: ${APPLICATION_NAME}-${ENVIRONMENT}-monitoringapi
	labels:
	  app: ${APPLICATION_NAME}-${ENVIRONMENT}
  spec:
	host: ${APPLICATION_NAME}-${ENVIRONMENT}-monitoringapi.int.ocp.cloud
	port:
	  targetPort: 9600
	tls:
	  termination: edge
	to:
	  kind: Service
	  name: ${APPLICATION_NAME}-${ENVIRONMENT}-monitoringapi

- apiVersion: v1
  kind: Route
  metadata:
	name: ${APPLICATION_NAME}-${ENVIRONMENT}-filebeat
	labels:
	  app: ${APPLICATION_NAME}-${ENVIRONMENT}
  spec:
	host: ${APPLICATION_NAME}-${ENVIRONMENT}-filebeat.int.ocp.cloud
	port:
	  targetPort: 5044
	tls:
	  termination: passthrough
	to:
	  kind: Service
	  name: ${APPLICATION_NAME}-${ENVIRONMENT}-filebeat

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.