In k8s, which archtecture is better?

Elastic Stack Beats
1

i have two options.

  1. multiple Web application pods, each WAS pods have a filebeat as a sidecar
    (if 3 WAS exists, than totally 3 Filebeats exists, totally 3 Pods)

  2. multiple Web application pods, only one filebeat pod as a main container
    (3 WAS exists, but only one Filebeat exists in another Pod without web application, totally 4 Pods)

in first option, i am worried about each filebeat race condition(filebeat.lock file) or resending logs(after restart pods or deploy new pods).

i know that file_identity.inode_marker or data.path options about filebeat.yml.
in this case, 2 options will be shared among filebeats.
(is this correct?)

I have 1 Persistence Volume, all WAS Pods will save log files in this PV's same file and logs will be rotated.
and all filebeat's data.path and file_identity.inode_marker.path properties shared in one file.

I'm configuring first option now, but suddenly thought second option.

logs must be sent without loss.
that's why i'm not save log files in each WAS Pods.

which option is better or general?
filebeat.yml will not be changed frequently.

filebeat version is 7.9.

if any other options better than above 2 options, please recommend to me.

2

If you can get on newer versions of Elastic stack, Elastic Agent provides a Daemonset and easy to configure integrations for collecting logs and metrics from Kubernetes.

It might be worth setting up a quick POC cluster with KIND or minikube so you can get a better Idea of how the agent based method works.

The elastic agent uses a Daemonset (one agent per node, not one agent per pod) so that's probably a safe place to start.

1 Like
3

thanks for your answer. i will consider your suggestion.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.