# Elastic Cloud on Kubernetes - Persist Fleet Server State

**URL:** <https://discuss.elastic.co/t/elastic-cloud-on-kubernetes-persist-fleet-server-state/377892>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Tags:** fleet\
**Created:** [May 7, 2025, 9:25am UTC](https://discuss.elastic.co/t/elastic-cloud-on-kubernetes-persist-fleet-server-state/377892 "2025-05-07T09:25:52Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![wederm](https://avatars.discourse-cdn.com/v4/letter/w/bbe5ce/32.png) [@wederm](https://discuss.elastic.co/u/wederm)\
**Post date:** [May 7, 2025, 9:25am UTC](https://discuss.elastic.co/t/elastic-cloud-on-kubernetes-persist-fleet-server-state/377892/1 "2025-05-07T09:25:52Z")

</div>

Hello all

Maybe someone can help me because I am at my wits end and I have not found anything that covers this case in the documentation.

I am currently setting up an elastic stack on on-prem kubernetes and my issue is that I cannot persist the state of the fleet server. Meaning if I delete my fleet server pod (to simulate a version update) a new fleet server gets added in kibana and the "old" fleet server goes offline.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/e/cea7dda25602adb265c9e183373f65f5d46e918e.png)

In my docker compose setup I could mount a volume at `/usr/share/elastic-agent/state` to keep the state, but this is not working in the kubernetes setup.

I used the statefulSet deployment method and overwrite the `agent-data` volume to use the persistent volume claim: (partial values.yml for the helm chart below)

```auto
eck-fleet-server:
  enabled: true
  fullnameOverride: "fleet-server"
  version: 8.18.0
  serviceAccountName: fleet-server
  statefulSet:
    replicas: 1
    podTemplate:
      spec:
        volumes:
          - name: agent-data
            persistentVolumeClaim:
              claimName: agent-data
    volumeClaimTemplates:
      - metadata:
          name: agent-data
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 500Mi
          storageClassName: ceph-block
  policyID: elk-fleet-server
  kibanaRef:
    name: kibana
  elasticsearchRefs:
  - name: elasticsearch

```

This setup results (at least in my opinion) in a correct pod creation. (abbreviated `kubectl describe` below)

```auto
...
Containers:
  agent:
    Container ID: containerd://...
    Image: docker.elastic.co/beats/elastic-agent:8.18.0
    Image ID: docker.elastic.co/beats/elastic-agent@sha256:c26375e5870b1efa211b5820645346784dc7a08e687314fe739aba294647ab5f
    Port: 8220/TCP
    Host Port: 0/TCP
    State: Running
      Started: Wed, 07 May 2025 10:53:46 +0200
    Ready: True
...
    Mounts:
      /etc/agent.yml from config (ro,path="agent.yml")
      /mnt/elastic-internal/elasticsearch-association/elk-test/elasticsearch/certs from elasticsearch-certs (ro)
      /usr/share/elastic-agent/state from agent-data (rw)
      /usr/share/fleet-server/config/http-certs from fleet-certs (ro)
Volumes:
  agent-data:
    Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName: agent-data-fleet-server-agent-0
    ReadOnly: false
...

```

I have also tried the setup with:

- Creating a new volume and mounting at `/usr/share/elastic-agent/state` -\> no change
- Trying to store `/usr/share/elastic-agent` in a volume -\> container failed to boot

Am I doing something wrong, have I overlooked something in the documentation or is this even possible to achieve?

Thanks in advance

---

<div class="post-metadata">

**Author:** ![prolle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prolle/32/114049_2.png) [@prolle](https://discuss.elastic.co/u/prolle)\
**Post date:** [May 12, 2025, 1:00pm UTC](https://discuss.elastic.co/t/elastic-cloud-on-kubernetes-persist-fleet-server-state/377892/2 "2025-05-12T13:00:59Z")

</div>

i'm having the same problem on docker, with compose. It worked on 8.17.4 and .6 when upgraded to 8.18.1 this problem started to happen. Every time i restart the compose the fleet server gets a new "id". Have you tried with 8.18.X on compose? Or did you use a older version? It may be something introduced in the newest version?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [May 12, 2025, 1:33pm UTC](https://discuss.elastic.co/t/elastic-cloud-on-kubernetes-persist-fleet-server-state/377892/3 "2025-05-12T13:33:07Z")

</div>

I do not use ECK, but on the cloud this is how it works, for example, when you upgrade or change anything on the integration server, it will spin-up a new fleet server.

Maybe this is by default? The default policy also has a small inactivity time of just 24 hours, so offline agents will not appear for long.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [May 12, 2025, 1:34pm UTC](https://discuss.elastic.co/t/elastic-cloud-on-kubernetes-persist-fleet-server-state/377892/4 "2025-05-12T13:34:05Z")

</div>

> [@prolle](#):
>
> i'm having the same problem on docker, with compose. It worked on 8.17.4 and .6 when upgraded to 8.18.1 this problem started to happen. Every time i restart the compose the fleet server gets a new "id". Have you tried with 8.18.X on compose?

This seems unrelated as this is about running on ECK, not docker compose.

I suggest that you open another topic and share your docker-compose and the issue you are having.

---

<div class="post-metadata">

**Author:** ![wederm](https://avatars.discourse-cdn.com/v4/letter/w/bbe5ce/32.png) [@wederm](https://discuss.elastic.co/u/wederm)\
**Post date:** [May 12, 2025, 4:30pm UTC](https://discuss.elastic.co/t/elastic-cloud-on-kubernetes-persist-fleet-server-state/377892/5 "2025-05-12T16:30:59Z")

</div>

You are right it is probably working as intended. I have checked the agent policy UI on kibana and the inactive agent unenrollment timeout specifically states:

> ... This can be useful for policies containing ephemeral agents, such as those in a Docker or Kubernetes environment.

---

<div class="post-metadata">

**Author:** ![wederm](https://avatars.discourse-cdn.com/v4/letter/w/bbe5ce/32.png) [@wederm](https://discuss.elastic.co/u/wederm)\
**Post date:** [May 12, 2025, 4:40pm UTC](https://discuss.elastic.co/t/elastic-cloud-on-kubernetes-persist-fleet-server-state/377892/6 "2025-05-12T16:40:21Z")

</div>

If anyone stumbles across this post.  
I have updated my fleet agent policy with `inactivity_timeout` and `unenroll_timeout`, to keep my agent list clean.

```yaml
xpack.fleet.agentPolicies:
    - name: Fleet Server policy
      id: elk-fleet-server
      namespace: default
      is_managed: true
      monitoring_enabled:
      - logs
      - metrics
      inactivity_timeout: 900 # remove from UI after 15min
      unenroll_timeout: 86400 # unenroll agent after 24h
      package_policies:
      - name: fleet_server-1
        id: fleet_server-1
        package:
          name: fleet_server

```