Elastic Agent and index names

Elastic Stack Elastic Agent
1

I'm deploying the Elastic Agent in standalone on Kubernetes. I've got he default yaml file, but I'm having a hard time with the index naming that it creates. It is a little unclear to me from the documentation. Do the index names for Elastic Agent logs and metrics always need to start with logs- or metrics- ? I would like to override this if possible. For example in my test the only way I was able to change the data stream names was at the end of the name like this:

logs-kubernetes.container_logs-my-custom-namespace
metrics-kubernetes.pod-my-custom-namespace

What I would like is for them to have index names like this:
myteam-logs-kubernetes.container_logs
myteam-metrics-kubernetes.pod

Here are some small sections of my related elastic agent settings:

 - id: container-log-${kubernetes.pod.name}-${kubernetes.container.id}
        type: filestream
        use_output: default
        meta:
          package:
            name: kubernetes
            version: 1.52.0
        data_stream:
          namespace: my-custom-namespace
        streams:
          # Stream ID for this data stream allowing Filebeat to track the state of the ingested files. Must be unique.
          # Each filestream data stream creates a separate instance of the Filebeat filestream input.
          - id: container-log-${kubernetes.pod.name}-${kubernetes.container.id}
            data_stream:
              dataset: kubernetes.container_logs
              type: logs
 
 
    inputs:
      - id: kubernetes-cluster-metrics
        condition: ${kubernetes_leaderelection.leader} == true
        type: kubernetes/metrics
        use_output: default
        meta:
          package:
            name: kubernetes
            version: 1.52.0
        data_stream:
          namespace: my-custom-namespace
        streams:
          - data_stream:
              dataset: kubernetes.apiserver
              type: metrics

If I try changing something like type: metrics to type: myteam-metrics I start getting 400 errors in the elastic agent and nothing is indexed.

2

Yes, the data stream naming scheme for the elastic agent is the following:

<type>-<dataset>-<namespace>

the type can be logs, metrics, traces or synthetics, it will depending o what is being collected, the integration.

the dataset is based on the integration and the namespace is the only part that can be customized.

3

Hi,

in your case, you're trying to change the <type> part of the index name, which is not allowed because it's a fundamental part of how data streams work.

However, you can customize the <namespace> part of the index name to achieve a similar effect. For example, you could set the namespace to myteam-logs or myteam-metrics. This would result in index names like logs-kubernetes.container_logs-myteam-logs and metrics-kubernetes.pod-myteam-metrics.

Regards

4

Ok. Thanks for confirming. I’ll make do with just changing the namespace.

What about using our own index template for this? Would it be correct to clone the logs and metrics index template and then set a higher priority on it? So we can then use things like our own index lifecycle policies or customize fields for extra stuff?

On a side note it would be nice if there was an official helm chart for elastic agent to configure in standalone. I can’t seem to find anything for it.

Thanks!

5

You can create custom templates to add extra fields and use a different log retention, but depending on the number of integrations this adds a lot of work because this needs to be done in the dataset level, and some integrations have multiple datasets.

For example, the Google Workspace has something around 13 different datasets, if you wanted to add a field named custom.field for example, you would need to clone and manage 13 different custom templates, the same thing applies to custom ingest pipelines for example, but at least for ingest pipelines this is going to be easier on version 8.12 if I'm not wrong.

At the moment any customization in mappings and lifecycle policies requires a lot of work.

There is work being done to improve this, but no ETA when this will be available.

Yeah, I don't think you will find anything official by Elastic as they do not publish Helm Charts anymore.

6

I created two index templates, by copying the default logs a metrics ones, creating the pattern for the index to match our namespace and just set a higher priority of 105 on them. It seems to have applied my lifecycle policy fine. Is this a bad way of doing it? It seems to work ok. I’m not sure what integrations we will end up using in the end.

That is unfortunate about the helm chart. I also see the ones for filebeat and metricbeat are marked on GitHub as read only project and seem to be unmaintained.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.