Fleet managed Elastic Agent environment/deployment name? custom field?

Elastic Stack Beats
,
1

Hi

I was wondering if there is a way to pass parameters to elastic agents when enrolling happens, to differentiate logs pushed from different environments.

On the other hand they are some configuration that can be set with hostnamectl that is for this purpose, I wonder that could be a feature for filebeat and elastic agent to push next to other host.* details.
https://www.freedesktop.org/software/systemd/man/hostnamectl.html#deployment%20[ENVIRONMENT]
Thanks
Mohsen

2

There is no such option at the moment. Can you share some examples on what values you would like to set?

3

the values could be " development ", " integration ", " staging ", " production ".

4

You can kind of do this with the existing tools by:
Using a policy for each environment, and set the namespace for that policy to the environment name. This would separate the data at the index level, but you can also filter via the data_stream.namespace field value as well. (This doesn't exactly work for somethings, I believe the osquery management module needs to use the default namespace, but I think this is the only integration with this limitation).

This is technically the purpose of the namespace value according to ECS

5

That does not scale very well, when there are more than 10 environments, keeping the configurations consistent across policies is a pain.
The use case here is only to differentiate streams pushed from different environments.

1 Like
6

We likely need to separate two different "use cases" here. If we talk about "staging", "production" the idea is indeed to use data streams. This also allows to easily give different access permissions per namespace.

And then there is "tags/labels" and additional host details which might be different per host. These likely should be set on the Elastic Agent and not in the policy itself. Adding metadata per agent is something we discussed in the past but did not implement yet.

@mohsen0 One thing I'm wondering is that you mention above the common "stating", "production" etc but then also about 10+ environments. Can you elaborate a bit more on what these will be?

7

As you know it is common to have some pre-live environments and then production environments. but some sectors demand dedicated hosting of our platform to separate their data from others. then 'you end up with 10+ production environments'.
I believe the elastic stack solution elastic.co provides is a good example. each customer will get their own dedicated deployment. labelling log events from different workloads are very important to query them independently.

8

Basically you are collecting logs for multiple "customers" and each should have its own namespace. That make sense.

But for all "namespaces" you have a very similar / identical config and you would like to reuse it as the only thing that is different is the namespace, correct? As you have found out yourself, it is not something that we support today but I think is an interesting use case. I see multiple ways on how this could be solved. I think this would be worth to open a Github issue with the use case as a feature request. Best in the Kibana repo (GitHub - elastic/kibana: Your window into the Elastic Stack) as I think it is something we would need to solve in Fleet.

@Nima_Rezainia FYI

1 Like
9

@mohsen0 the best way to segregate the data from each environment is to create a policy specific to that environment and set the namespace (as it was mentioned above). You'll get a unique Enrollment Token that make agent membership into the policy seamless.
Would a policy "clone" action help in this case to ease the pain of re-applying the configurations?

The other unrelated thought is the tagging option during installation. an option to the Elastic Agent command allowing the user to pass in "Tags/Labels" and having these labels show up on the fleet console.

2 Likes
10

@Nima_Rezainia Setting up individual policies is too much manual work. we store the configuration in code in order to spin up environments very quickly and without any mistakes. Supporting setting you have mentioned (Tags/labels or even the org id) in enrollment of agent into fleet manager would be ideal.
Otherwise, we have to go toward the direction of adding the field in the logs of the application which is not ideal for us since it requires development work for each service (or use filebeat since we can add fields there).

@ruflin :

But for all "namespaces" you have a very similar / identical config and you would like to reuse it as the only thing that is different is the namespace, correct?

Yes that is correct

11
12

thank you for raising this issue.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.