Environment Variable as Custom Field - Fleet Policy

Elastic Stack Elastic Agent
1

Hi,
We are trying to fetch an environment variable to be added as a field for every log message coming in from the server where elastic agent is deployed. Similar to this issue link
The Elastic agent deployments are fleet enrolled
Setup Details

  • Cluster 1 - Version details:
    -- Elastic Agent: 8.16.6 and 8.12.2
    -- ES and Kibana : 8.18.1
    -- Fleet Server : 8.18.1
    -- Elastic Defend Integration: v8.18.1-prerelease.0

  • Cluster 2 - Version details:
    -- Elastic Agent: 8.16.6 and 8.12.2
    -- ES and Kibana : 8.17.0
    -- Fleet Server : 8.17.0
    -- Elastic Defend Integration: v8.17.1

  • Tried custom field options - no luck with no error messages and the datastream totally stops from that agent.
    -- ${env.VARIABLE_NAME}
    -- {env.VARIABLE_NAME}
    -- ${VARIABLE_NAME}

  • Tried adding the process.env.vars , which did bring some events with value VARIABLE_NAME=variable_value

  • Tried the enrichment value as well, no new field added but the datastream runs fine.

Objective:

  • Have this field (application ID) available in every log message which can then be queried by application owners from a common ECE instance with required access permissions.
  • Tagging alerts and enrichment with data against other tools which also have same VARIABLE_NAME and value
2

Alright so there are few things I had configure to achieve this (not completely clean)

  1. Update the elastic-agent service to pick a environment variable, in my case DEPLOYMENT_NAME=edragent1 Ideally have an override.conf.
  2. Since this policy has only a defend integration, update the following in the advanced setting as Custom.app_id=${env.DEPLOYMENT_NAME} within linux.advanced.document_enrichment.fields
  3. You should be able to get that field however it will not be Mapped and that is to be handled on the template side.
  4. Conclusion - Custom field option on the policy setting will not work with Elastic Defend integration unless someone comes with a WOW solution!! please help :slight_smile:
3

As mentioned in your Slack thread on the public slack, this is indeed how it works.

It seems that custom fields does not work for some type of inputs according to the documentation.

Note that adding custom tags is not supported for a small set of inputs:

  • apm
  • cloudbeat and all cloudbeat/* inputs
  • cloud-defend
  • fleet-server
  • pf-elastic-collector, pf-elastic-symbolizer, and pf-host-agent
  • endpoint inputs. Instead, use the advanced settings (*.advanced.document_enrichment.fields) of the Elastic Defend Integration.

Your approach is the correct one.

I think that this information should be more clear and present on the agent settings configuration as well.

4

Hi @karnamonkster, unfortunately no, Endpoint doesn't expect environment variables to resolve in the custom values.

PS. Even if it did that the variable would have to be set on ElasticEndpoint service, not elastic-agent

5

I think that might be the case, do we know if there is a possibility to have it?
Also we can add a warning while adding custom fields on Policy level for integration type similar to what happens when you select fleet-server integration.
We are now going ahead with the pipeline processor to map and rename the unmapped field

6

This Endpoint feature wasn't meant to handle custom processors like the fleet.

I'd suggest to file enhancement request issue about this on GitHub - elastic/endpoint

7

I think that one of the issues is that this is not clear on the configuration page for the policy settings, it should have an warning saying that custom fields do not work for the Endpoint integration in the same way that we get an warning saying that the Logstash output does not work for Fleet Server.

1 Like
8

  • Sure, i will put in the request, but if there is an option to provide enrichment data / fields, they should be mapped at least, and should have ignore_missing by default. Cause for some reason a server does not run the service with that environment variable shouldn't stop sending all datastreams.

  • We will ensure first to have all the elastic agents restarted to capture that environment variable running EDR, then we move ahead with the enrichment fields

  • We have a working solution now after setting that in the index template and a custom pipeline with an ingest processor.