# Dec 15th, 2024: \[EN\] Anatomy of a Fleet-managed Elastic Agent integration

**URL:** <https://discuss.elastic.co/t/dec-15th-2024-en-anatomy-of-a-fleet-managed-elastic-agent-integration/368423>\
**Category:** Advent Calendar\
**Created:** [December 15, 2024, 8:00am UTC](https://discuss.elastic.co/t/dec-15th-2024-en-anatomy-of-a-fleet-managed-elastic-agent-integration/368423 "2024-12-15T08:00:01Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Luca\_Belluccini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luca_belluccini/32/33239_2.png) [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)\
**Post date:** [December 15, 2024, 8:00am UTC](https://discuss.elastic.co/t/dec-15th-2024-en-anatomy-of-a-fleet-managed-elastic-agent-integration/368423/1 "2024-12-15T08:00:01Z")

</div>

[Elastic integrations](https://docs.elastic.co/en/integrations) provide an easy way to get logs, metrics or other kind of data from different sources into Elasticsearch.

An Integration can be seen as a pre-packaged configuration template which allows a user to collect logs or metrics and ingest them into Elasticsearch as structured data following the ECS conventions.

Elastic Beats & Beats modules are able to do the same thing, but with Elastic Agent integrations, the user doesn't have to write the configuration files manually.

The integration can produce one or more data streams.

Each data stream has a `type` (e.g. `logs` or `metrics`…) and a `dataset` (e.g. `system.fs` , `system.cpu` ).

In most integrations, an out-of-the-box ingest pipeline allows to parse the data prior being indexed into Elasticsearch.

Each data stream is provided with an Index Template and Component Templates in order to define the mappings of the fields we expect to store.

All those pre-packaged configurations (`ingest pipeline`, `index templates`, `component templates`, `dashboards`, etc.) are called Integration Assets.

## Objectives of the post

You'll get a glimpse under the hood of Fleet & Elastic Agent integrations, useful when troubleshooting.

It points out key elements of an integration, useful for starting developing your own integration or contributing to them (see [our developer guide](https://www.elastic.co/guide/en/integrations-developer/current/index.html)).

⚠ _Note. The post doesn't replace our documentation and might include implementation details which might change in the future. The post is focusing on Fleet-manage Elastic Agents and using Elasticsearch as destination of the data. Some of the steps described here are simplified. The article has some pre-requisite read, in particular the [Fleet & Elastic Agent guide](https://www.elastic.co/guide/en/fleet/current/fleet-overview.html). The Nginx example mentioned in this post is an expanded version of [our tutorial](https://www.elastic.co/guide/en/observability/current/monitor-nginx.html#monitor-nginx-ml)_.

## Where can I check which are list of available integrations and their documentation?

On our [documentation page](https://www.elastic.co/guide/en/integrations/current/introduction.html) or directly from the Fleet / Integration application within Kibana.

## How can I use Elastic Agent integrations?

Elastic Agent Integrations have been created specifically for Elastic Agents.

In order to use a Fleet-managed Elastic Agent, you'll need to use a Fleet Server or Fleet Service first to "manage" Elastic Agents.  
To have a Fleet Server, you can:

- Use the Fleet Server hosted on [Elastic Cloud](https://www.elastic.co/guide/en/fleet/current/add-fleet-server-cloud.html)
- Use the Fleet Service offered in our [Elastic Serverless, available on Observability and Security Projects](https://www.elastic.co/guide/en/fleet/current/fleet-agent-serverless-restrictions.html#fleet-serverless-restrictions)
- Deploy it [on-premises and self-managed](https://www.elastic.co/guide/en/fleet/current/add-fleet-server-on-prem.html)

The [Fleet application in Kibana](https://www.elastic.co/guide/en/fleet/current/agent-policy.html#add-integration) connects to the [Elastic Package Registry](https://github.com/elastic/package-registry) to get the list of the available packages, filtering them by using the Kibana version currently running and other criteria.

Once you have a Fleet Server/Service running, you can enroll an Elastic Agent to a policy previously created on [Fleet UI](https://www.elastic.co/guide/en/fleet/current/install-fleet-managed-elastic-agent.html) (or via Fleet APIs).

An Elastic Agent policy can contain one or more integration policies.  
It also defines the general settings of the Elastic agent policy, such as the destination output (generally the `elasticsearch` cluster associated to the Fleet Server) and the namespace (`default`).

## What is an integration policy?

An integration policy is a particular way of configuring an integration package.

Let's say we want to ingest the logs of an Nginx instance running on a Linux host.

The Nginx integration knows how to parse the `access` and `error` logs of Nginx. It just needs to know where the Nginx logs are located.  
The path to the Nginx `access` logs and `error` logs are 2 properties you'll need to set in the integration policy.

 ![Screenshot 2024-12-08 at 18.25.35](https://us1.discourse-cdn.com/elastic/original/3X/4/0/4071f1b6032a11a4d0455c3d6336ad15e483a169.png)

Those parameters are materialized into the complete Elastic Agent policy, which we can represent in YAML (`Kibana / Fleet / Agent policies`, selecting the policy and then using `Actions / View Policy`).

```auto
  - id: logfile-nginx-1fe825ea-1dfa-472d-93fb-1d459057713a
    name: nginx-1
    revision: 1
    type: logfile
    use_output: default
    meta:
      package:
        name: nginx
        version: 1.24.0
    data_stream:
      namespace: default
    package_policy_id: 1fe825ea-1dfa-472d-93fb-1d459057713a
    streams:
      - id: logfile-nginx.access-1fe825ea-1dfa-472d-93fb-1d459057713a
        data_stream:
          dataset: nginx.access
          type: logs
        ignore_older: 72h
        paths:
          - /var/log/nginx/access.log*
        tags:
          - nginx-access
        exclude_files:
          - .gz$
        processors:
          - add_locale: null
      - id: logfile-nginx.error-1fe825ea-1dfa-472d-93fb-1d459057713a
        data_stream:
          dataset: nginx.error
          type: logs
        ignore_older: 72h
        paths:
          - /var/log/nginx/error.log*
        tags:
          - nginx-error
        exclude_files:
          - .gz$
        multiline:
          pattern: '^\d{4}\/\d{2}\/\d{2} '
          negate: true
          match: after
        processors:
          - add_locale: null

```

## How does the Fleet Integration policy editor gets generated?

Each integration has a `manifest.yml` ([Nginx example](https://github.com/elastic/integrations/blob/main/packages/nginx/manifest.yml)).

The Manifest, on top of declaring some metadata information (owners, license, etc...) has few key information:

- `version` of the integration
- `kibana.version` which represents the minimum version of Kibana in which a specific version of the integration will be shown. Elastic recommends to consider `kibana.version` also the minimum version of Elastic Agent to run the policy on
- `policy_templates`, with a list of `inputs`. Each `input` typically represents a type of metrics or logs which can be collected using the integration. Each `input` has a `type`. The `type` will be used by Elastic Agent to pick which will be the correct "component" to run in order to collect the metrics or logs.

Each integration declares also a set of one or more `data_streams` ([Nginx example](https://github.com/elastic/integrations/tree/main/packages/nginx/data_stream)).  
Within the `data_stream`, we have a `manifest` and a configuration template.  
Looking into the Nginx `access` example above, we have:

- [`nginx/data_stream/error/manifest.yml`](https://github.com/elastic/integrations/blob/main/packages/nginx/data_stream/error/manifest.yml), which declares all the possible "variables/parameters" open to the user to tweak, such as the log file [`paths`](https://github.com/elastic/integrations/blob/main/packages/nginx/data_stream/error/manifest.yml#L6C15-L6C20)
- [`nginx/data_stream/access/agent/stream/stream.yml.hbs`](https://github.com/elastic/integrations/blob/main/packages/nginx/data_stream/access/agent/stream/stream.yml.hbs), the actual configuration "template" ([handlebars](https://handlebarsjs.com/)) deployed on the component which will be spawned by Elastic Agent.

Fleet in Kibana will take care of _filling in_ the template with the parameters/configuration options rendered from the Fleet UI using the manifest as reference and materialize it in the Integration Policy (and consequently, in the Elastic Agent policy).

## What happens when we add an integration policy to an Elastic Agent policy?

When adding an integration policy to an Elastic Agent policy, Kibana & Fleet take care of:

- Installing all the necessary assets to the Elasticsearch cluster (Index Templates, Component templates, etc...) and to Kibana (Dashboards, Visualizations, etc...), if they were not installed already
- Creating or updating the API Key which will be used by the Elastic Agent to ship data to Elasticsearch in order to have write permissions to the destination data streams (following up on the Nginx example, `logs-nginx.error-default` and `logs-nginx.access-default` data streams).
- _(Only for some integrations)_ Installing some extra asset such as Anomaly Detection Machine Learning Jobs (Platinum or Enterprise license required) in Kibana. Those are going to leverage the logs collected to give you even more insights on your data (see [our documentation](https://www.elastic.co/guide/en/observability/current/monitor-nginx.html#monitor-nginx-ml)).
- Dispatching the updated policy to the Elastic Agent(s) enrolled. Elastic Agents connected to Fleet Server will "receive" a policy update.

Elastic Agents periodically report their status ("check-ins") using a long-polling communication with Fleet Server. Technically Fleet Server "unlocks" the long polling, returning back the action to the Elastic Agent(s) which will receive the policy update action.

When the Elastic Agent receives the policy update:

- Resolves some variables using the information available locally on the host (for example, an environment variable). If in the Nginx `paths` above we would have set `/var/log/${env.NGINX_INSTANCE_NAME}/access.log*`, Elastic Agent would have replaced the `${env.NGINX_INSTANCE_NAME}` using environment variable available on the host (the variable has to be declared for the service unit running Elastic Agent).
  - In presence of specific integrations, such as Kubernetes integration, the configuration might be "exploded" based on the variables provided by the Kubernetes provider.

- Detects the necessary "components" to be run in order to _implement_ the policy. Elastic Agent is able to determine the component to be used looking up a "capabilities" reference file. The lookup will use the `input` type present in the integration policy. Following up on the Nginx example above, Elastic Agent will spawn a Filebeat instance with 2 `logs` inputs, using the configuration in the Nginx integration policy. It will also configure Filebeat to send the logs to Elasticsearch.
- If Elastic Agent is able to startup all the necessary components and if they report back a good status, the Elastic Agent will report an `healthy` status to Fleet Server and acknowledge the policy update is successful.
  - If the input or outputs of the component have a major issue (unable to start, errors connecting, etc...), then Elastic Agent will report an `unhealthy` status back to Fleet Server with the error message.

## How the data harvested by the integration(s) gets into Elasticsearch?

Each component spawned under Elastic Agent will send the data to the output defined in the policy. In general, each integration will send data to the data stream following the convention `type`, `dataset`, `namespace` defined in the policy.

For our Nginx example, data will end up in `logs-nginx.error-default` and `logs-nginx.access-default`.  
The index templates bundled with the Nginx integration and already installed by Fleet, will take care of:

- parsing the logs using an ingest pipeline developed by the maintainers of the integration (example for Nginx [`access` logs](https://github.com/elastic/integrations/blob/main/packages/nginx/data_stream/access/elasticsearch/ingest_pipeline/default.yml))
- mapping the fields in each document to a proper data type thanks to the `component` template of the package (example for Nginx [`access` logs](https://github.com/elastic/integrations/tree/main/packages/nginx/data_stream/access/fields)), on top of the general `ecs@mappings` component template, deployed by the Elastic Stack (versions 8.13+)