Dec 15th, 2024: [EN] Anatomy of a Fleet-managed Elastic Agent integration

Elastic Community and Ecosystem Advent Calendar
1

Elastic integrations provide an easy way to get logs, metrics or other kind of data from different sources into Elasticsearch.

An Integration can be seen as a pre-packaged configuration template which allows a user to collect logs or metrics and ingest them into Elasticsearch as structured data following the ECS conventions.

Elastic Beats & Beats modules are able to do the same thing, but with Elastic Agent integrations, the user doesn't have to write the configuration files manually.

The integration can produce one or more data streams.

Each data stream has a type (e.g. logs or metrics…) and a dataset (e.g. system.fs , system.cpu ).

In most integrations, an out-of-the-box ingest pipeline allows to parse the data prior being indexed into Elasticsearch.

Each data stream is provided with an Index Template and Component Templates in order to define the mappings of the fields we expect to store.

All those pre-packaged configurations (ingest pipeline, index templates, component templates, dashboards, etc.) are called Integration Assets.

Objectives of the post

You'll get a glimpse under the hood of Fleet & Elastic Agent integrations, useful when troubleshooting.

It points out key elements of an integration, useful for starting developing your own integration or contributing to them (see our developer guide).

:warning: Note. The post doesn't replace our documentation and might include implementation details which might change in the future. The post is focusing on Fleet-manage Elastic Agents and using Elasticsearch as destination of the data. Some of the steps described here are simplified. The article has some pre-requisite read, in particular the Fleet & Elastic Agent guide. The Nginx example mentioned in this post is an expanded version of our tutorial.

Where can I check which are list of available integrations and their documentation?

On our documentation page or directly from the Fleet / Integration application within Kibana.

How can I use Elastic Agent integrations?

Elastic Agent Integrations have been created specifically for Elastic Agents.

In order to use a Fleet-managed Elastic Agent, you'll need to use a Fleet Server or Fleet Service first to "manage" Elastic Agents.
To have a Fleet Server, you can:

The Fleet application in Kibana connects to the Elastic Package Registry to get the list of the available packages, filtering them by using the Kibana version currently running and other criteria.

Once you have a Fleet Server/Service running, you can enroll an Elastic Agent to a policy previously created on Fleet UI (or via Fleet APIs).

An Elastic Agent policy can contain one or more integration policies.
It also defines the general settings of the Elastic agent policy, such as the destination output (generally the elasticsearch cluster associated to the Fleet Server) and the namespace (default).

What is an integration policy?

An integration policy is a particular way of configuring an integration package.

Let's say we want to ingest the logs of an Nginx instance running on a Linux host.

The Nginx integration knows how to parse the access and error logs of Nginx. It just needs to know where the Nginx logs are located.
The path to the Nginx access logs and error logs are 2 properties you'll need to set in the integration policy.

Screenshot 2024-12-08 at 18.25.35
Screenshot 2024-12-08 at 18.25.351596×1742 255 KB

Those parameters are materialized into the complete Elastic Agent policy, which we can represent in YAML (Kibana / Fleet / Agent policies, selecting the policy and then using Actions / View Policy).

  - id: logfile-nginx-1fe825ea-1dfa-472d-93fb-1d459057713a
    name: nginx-1
    revision: 1
    type: logfile
    use_output: default
    meta:
      package:
        name: nginx
        version: 1.24.0
    data_stream:
      namespace: default
    package_policy_id: 1fe825ea-1dfa-472d-93fb-1d459057713a
    streams:
      - id: logfile-nginx.access-1fe825ea-1dfa-472d-93fb-1d459057713a
        data_stream:
          dataset: nginx.access
          type: logs
        ignore_older: 72h
        paths:
          - /var/log/nginx/access.log*
        tags:
          - nginx-access
        exclude_files:
          - .gz$
        processors:
          - add_locale: null
      - id: logfile-nginx.error-1fe825ea-1dfa-472d-93fb-1d459057713a
        data_stream:
          dataset: nginx.error
          type: logs
        ignore_older: 72h
        paths:
          - /var/log/nginx/error.log*
        tags:
          - nginx-error
        exclude_files:
          - .gz$
        multiline:
          pattern: '^\d{4}\/\d{2}\/\d{2} '
          negate: true
          match: after
        processors:
          - add_locale: null

How does the Fleet Integration policy editor gets generated?

Each integration has a manifest.yml (Nginx example).

The Manifest, on top of declaring some metadata information (owners, license, etc...) has few key information:

  • version of the integration
  • kibana.version which represents the minimum version of Kibana in which a specific version of the integration will be shown. Elastic recommends to consider kibana.version also the minimum version of Elastic Agent to run the policy on
  • policy_templates, with a list of inputs. Each input typically represents a type of metrics or logs which can be collected using the integration. Each input has a type. The type will be used by Elastic Agent to pick which will be the correct "component" to run in order to collect the metrics or logs.

Each integration declares also a set of one or more data_streams (Nginx example).
Within the data_stream, we have a manifest and a configuration template.
Looking into the Nginx access example above, we have:

Fleet in Kibana will take care of filling in the template with the parameters/configuration options rendered from the Fleet UI using the manifest as reference and materialize it in the Integration Policy (and consequently, in the Elastic Agent policy).

What happens when we add an integration policy to an Elastic Agent policy?

When adding an integration policy to an Elastic Agent policy, Kibana & Fleet take care of:

  • Installing all the necessary assets to the Elasticsearch cluster (Index Templates, Component templates, etc...) and to Kibana (Dashboards, Visualizations, etc...), if they were not installed already
  • Creating or updating the API Key which will be used by the Elastic Agent to ship data to Elasticsearch in order to have write permissions to the destination data streams (following up on the Nginx example, logs-nginx.error-default and logs-nginx.access-default data streams).
  • (Only for some integrations) Installing some extra asset such as Anomaly Detection Machine Learning Jobs (Platinum or Enterprise license required) in Kibana. Those are going to leverage the logs collected to give you even more insights on your data (see our documentation).
  • Dispatching the updated policy to the Elastic Agent(s) enrolled. Elastic Agents connected to Fleet Server will "receive" a policy update.

Elastic Agents periodically report their status ("check-ins") using a long-polling communication with Fleet Server. Technically Fleet Server "unlocks" the long polling, returning back the action to the Elastic Agent(s) which will receive the policy update action.

When the Elastic Agent receives the policy update:

  • Resolves some variables using the information available locally on the host (for example, an environment variable). If in the Nginx paths above we would have set /var/log/${env.NGINX_INSTANCE_NAME}/access.log*, Elastic Agent would have replaced the ${env.NGINX_INSTANCE_NAME} using environment variable available on the host (the variable has to be declared for the service unit running Elastic Agent).
    • In presence of specific integrations, such as Kubernetes integration, the configuration might be "exploded" based on the variables provided by the Kubernetes provider.
  • Detects the necessary "components" to be run in order to implement the policy. Elastic Agent is able to determine the component to be used looking up a "capabilities" reference file. The lookup will use the input type present in the integration policy. Following up on the Nginx example above, Elastic Agent will spawn a Filebeat instance with 2 logs inputs, using the configuration in the Nginx integration policy. It will also configure Filebeat to send the logs to Elasticsearch.
  • If Elastic Agent is able to startup all the necessary components and if they report back a good status, the Elastic Agent will report an healthy status to Fleet Server and acknowledge the policy update is successful.
    • If the input or outputs of the component have a major issue (unable to start, errors connecting, etc...), then Elastic Agent will report an unhealthy status back to Fleet Server with the error message.

How the data harvested by the integration(s) gets into Elasticsearch?

Each component spawned under Elastic Agent will send the data to the output defined in the policy. In general, each integration will send data to the data stream following the convention type, dataset, namespace defined in the policy.

For our Nginx example, data will end up in logs-nginx.error-default and logs-nginx.access-default.
The index templates bundled with the Nginx integration and already installed by Fleet, will take care of:

  • parsing the logs using an ingest pipeline developed by the maintainers of the integration (example for Nginx access logs)
  • mapping the fields in each document to a proper data type thanks to the component template of the package (example for Nginx access logs), on top of the general ecs@mappings component template, deployed by the Elastic Stack (versions 8.13+)
2 Likes