Using Elastic Agent integration with different pipelines

Hello,

We are looking into switching over from Beats to Elastic Agent and came across an issue with not being able to define different ingest pipelines when using Custom Logs integrations e.g.:

  • we have two different sources of data going into two separate data streams:

    logs-source1-dev
    logs-source2-dev

Now I'd like to apply different ingest pipelines to each but the only option I get is to update: logs-log.log@custom pipeline which is then shared by all Custom Logs integrations.

With Beats we could specify pipeline name for each data source but with Elastic Agent the approach seems different. What is the logic behind this approach and how do we apply individual pipelines to each integration:

logs-source1-dev -> pipeline1 -> elasticsearch
logs-source2-dev -> pipeline2 -> elasticsearch

Thank you

If you’re using macOS, some versions may require you to grant full disk access to different kernels, system extensions, or files. See Enable Full Disk Access for more information.

Add the Endpoint Security integrationedit
In Kibana, select Security → Endpoints. If this is not your first time using Elastic Security, select Management → Integrations, then search for and select Endpoint Security.

Search result for "Endpoint Security" on the Integrations page.
Select Add Endpoint Security on either the Endpoints page of the Elastic Security app or the Endpoint Security integration page (Management → Integrations). The integration configuration page appears.

Add Endpoint Security integration page.
Configure the Endpoint Security integration with a name and optional description.
Select an agent policy for the Elastic Agent. You can select the default policy, an existing custom policy, or select Create agent policy to create a new one. For more details on Elastic Agent configuration settings, see Elastic Agent policies.
When the configuration is complete, click Save and continue.
To complete the integration, continue to the next section to install the Elastic Agent on your hosts.
Configure and enroll the Elastic Agentedit
To configure the Elastic Agent, Endpoint Security requires enrollment through Fleet to enable the integration.

Before you add an Elastic Agent, a Fleet Server must be running. See Add a Fleet Server.

Important information about the new Fleet Serveredit
If you are running an Elastic Stack version earlier than 7.13.0, you can skip this section.

If you have upgraded to an Elastic Stack version that includes the new Fleet Server (>=7.13.0), you will need to redeploy your agents. Review the following scenarios to ensure you take the appropriate steps.

If you redeploy the Elastic Agent to the same machine through the Fleet application after you upgrade, a new agent will appear.
If you want to remove the Elastic Agent entirely without transitioning to the Fleet Server, then you will need to manually uninstall the agent on the machine. This will also uninstall the endpoint. See Uninstall Elastic Agent.
In the rare event that the Elastic Agent fails to uninstall, you might need to manually uninstall the endpoint. See Uninstall an endpoint at the end of this topic.
Endpoint Security cannot be integrated with an Elastic Agent in Standalone mode.

Go to Fleet → Agents → Add agent.

Add agent flyout on the Fleet page.
In the Add agent pane, download the Elastic Agent on your host’s machine.
After the download is complete, select the Endpoint Security integration.

Detail of Add agent flyout with Endpoint Security integration highlighted.
After the Elastic Agent is installed on your host machine, open a command-line interface and navigate to the directory where the agent is located. Copy the commands from Fleet for your OS to enroll and run the agent.
After you have enrolled the Elastic Agent on your host, select Close. The host now appears on the Endpoints page in the Elastic Security app.
Enable Elastic Endpoint kerneledit
If you are running the Elastic Agent with the Elastic Endpoint integrated on macOS 10.13, 10.14 and 10.15, you will be prompted to approve a kernel extension from "Endgame, Inc". To approve the extension:

Select Open Security Preferences. The Security and Privacy pane opens.

macOS notification with Open Security Preferences button highlighted.
Select the Lock icon at the bottom left of the window to make changes to your security settings.

Detail of Security and Privacy pane with Lock icon highlighted.
Allow "Endgame, Inc" by clicking the Allow button.

Security and Privacy pane with Allow button highlighted.
Endgame Sensor users can approve the kernel the same way for the Elastic Endgame app.

If the prompt does not appear because you’re using a version earlier than macOS Big Sur (11.0), enable the extension by doing the following:

Open a Terminal application.
Enter kextload /Library/Extension/kendpoint.kext. Prepend the command with sudo if necessary.
To confirm the kernel extension has loaded, enter kextstat | grep co.elastic.kendpoint.
You should receive an output similar to 149 0 0xffffff7f82e7b000 0x21000 0x21000 co.elastic.kendpoint (7.11.0) BD152A57-ABD3-370A-BBE8-D15A0FCBD19A <6 5 2 1>. If you receive this output, the kernel extension is enabled.
Configure an integration policy (optional)edit
After the Elastic Agent is installed with the Endpoint Security integration, several protections features — including preventions against malware, ransomware, memory threats, and malicious behavior — are automatically enabled on protected hosts (a Platinum or Enterprise license feature). If needed, you can update the integration policy to configure protection settings, event collection, antivirus settings, and trusted applications to meet your organization’s security needs.

See Configure the Endpoint Security integration policy for more information on configuring these settings in an Endpoint Security integration policy.

Regards,
Rachel Gomez

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.