Elasticsearch output for Elastic Agent - adding an ingest pipeline

[bsanderRMG (Ben Sander)]

Hello. Is it possible to assign an ingest pipeline for the Elastic Agent? ideally we'd like to push this out to the entire fleet as well. I've tried below and it does not seem to have any effect.

Let me know if you have any questions. Thanks!

[bsanderRMG (Ben Sander)]

I've found the solution and feel it may help others who have this problem in the future.

Didn't realize that the Elastic Agent automatically builds data streams that are processed by preset ingest pipelines. I edited the existing pipeline to capture the organization data that we wanted, and know now that all data received from the agent goes through these pipelines.

[ruflin]

Hi @bsanderRMG Great to hear you found a solution. One thing you should be careful about is to not modify existing data streams related to an integration. So for example data streams created for nginx or system. The reason is, that as soon as a package gets upgraded these are overwritten so if you modified these, yours get overwritten

If you just used the logfile package and set your own dataset, then you should be fine.

[Grim (Patrick)]

I had a similar challenge when adding the "Custom Logs" Integration (for simply tailing log files) to my fleet. This will not (at least in my case) create a dedicated Ingest Pipeline for those logs, even though it also creates a data stream.
I managed to solve this by manually creating the pipeline and then adding the pipeline name in the "Advanced Options" part of the Integration settings:

[ruflin]

@Grim Welcome to the board and sharing this trick. I must confess I missed that this is one more option that it can be done. With the new indexing strategy (I just did a talk on it here yesterday: https://www.youtube.com/watch?v=ls1O-gB-Voo) we try to define the processing on the data stream itself instead of on the request. Of course, both is working.

So in your case, even if you specify the processing as part of the custom config, I would encourage you to modify the dataset name to your needs instead of using generic as the default.