Migrating from beats to the agent - best approach?

Elastic Stack Elastic Agent
1

Greetings! I am somewhat new to Elasticsearch and have been tasked with migrating from beats to the Elastic agent. We just finished migrating to Elastic cloud so this is the next thing we want to tackle to modernize.
For context: We have metricbeat, winlogbeat, and filebeat deployed to ~400 hosts (mostly windows, some linux). The beats send their data to 6 load balanced Logstash servers in our data center and then is shipped to our elastic cloud instance. There’s quite a bit of complex parsing and enrichment going on in logstash for filebeat and winlog beat.

Does it make more sense to send the data directly to Elastic Cloud rather than logstash as a “middleware”? While it might take more effort to recreate the logstash pipelines and parsing logic in Kibana, i feel like from an administration perspective it would make things easier rather than having to maintain additional logstash servers.

Any other general advice for migrating to beats from the agent is appreciated :slight_smile:

2

Hello and welcome,

There’s quite a bit of complex parsing and enrichment going on in logstash for filebeat and winlogbeat.

What kind of data are you collecting with Filebeat, is it custom logs or something that has an equivalent Elastic Agent Integration?

The data collected by Elastic Agent will be parsed using Elasticsearch ingest pipelines, so it is expected that the original message to not be changed.

For example, for Winlogbeat you would need to use two integrations in your Agent policy, the System integration and the Windows integration, then the parse would be done by different ingest pipelines.

You can use custom pipelines to make extra changes on your data, but all custom pipelines will run after the integration pipeline, so you do not have the same flexiblity as you have with Logstash.

Also, depending on what kind of enrichment you are doing in Logstash, you may not be able to do that using an Ingest Pipeline.

I would test this on a couple of servers to see if you can achieve the same final data before moving entirely to Elastic Agent.