Using Azure integration without Azure storage

Elastic Stack Elastic Agent
1

As per Azure Logs | Documentation , setting up Azure integration requires Azure storage for state registry to share it between agents.
Is it possible to skip Azure storage and use local state registry (single agent or shared storage)?

Rationale: state registry is a low-volume, high rate usecase. This both makes agent increasingly reliant on low latency, and increases storage expenses.

2

Hello and welcome,

No, it is not possible, the storage account is a required configuration, you cannot add the integration without configuring it.

But one alternative would be to try to use Logstash to get the logs using the Kafka interface of the Event Hubs, split the records and then send it back to Elasticsearch configuring to use the ingest pipeline of each dataset.

One big issue here is that azure logs are all in a array named records, the eventhub integration get those logs and split this array into one event per item, you cannot do that with any other Elastic Agent input or Ingest pipeline processor, so this requires Logstash.

3

Thanks, that sounds very interesting, but potentially fairly complex in the end.
I guess Azure storage usage isn't a technical limitation - as this is already passing through Elastic Agent, theoretically it could use local storage, right?