Multiple logs to a single index

Elastic Stack Elasticsearch
1

Hello,

I have a use case to fetch the log messages from Kafka audit cluster and send it to Elastic. But the log messages each are of different format and if I send the messages to respective index I'm ending up with creating multiple indexes which will later become tedious to manage.

Can I send them all to a single index? What's the best way to implement it?

Regards,
Nagendra

2

Hello Nagendra,

Yes, you can send all log messages to a single index. You can use Logstash or Filebeat to consume messages from Kafka and send them to Elasticsearch. To handle different log formats, you can use Elasticsearch ingest pipelines or Logstash filters for processing and normalizing data before indexing.

Best regards