A New TiDB Module for Filebeat

Hello everyone,

I'm a developer from TiDB Community.

TiDB is a popular open source HTAP database. TiDB users use tools such as TiUP and TiDB-operator to deploy TiDB clusters in a bare-metal or k8s environment.

Many TiDB users use ELK stack to collect TiDB cluster logs. Due to the complexity of TiDB cluster components and log configuration, it is also complicated for users to configure ELK.

To reduce the complexity of using ELK, I have developed a TiDB module.

[Filebeat] A New TiDB Module for Filebeat by SabaPing · Pull Request #27663 · elastic/beats (github.com)

This module has 5 filesets. ["tidb", "pd", "tikv", "tiflash", "slowlog"].
Four of them map to different components in a TiDB cluster:

  • TiDB
  • PD
  • TiKV
  • TiFlash

And the last one maps to slow logs generated by the TiDB component.

As to log parsing, it follows two specifications:

After parsing, users should be able to search log text and analyze slow logs in elasticsearch.

Pinging @filebeat-module-team. Could you help me take a look at this PR?

Thanks a lot! :partying_face:

1 Like

Can someone please help me review my PR?
Out customers, who use Elasticsearch to search TiDB runtime logs, really want this feature :heart_eyes:.