I'm a developer from TiDB Community.
TiDB is a popular open source HTAP database. TiDB users use tools such as TiUP and TiDB-operator to deploy TiDB clusters in a bare-metal or k8s environment.
Many TiDB users use ELK stack to collect TiDB cluster logs. Due to the complexity of TiDB cluster components and log configuration, it is also complicated for users to configure ELK.
To reduce the complexity of using ELK, I have developed a TiDB module.
This module has 5 filesets. ["tidb", "pd", "tikv", "tiflash", "slowlog"].
Four of them map to different components in a TiDB cluster:
And the last one maps to slow logs generated by the TiDB component.
As to log parsing, it follows two specifications:
- Normal logs: https://github.com/tikv/rfcs/blob/master/text/0018-unified-log-format.md
- Slow logs: https://docs.pingcap.com/tidb/stable/identify-slow-queries
After parsing, users should be able to search log text and analyze slow logs in elasticsearch.
Pinging @filebeat-module-team. Could you help me take a look at this PR?
Thanks a lot!