How to create a filebeat processor?

Elastic Stack Beats
1

We want to use filebeat to gather mysql slow log. I want to create a filebeat processor to convert mysql slow log to json format and add a sql fingerprint field.
I find a related issue here: https://github.com/elastic/beats/issues/6760

I follow up the issue but encounter an error:

[root@c3-b2c-dba-dbshard17 filebeat-7.4.0]# ./filebeat -e -c filebeat.yml --plugin sql_fingerprint.so
Exiting: plugin.Open("sql_fingerprint"): plugin was built with a different version of package github.com/elastic/beats/libbeat/common/file

We use filebeat 7.4.0 version online, I find that this version filebeat is compiled by go 1.12.9.

[root@c3-b2c-dba-dbshard17 filebeat-7.4.0]# ./filebeat version
filebeat version 7.4.0 (amd64), libbeat 7.4.0 [f940c36884d3749901a9c99bea5463a6030cdd9c built 2019-09-27 07:45:44 +0000 UTC]

git checkout f940c36884d3749901a9c99bea5463a6030cdd9c

[root@sgp2-b2c-b2cop-beta-neo-node01 beats]# cat .go-version
1.12.9

The steps I use to create processor:

  1. Install go 1.12.9
  2. git clone filebeat repo
  3. git checout f940c36884d3749901a9c99bea5463a6030cdd9c
  4. write code in {go/src}github.com/elastic/beats/filebeat/processor/sql_fingerprint
  5. cd {go/src}github.com/elastic/beats/filebeat/processor/sql_fingerprint and build plugin with:
    GO111MODULE=on go build -mod=vendor -buildmode=plugin
  6. use the sql_fingerprint.so generated by step 4 and run:
[root@c3-b2c-dba-dbshard17 filebeat-7.4.0]# ./filebeat -e -c filebeat.yml --plugin sql_fingerprint.so
Exiting: plugin.Open("sql_fingerprint"): plugin was built with a different version of package github.com/elastic/beats/libbeat/common/file

Help.

1 Like
2

Actually you don't need to touch Go code, you can always use script processor: https://www.elastic.co/guide/en/beats/filebeat/master/processor-script.html . Please take a look and tell if it meets your needs.

3

I tried. There is another problem, I want to calculate sql fingerprint with another program. Can script processor call local program, such as shell?

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.