Extend Metricbeat

Elastic Stack Beats
1

I understand there is a way to extend metric beats

but documentation is bit confusing it says there are two ways to extend

  • Extend Metricbeat directly
  • Create your own Beat and use Metricbeat as a library

what i understand is extending metricbeat directly is creating module and metricset under existing metric repo and start extending this new metric set

Create your own Beat is create a new beat with --type=metricbeat and start extending this new beat

but

when i fallow 1st method to direcltly extend metricbeat after creating
a metric set inside elastic/beat/metricbeat folder and enabled my module and tried to run i got and error
Exiting: 1 error: 1 error: metricset 'micro/mic' is not registered, module not found

i don't really understand how to build using Create your own Beat and use Metricbeat as a library

generate a new beat by passing --type as metricbeat and it asks for metricset and module name

after this i compiled using make and executed using ./beatname -c myyml.yml i get error as
Exiting: one or more modules must be configured

i don't understand whats going one please help

2

Let's focus on the second path (create your own beat and use metricbeat as a library) for now, since that is the recommended starting path. Can you share the contents of your myyml.yml here please?

3

ok i got it

issue was the name in .yml i created my beat named xyz and in .yml which i was using still had config pointing to metricbeat.config.modules: i dint find this anywhere in documentation that we need to change the name in .yml i just gave a try and it worked :slight_smile:

but
i have one more issue with packaging if i do make package under my new beat i get make: Nothing to be done for 'package'.

Any idea why this happens ??

4

ok further to this i downgraded beat to 6.2 and created new normal beat (without --type=metricbeat)
under new beat folder make setup was success and make package started buildinf triggering docker

and i created new beat with --type=metricbeat and did make setup go below error

cp: cannot stat 'module/*/_meta/kibana': No such file or directory vendor/github.com/elastic/beats/metricbeat/Makefile:23: recipe for target 'kibana' failed make[1]: [kibana] Error 1 (ignored) Traceback (most recent call last): File "./vendor/github.com/elastic/beats/script/generate_imports.py", line 7, in <module> from generate_imports_helper import comment, get_importable_lines ImportError: No module named generate_imports_helper vendor/github.com/elastic/beats/metricbeat/Makefile:57: recipe for target 'imports' failed make[1]: *** [imports] Error 1 make[1]: Leaving directory '/home/misra_debasish90/go/src/github.com/Manjukb/metrictest' Makefile:17: recipe for target 'setup' failed make: *** [setup] Error 2

please help

5

Hi @Manjukb

When using the 6.2 branch before executing make setup in step 3 do:

  • cp -r $GOPATH/src/github.com/elastic/beats/metricbeat/scripts .
  • vi ${GOPATH}/src/github.com/elastic/beats/metricbeat/Makefile
    • line 25 replace
      @-cp -r module/*/_meta/kibana _meta/
      by
      @-cp -r ../../elastic/beats/metricbeat/module/*/_meta/kibana _meta/

Now you can run make setup.

6

@dcroonen thanks make setup worked fine

after i want to package so did make package got bellow error :frowning:

Copying main source folder /source to folder /go/src/github.com/***
Fetching dependencies...
Execute /scripts/before_build.sh github.com/**/metricbeat ./vendor/github.com/elastic/beats
# disable the system/load metricset on windows
sed -i.bk 's/- load/#- load/' /build/modules.d-win/system.yml
sed: can't read /build/modules.d-win/system.yml: No such file or directory
make: *** [before-build] Error 2
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:424: recipe for target 'prepare-package-cgo' ``failed make[1]: *** [prepare-package-cgo] Error 2 make[1]: Leaving directory '/home/misra_debasish90/work/src/github.com/**/metricbeat' vendor/github.com/elastic/beats/libbeat/scripts/Makefile:465: recipe for target 'package' failed make: *** [package] Error 2

7

The file /build/modules.d-win/system.yml isn't found. My guess is that some variable isn't set correctly. I suspect there will a file system.yml in <path to your module>/build/modules.d-win but the the sed program is looking in /build/modules.d-win and that's your root directory.

8

/build/modules.d-win/ don't have system.yml file it has only file called mymodule.yml.disabled

pasting go env

GOARCH="amd64"
GOBIN="/home/misra_debasish90/work/bin"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/misra_debasish90/work"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build723023866=/tmp/go-build -gno-record-gcc-switches"``CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

9

can anyone please look into it ??

10

Is there a reason you are working off the 6.2 branch and not master or even the 6.x branch? The reason I ask is because the sed ... command for modules.d-win/system.yml seems to have been removed in 6.3 as part of a larger refactoring effort: https://github.com/elastic/beats/pull/7388.

Unless you have a very strong reason to develop off the 6.2 branch, I'd recommend working off master or 6.x.

11

i have no reason to use 6.2

in master branch make package won't even start it says make: *** No rule to make target 'package'. Stop.

so i tried downgrading and found that packaging was starting and it even packaged without errors for normal beat(not --type=metricbeat)

12

any idea why this happens ??

pasting more details
python --version = Python 2.7.13
go version = go version go1.10 linux/amd64
docker --version= Docker version 18.06.1-ce, build e68fc7a

please help

13

Hi @Manjukb,

I just went through all the steps in https://www.elastic.co/guide/en/beats/devguide/current/creating-beat-from-metricbeat.html (working off master). There are two issues in that tutorial which need to be fixed:

Thank you for pointing us to these issues!

14

Thanks this works :slight_smile:

i'm trying to implement CI using travis CI but i fails

do you have any example YML which works for master both for type=metricbeat and normal beat

this will be helpful for me

Thanks

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.