Export template from custom beat based on metricbeat

Elastic Stack Beats
1

Hi

I created my own beat based on metricbeat 6.0 with it's own modules and metricsets. I also created the necessary fields.yml files for the modules and the metricsets and ran successfully make update which gave me a kibana directory under the beat _meta directory, so far so good.

Now I want to generate an index template to import into Kibana but when I run beat export template > template.json the process gets stuck forever resulting in an empty template.json file. Is this the correct way to get the index template file? Any suggestions how to get this index template file from a custom beat?

Kind Regards
Davy

2

The export template command exports the template mapping for use with Elasticsearch. I guess this is what you need.

The index pattern for Kibana is generated on setup and installed when installing the dashboards via setup.

Can you share your fields.yml or even better link to repo for us to investigate? beat export template must not hang. I'm curious if this is an error in fields.yml or a bug in the exporter.

3

Hi @steffens

The export template command exports the template mapping for use with Elasticsearch. I guess this is what you need.

Correct. As I'm pretty new to this, I mess up some terms from time to time :worried:.

I uploaded my code and (config) files to this repo on Github. I hope it's enough for you to reproduce the issue.

Thanks in advance.

Regards

4

The problem is in your code. You're using cmd/instance directly. This creates and starts the beat, but removes the sub-command functionality.

Metricbeat has it's own 'root command': https://github.com/elastic/beats/blob/master/metricbeat/cmd/root.go#L22

The libbeat/cmd package is used by metricbeat, to create a standardized beat with sub-commands + hooks the export template sub-command into the root command.

Makes me wonder, how come you're using the cmd/instance package instead of having a root command?

5

@steffens Thanks for your insights and effort.

To be honest I have no idea. I had some issues when creating a new beat based on metricbeat. After searching a bit, I found this thread, which has a solution that also worked for me. Maybe this is the root cause of the problem?

Can I fix this manually or would it be better to create a new beat based on metricbeat?

6

@steffens May I ask in which file you saw that I'm using the cmd/instance directly? I suppose it was main.go but when I generate a new beat based on metricbeat with the command python ${GOPATH}/src/github.com/elastic/beats/script/generate.py --type=metricbeat I'll get a main.go file more or less the same as the one I got?

Based on the 6.0 branch of metricbeat

package main

import (
	"os"

	"github.com/elastic/beats/libbeat/cmd/instance"
	"github.com/elastic/beats/metricbeat/beater"

	// Comment out the following line to exclude all official metricbeat modules and metricsets
	_ "github.com/elastic/beats/metricbeat/include"

	// Make sure all your modules and metricsets are linked in this file
	_ "github.com/dcroonen/examplebeat/include"
)

var Name = "examplebeat"

func main() {
	if err := instance.Run(Name, "", beater.New); err != nil {
		os.Exit(1)
	}
}

and based on the master branch of metricbeat:

package main

import (
	"os"

	"github.com/elastic/beats/libbeat/cmd/instance"
	"github.com/elastic/beats/metricbeat/beater"

	// Comment out the following line to exclude all official metricbeat modules and metricsets
	_ "github.com/elastic/beats/metricbeat/include"

	// Make sure all your modules and metricsets are linked in this file
	_ "github.com/dcroonen/masterbeat/include"
)

var Name = "masterbeat"
var IndexPrefix = "masterbeat"

func main() {
	if err := instance.Run(Name, IndexPrefix, "", beater.DefaultCreator()); err != nil {
		os.Exit(1)
	}
}

So at the moment I don't have clue where to look to hook in the sub commands again....

7

Checking the code generators template, it indeed uses instance.Run. This is somewhat ok, but doesn't give you the sub-command functionality as used/provided by the other beats. Feel free to open a github issue on missing sub-commands when using the generator.

Checking metricbeat code-base, I see why sub-commands support is missing. The code-base of local cmd packages is not much re-usable for use by custom beats right now. I just created this ticket.

If you want sub-command support you need to copy the main.go and the cmd package from metricbeat. Then update cmd/root.go file to use your beater instance.

8

OK, that explains a lot. I'll open a ticket on github referencing this threat.

Thanks for your support.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.