How to add a new filebeat.input tag in filebeat go code repo

Elastic Stack Beats
1

These are the steps I followed -

  1. I cloned and copied the log folder (https://github.com/elastic/beats/tree/master/filebeat/input), renamed it to newlog and placed it into that folder only.
  2. I changed the package name to newlog for all files inside it
  3. I tried to register that input in init() function inside input.go and also registered the input in include.go file.

I faced with duplicate harvester error - (panic: name harvester already used) so I tried to change the harvester registry name to filebeat.harvestor.newlog and tried to build it.

It builds successfully with filebeat.input to newlog in filebeat.yml config but as an when I try to run it, it throws error
Harvester could not be started on new file: /root/fakeRBOS3253.log, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: Invalid harvester type

Can anybody help regarding this on how to add your custom tag input to beats?
Here, the problem is just not about having a custom tag name but I want to implement some custom logic functionality behind it.
Please help!

2

Hi,

I'm not quite sure what you're trying to achieve and if it's really required to build an input for that. Could you describe your use case?

3

Hi mtojek,

The use case is quite simple. We want a custom input tag to be included in the filebeat.yml file to monitor the specific files. Now those specific files are not plain text files but custom encrypted binary files so essentially, what we want is, to monitor such files and write custom logic to decrypt and push those logs to output.

So to tell you in brief as an example-
filename.bin (our binary encoded file) ---> filebeat -----> custom tag(decodelog) ----> decodelog will convert the binary log to plain text log ---> Send output just like log output.

The decode log functionality is a custom wrapper which basically tries to convert the whole binary file to plain text format which each line terminated by line seperator \n.

Basically, we want to extend out and patch our decode functionality into filebeat.

4

Could you share your filebeat.yml file (use code tags)? I think there might be an issue around the type: log.

See:

// open does open the file given under h.Path and assigns the file handler to h.log
func (h *Harvester) open() error {
	switch h.config.Type {
	case harvester.StdinType:
		return h.openStdin()
	case harvester.LogType, harvester.DockerType, harvester.ContainerType:
		return h.openFile()
	default:
		return fmt.Errorf("Invalid harvester type: %+v", h.config)
	}
}

You might need to add another entry here.

5

Oh yes. It worked.
I added a case statement for the custom config type.
Thanks

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.