Filebeat errors on setup - Apache ingest pipelines failing to load

QueenAmidala · February 23, 2021, 4:00pm

So here is my problem it seems filbeats setup is failing for me. I get a bunch of empty error messages

**"Exiting: 1 error: Error setting up ML for apache_ecs: 17 errors: ; ; ; ; ; ; ; ; ; ; error while".**

I'm thinking that this is because the error structure used by filebeats is wrong.
The structure is defined here: beats/importer.go at v7.11.1 · elastic/beats · GitHub.

And, the errors get appeneded here:beats/importer.go at v7.11.1 · elastic/beats · GitHub.

The errors in the response looks more like: kibana/types.ts at v7.11.1 · elastic/kibana · GitHub.

There was a merge request a while ago that looks like it changed the error responses https://github.com/elastic/kibana/pull/74965.

Does anyone know if filebeats is representing errors incorrectly or is this just another kind of error it doesn't know about?

The errors are just datafeed and job conflicts which it looks like the code was trying to watch for.

stephenb · February 22, 2021, 11:22pm

Hi @QueenAmidala Welcome to the community.

Long story short, don't worry about that right now... it has to do with some legacy ml jobs / mapping and should not be an issue.

Curious what version you are running though?

Go ahead and run filebeat and use the Visualizations, Dashboards, Alerts actions as you see fit let us know if you have any issues.

Thanks for checking in with this.

QueenAmidala · February 23, 2021, 4:05am

This seems to also be a problem for these people Error trying to setup filebeat because the error response structure for a visualization is actually more like kibana/types.ts at v7.11.1 · elastic/kibana · GitHub when you run filebeats setup for the first time and the error message doesn't match with what is looked for in the code.

Is there no desire to fix this because it is legacy code? Can the machine learning option for filebeat setup at least be disabled by default?

I'm using version 7.11.1 of Elastic. So, like Elasticsearch 7.11.1, Kibana 7.11.1, and Filebeat 7.11.1. I'm using the container images.

stephenb · February 23, 2021, 4:35am

Hi @QueenAmidala

Thank you for opening the issue, I will poke engineering a bit tomorrow if I get a chance. When I asked before (as I ran into myself) I was told this will be fixed in / mitigated in a future release and at this point it is does not indicate any actual issues.

Is there any functionality other than the error message that is broken for you? If so let us know, otherwise I would proceed with the stack.

If you are using 7.11.1 and enable apache and / or ngnix and then run setup the correct ML jobs will be loaded.

I just built a brand new stack ...

$ ./filebeat modules enable apache nginx 
$ Enabled apache
$ Enabled nginx

$ ./filebeat setup
......
Exiting: 3 errors: Error setting up ML for apache_ecs: 10 errors: ; ; ; ; ; ; ; ; ; ; Error setting up ML for apache_ecs: 10 errors: ; ; ; ; ; ; ; ; ; ; Error setting up ML for nginx_ecs: 10 errors: ; ; ; ; ; ; ; ; ;

I get the same error however when I go into Kibana the correct ML jobs are loaded for apache and nginx, my understanding is that is what would be broken if it was an actual issue.

QueenAmidala · February 23, 2021, 2:01pm

It is a conflict error about a resource already existing since Nginx and Apache use the same names for things. But it has other repercussions besides being an error message, for example the filebeat setup command fails with a failing exit code, and also the pipeline loading gets skipped because it happens after the error

github.com

elastic/beats/blob/v7.11.1/libbeat/cmd/instance/beat.go#L578


		if setup.MachineLearning && b.SetupMLCallback != nil {
			cfgwarn.Deprecate("8.0.0", "Setting up ML using %v is going to be removed. Please use the ML app to setup jobs.", strings.Title(b.Info.Beat))
			fmt.Println("Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead.\nSee more: https://www.elastic.co/guide/en/machine-learning/current/index.html")
			err = b.SetupMLCallback(&b.Beat, b.Config.Kibana)
			if err != nil {
				return err
			}
			fmt.Println("Loaded machine learning job configurations")
		}
		if setup.Pipeline && b.OverwritePipelinesCallback != nil {
			esConfig := b.Config.Output.Config()
			err = b.OverwritePipelinesCallback(esConfig)
			if err != nil {
				return err
			}
			fmt.Println("Loaded Ingest pipelines")
		}
		return nil

It's also not very intuitive to see empty error messages instead of the actual errors.

stephenb · February 23, 2021, 3:32pm

Hi @QueenAmidala

Yeah agree needs to be fixed, interesting I was about to tell you that the pipelines are being added because the nginx pipeline is loaded but the apache pipelines are not loaded! That is not good.

Could you add to your issues that the apache ingest pipelines are failing to load. (Bad error messages are one thing but failing to load the pipeline is not good at all) I will pass on the issue to engineering.

Actually if you editing the Title to something like

Filebeat errors on setup - Apache ingest pipelines failing to load

That would probably get more attention.

Thanks again.

stephenb · February 23, 2021, 4:07pm

As a workaround in the meantime you can run the following to manually load the pipelines.

./filebeat setup --modules=apache --pipelines

stephenb · February 23, 2021, 10:28pm

Hi @QueenAmidala

Thanks again as you can see Engineering has picked up your Github issue.

Above I was hoping that you would update the Github Issue Title... to

Filebeat errors on setup - Apache ingest pipelines failing to load

Its a little more descriptive