Filebeat - There are conflicts between additional prospectors and reload function

Elastic Stack Beats
1

When I use the additional prospectors function and reload function, the reload function is not working.

Filebeat read the additional prospector configurations in the config.FetchConfigs, which call the mergeConfigFiles.
In the mergeConfigFiles, the code is :

	tmpConfig := struct {
		Filebeat Config
	}{}
	err := cfgfile.Read(&tmpConfig, file)
	if err != nil {
		return fmt.Errorf("Failed to read %s: %s", file, err)
	}

	config.Prospectors = append(config.Prospectors, tmpConfig.Filebeat.Prospectors...)

The structure decides that the additional.yml should be :



filebeat.prospectors:

  • input_type: log
    paths:
    • d:\temp\test\log\test1*.log
  • input_type: log
    paths:
    • d:\temp\test\log\test2*.log
when I enable the reload function: filebeat.config.prospectors: path: config/*.yml reload.enabled: true reload.period: 10s

I find that it doesn't working. I do the debug and check the libbeat/cfgfile/reload.go, and find a doubt.
First, the watcher is OK, filebeat can catch the change of the file.
Second, when the reloader read the yml file, return empty.
check the code in cfgfile.Run:

		// Load all config objects
		configs := []*common.Config{}
		for _, file := range files {
			c, err := LoadList(file)
			if err != nil {
				logp.Err("Error loading config: %s", err)
				continue
			}

			configs = append(configs, c...)
		}

		debugf("Number of module configs found: %v", len(configs))

The structure []*common.Config{} require the yml like this:



    

  • input_type: log
    
paths:

      

    • d:\temp\test\log\test1*.log
      • 

    

    • 

  • input_type: log
    
paths:

      

    • d:\temp\test\log\test2*.log
      • 

    

    • 




Because of the filebeat.prospectors, it can't get anything.


Please help me to check the issue.

If I do something wrong, please tell me. Thanks.


By the way, if only use the main filebeat.yml. the reload also doesn't work because of it can't get the configurations.

            


            

              
              
              
            


          

          

            

              



              
                  
                  
              2
              
            

            

              
Do not use the former one requiring filebeat.prospectors. It's deprecated and will be removed in the future.


The new path settings only do support prospectors configuration in array style (without filebeat.prospectors). This is fully intentional and documented this way.

            


            

              
              
              
            


          

          

            

              



              
                  
                  
              3
              
            

            

              
Hi steffens, thanks for your reply. if I remove the filebeat.prospectors in the additional yaml, filebeat can't get the prospector configurations, as the reason I said above.

I tried to modify the source code in config.mergeConfigFiles as below:


	tmpConfig := []*common.Config{}
	err := cfgfile.Read(&tmpConfig, file)
	if err != nil {
		return fmt.Errorf("Failed to read %s: %s", file, err)
	}

	config.Prospectors = append(config.Prospectors, tmpConfig...)



It work normally.

            


            

              
              
              
            


          

          

            

              



              
                  
                  
              4
              
            

            

              
Sorry, but I can't follow you. Why exactly did you modify these lines? Which file? Line number?


What's you actual on-disk file layout (where did you put which file)?


Any log/error message? Have you tried to run filebeat with -d '*,config', to also print a json representation of the configuration?

            


            

              
              
              
            


          

          

            

              



              
                  
                  
              5
              
            

            

              
Which version of filebeat are you using? Could it be that you hit https://github.com/elastic/beats/pull/4227 ?

            


            

              
              
              
            


          

          

            

              



              
                  
                  
              6
              
            

            

              
@ruflin, I use the master branch two week ago in the github.

@steffens  , the source is :

beats/filebeat/config/config.go

line 91 ~ 99

it unpack the file settings as struct {Filebeat Config}.

I do the debug. if without filebeat.prospectors, it can get the yaml file, but return nothing after reading.

additional.yml:




    

  • input_type: log
    
paths:

      

    • d:\temp\test\log\test2*.log
      • 

    

    • 



            


            

              
              
              
            


          

          

            

              



              
                  
                  
              7
              
            

            

              
The above PR was only merged 7 days ago. So you should rebase on the current master.


As you are working on your own fork: What other changes did you made? What are your plans in enhancing filebeat? Interested to hear more.

            


            

              
              
              
            


          

          

            

              



              
                  
                  
              8
              
            

            

              
This topic was automatically closed after 21 days. New replies are no longer allowed.