Strange error - Can only start a prospector when all related states are finished

[Flaviu]

Hello,

I am getting this error messages every few minutes in /var/log/filebeat/filebeat

2017-12-10T19:08:52Z ERR Error creating prospector: Can only start a prospector when all related states are finished: {Id: Finished:false Fileinfo:0xc42045ca90 Source:/var/log/audit/audit.log Offset:6592549 Timestamp:2017-12-10 19:07:13.538039832 +0000 UTC TTL:-1ns Type:log FileStateOS:4469693-64769}

2017-12-10T19:12:12Z ERR Error creating prospector: Can only start a prospector when all related states are finished: {Id: Finished:false Fileinfo:0xc4202a1110 Source:/var/log/messages Offset:331457 Timestamp:2017-12-10 19:07:12.52187511 +0000 UTC TTL:-1ns Type:log FileStateOS:8726640-64769}

2017-12-10T19:12:12Z ERR Unable to create runner due to error: Can only start a prospector when all related states are finished: {Id: Finished:false Fileinfo:0xc4202a1110 Source:/var/log/messages Offset:331457 Timestamp:2017-12-10 19:07:12.52187511 +0000 UTC TTL:-1ns Type:log FileStateOS:8726640-64769}

How can I resolve it? The filebeat seems to work correctly, but I would like to get the error out.

filebeat version 6.0.1 (amd64), libbeat 6.0.1 on centos 7

Thank you,

Flaviu

[kvch (Noémi Ványi)]

It seems like a race ended badly between saving the state of /var/log/messages and other files and stopping Filebeat previously. Thus, the prospector of that log file cannot start.

Could you share the contents of your registry file?

[Flaviu]

I was not able to post the entire file because of the 7k limitation of the post characters

[  
   {  
      "source":"/var/log/elasticsearch/elasticsearch_index_search_slowlog.log",
      "offset":0,
      "timestamp":"2017-12-11T16:29:28.817055274Z",
      "ttl":-1,
      "type":"log",
      "FileStateOS":{  
         "inode":4202636,
         "device":64769
      }
   },
   {  
      "source":"/var/log/logstash/logstash-plain.log",
      "offset":330501,
      "timestamp":"2017-12-11T16:34:41.084992072Z",
      "ttl":-1,
      "type":"log",
      "FileStateOS":{  
         "inode":83891184,
         "device":64769
      }
   },
   {  
      "source":"/var/log/grubby_prune_debug",
      "offset":193,
      "timestamp":"2017-12-11T16:29:28.819050201Z",
      "ttl":-1,
      "type":"log",
      "FileStateOS":{  
         "inode":8566238,
         "device":64769
      }
   },
   {  
      "source":"/var/log/yum.log",
      "offset":4298,
      "timestamp":"2017-12-11T16:39:50.025673054Z",
      "ttl":-1,
      "type":"log",
      "FileStateOS":{  
         "inode":8409594,
         "device":64769
      }
   },
   {  
      "source":"/var/log/anaconda/ks-script-Kmgz5C.log",
      "offset":0,
      "timestamp":"2017-12-11T16:29:28.820669678Z",
      "ttl":-1,
      "type":"log",
      "FileStateOS":{  
         "inode":12584190,
         "device":64769
      }
   },
   {  
      "source":"/var/log/metricbeat/metricbeat.3",
      "offset":632,
      "timestamp":"2017-12-11T16:29:28.821269824Z",
      "ttl":-1,
      "type":"log",
      "FileStateOS":{  
         "inode":71303390,
         "device":64769
      }
   },
   {  
      "source":"/var/log/tallylog",
      "offset":0,
      "timestamp":"2017-12-11T16:29:28.821858471Z",
      "ttl":-1,
      "type":"log",
      "FileStateOS":{  
         "inode":8565890,
         "device":64769
      }
   },
   {  
      "source":"/var/log/wtmp",
      "offset":71771,
      "timestamp":"2017-12-11T16:39:50.025683344Z",
      "ttl":-1,
      "type":"log",
      "FileStateOS":{  
         "inode":8609422,
         "device":64769
      }
   },
   {  
      "source":"/var/log/elasticsearch/elasticsearch_index_indexing_slowlog.log",
      "offset":0,
      "timestamp":"2017-12-11T16:29:28.823138028Z",
      "ttl":-1,
      "type":"log",
      "FileStateOS":{  
         "inode":4202637,
         "device":64769
      }
   },
   {  
      "source":"/var/log/ntpstats/loopstats.20171211",
      "offset":463,
      "timestamp":"2017-12-11T16:40:19.25405861Z",
      "ttl":-1,
      "type":"log",
      "FileStateOS":{  
         "inode":33731674,
         "device":64769
      }
   },
   {  
      "source":"/var/log/btmp",
      "offset":0,
      "timestamp":"2017-12-11T16:39:49.010893985Z",
      "ttl":-1,
      "type":"log",
      "FileStateOS":{  
         "inode":8609421,
         "device":64769
      }
   },
   {  
      "source":"/var/log/cloud-init.log",
      "offset":100550,
      "timestamp":"2017-12-11T16:29:28.82496937Z",
      "ttl":-1,
      "type":"log",
      "FileStateOS":{  
         "inode":8409577,
         "device":64769
      }
   },
   {  
      "source":"/var/log/messages",
      "offset":220773,
      "timestamp":"2017-12-11T16:39:56.246407071Z",
      "ttl":-1,
      "type":"log",
      "FileStateOS":{  
         "inode":8726640,
         "device":64769
      }
   },
   {  
      "source":"/var/log/anaconda/ks-script-Umg17O.log",
      "offset":9342,
      "timestamp":"2017-12-11T16:29:28.826353966Z",
      "ttl":-1,
      "type":"log",
      "FileStateOS":{  
         "inode":12584191,
         "device":64769
      }
   }
   {  
      "source":"/var/log/metricbeat/metricbeat",
      "offset":66,
      "timestamp":"2017-12-11T16:29:28.836419875Z",
      "ttl":-1,
      "type":"log",
      "FileStateOS":{  
         "inode":71353620,
         "device":64769
      }
   }
]