Filebeat on Windows seem to not use the registry file

Elastic Stack Beats
1

Hello,

We have just migrated to Elastic Stack 5.2.

Filebeat version 5.2.1

We have filebeats running on Windows Server 2012 R2 and every time the filebeat service is restart all lines from all harvested logs gets send again.

The registry file is updated (Can be seen from the modification time of the file).
We have furthermore tried to close filebeat, delete the registry file, start filebeat which results in a new registry file being created which seems to be valid.
However, when the service is restarted after the new registry file is created all log lines gets send once more.

Filebeat configuration: https://gist.github.com/Steiniche/d2c62c6aaac71d989039346340412203
Registry file from a server: https://gist.github.com/Steiniche/5893b3b5ad8d6e5fb63f2004a3679129

1 Like
2

All the config options and the registry file seem to be as expected. Can you share some log output from filebeat, best in debug level? Especially the first 200 lines when starting filebeat again with an existing registry file would be interesting. As the lines will not fit in the forum, best post them into a gist and link it here.

3

Hi @ruflin thanks for helping out!

Turning on the debug log quickly produced many 1MB log files which contains mostly publish events - this confirms my suspicion that everything gets send again.
I have taken the first ~100 lines and posted here: https://gist.github.com/Steiniche/029069e134aa232f8cee30142b98f4ef
However, I have only included the first Publish event.

4

Thanks for the logs. It seems that filebeat first finds the states in the registry: States Loaded from registrar: 21 but then fails to match the files to the prospectors and prospectors are started without states. I remember we had an issue about path matching in the 5.0-beta versions but this should have been fixed. I'm curious if this is a similar issue again that it does not match C:/logs/a/server.log and C:\/logs\/a\/server.log from the registry file.

From which version of filebeat were you migrating?

5

No problem.

We upgraded from 1.3.1 to 5.2.1

6

Can you check if the problem persist in case you start with an empty registry file in 5.2.1, stop filebeat and start filebeat again?

7

I have now tried deleting the old registry files and restarted filebeat a couple of times.
It does however not work and events still get resend.

Anymore ideas?

8

That is really strange :frowning: Could you share again the log file and registry from 5.2.1 (same as above) so I can have a look again, now without the migration.

9

I agree with you @ruflin it is pretty strange.

1st startup with clean registry: https://gist.github.com/Steiniche/eda6d15b035efc578587d6df036e5546

2nd startup using registry from 1st startup: https://gist.github.com/Steiniche/eb2d8fffd10080b72b41a3c419f00df0

10

I really need to do some testing for this on a Windows machine and try to reproduce it. I'm probably only going to be able to do this next week.

The part that bugs me: In case it is a "general" bug it would affect a lot of user and I would hope it would have popped up much earlier. At the same time, users don't restart filebeat often. Will definitively dig deeper into this one.

1 Like
11

@ruflin Another similar issue: Duplicate events with Filebeat on windows on service restart

1 Like
12

Yeah this looks like it's exactly the same issue, should I close my thread?

I'll be back on Monday to do that though :slight_smile:

13

No need to close the thread as both have additional infos inside.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.