Logstash stopped working (sincedb is not updated) after upgrade from 6.2.4 –> 6.4.0 (Update 6.5 doesn't work also)

Elastic Stack Logstash
61

Oh, I missed your answer, just saw it now. will try!

63

Okay, I thought it didn't work but actually I forgot to update the files. I just updated the files and they were consumed (and deleted by the way...)

Now moving to my second stage - try to do that from a network drive.....

64

@guyboertje well, now that it seemed to be working I hit another wall... I didn't want the files to be deleted so I looked at the optional configurations and found out that the files are always deleted regardless of what I choose... why is that?

I will tell you what my current workflow is to explain:

  • we collect logs from customers
  • the logs then need to go through another process which analyzes them and creates a JSON file with the results of the analysis. these results are then being sent to the directory that is being monitored by logstash
  • to make sure that we do not process the same log file again and again, the name of the analysis output file is predictable, and before running the analysis process we make sure that there is no such file yet, otherwise we might do work for nothing and have a lot of duplicates in ES, so I want to keep all these files in the directory.

This actually means that I can't use the read mode unless I somehow create a new process which duplicates these files - once for logstash to delete, and once for me to make sure if they existed...

65

Look at the setting file_completed_action

You can set this to "log" and provide a path to a file that will contain the paths of the completed files with file_completed_log_path setting.

66

Thanks, I saw that but I need the actual files not just the paths as I might run several things on the same file

67

The paths recorded in the "completed" file are full paths.

68

Correct, but I sometimes need the actual file more than once....

69

By the way, I just finished upgrade to version 7. unlike the other versions it wasn't an easy one at all... the upgrade assistance indeed helps you understand what will stop you from upgrading, but doesn't help you at all with the actual upgrade. unless you are very familiar elastic it isn't easy at all...

70

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.