Logstash stopped working (sincedb is not updated) after upgrade from 6.2.4 –> 6.4.0 (Update 6.5 doesn't work also)


(Ron Grosberg) #61

Oh, I missed your answer, just saw it now. will try!


(Ron Grosberg) #63

Okay, I thought it didn't work but actually I forgot to update the files. I just updated the files and they were consumed (and deleted by the way...)

Now moving to my second stage - try to do that from a network drive.....


(Ron Grosberg) #64

@guyboertje well, now that it seemed to be working I hit another wall... I didn't want the files to be deleted so I looked at the optional configurations and found out that the files are always deleted regardless of what I choose... why is that?

I will tell you what my current workflow is to explain:

  • we collect logs from customers
  • the logs then need to go through another process which analyzes them and creates a JSON file with the results of the analysis. these results are then being sent to the directory that is being monitored by logstash
  • to make sure that we do not process the same log file again and again, the name of the analysis output file is predictable, and before running the analysis process we make sure that there is no such file yet, otherwise we might do work for nothing and have a lot of duplicates in ES, so I want to keep all these files in the directory.

This actually means that I can't use the read mode unless I somehow create a new process which duplicates these files - once for logstash to delete, and once for me to make sure if they existed...


(Guy Boertje) #65

Look at the setting file_completed_action

You can set this to "log" and provide a path to a file that will contain the paths of the completed files with file_completed_log_path setting.


(Ron Grosberg) #66

Thanks, I saw that but I need the actual files not just the paths as I might run several things on the same file


(Guy Boertje) #67

The paths recorded in the "completed" file are full paths.


(Ron Grosberg) #68

Correct, but I sometimes need the actual file more than once....


(Ron Grosberg) #69

By the way, I just finished upgrade to version 7. unlike the other versions it wasn't an easy one at all... the upgrade assistance indeed helps you understand what will stop you from upgrading, but doesn't help you at all with the actual upgrade. unless you are very familiar elastic it isn't easy at all...