Logstash stopped working (sincedb is not updated) after upgrade from 6.2.4 –> 6.4.0 (Update 6.5 doesn't work also)

Elastic Stack Logstash
1

Hi,
In continuation of the following topic which is now closed:

We found no solution for that. moreover - we just tried installing Logstash 6.5 on a new machine and we get the same results - the logstash runs, but sincedb is never updated. it is being created and "touched" when logstash starts running, but then nothing happens, no new file is added, no existing file is being updated.
The same configuration worked on all previous version (starting from 2.x, 4.x, 5.x, 6.2.4....)
Any way I can debug it and understand?
Really need some help here,
Thanks!

1 Like
2

Hi,

I'm encountering a similar issue.
A couple of days ago, I installed the latest elasticsearch-6.5.3 with kibana-6.5.3-windows-x86_64 and logstash-6.5.3 on a fresh Windows server 2012r2. couldn't make this setup to work.
It seems all components are running with no errors, but no indexing occurs (nothing is written to sincedb file).
Following the advise above, I installed logstash-6.2.4 (instead of logstash-6.5.3 and without any other change) and now the setup is running with no issues.
What can be done to debug this issue further?

Thank you!

1 Like
3

Anyone? any idea? we really wont to understand why logstash isn't working for us//////

4

@guyboertje, I understand that you did the pull request that changed the file input. it's been months now that we cannot work with the new versions as they just ignore our file inputs. please let me know what we can do to debug this and make this work again... I am stuck with 6.2.4.

5

Enable --log.level debug and see what the filewatch has to say.

6

@Badger the log doesn't say anything. I can post a "good" log vs "bad" log, the changes are minimal

7

@RonGros

This is windows, correct?

If so, yes, I did refactor the way the stat call is done on windows - to use a different Kernel32 dll call. I looks like this fails silently in some circumstances. I did and have done some tests in Google Compute Cloud on windows servers but I did not create a realistic "prod" environment.

I have been meaning to make some time to fix this but we are a small team with too much to do.

Can you use filebeat?

8

@guybowden so it seems that windows is completely broken. I tried moving the files from the network to a local file, it still doesn't work.

Regarding filebeat, I can try, it will probably take me a couple of weeks to set it up as we are under a lot of stress here, but in general i would wish logstash to work again...
currently it is holding me from upgrading all the other versions as well, and it prevents me from having the "Read" mode which I really need

9

I understand. I try to get some time to work on this shortly. I'm going to FOSSDEM this weekend so maybe.

10

That would be great

11

Hi @guyboertje, just keeping this thread alive, I guess there are no updates from your side.
I am trying to use the filebeat, will update when I can

12

@guyboertje an update - I installed filebeat and configured everything. it has the same problem - it doesn't deal well with windows network paths, so it does not help us, it also makes the architecture much more complex and requires multiple filebeat instances which I don't really like anyhow, as someone said in one of the forums - if you are going to use logstash anyway (and I do because I need to make some adjustments on the files), there is no sense in using filebeat....

so bottom line - it doesn't work, and it seems that the changes made in the file input broke logstash for network files. can you please try to see if you can get it working again?

13

Just checking.
Are you using / forward slashes or \ in your paths?

14

I am using / slash in my paths. but I Think i also tried the normal windows backslash

15

Forward slashes are best for LS on Windows.

16

Yes, that's what I thought. I actually got the logstash configuration as inheritance from someone and he used the forward slashes which is very uncommon in windows so I figured he had issues with backslashes.....

17

Did you run LS in trace log level?

18

I don't think so, I can do it now if it helps?

I am actually trying to run it on a local directory right now (which works with filebeat, but I wanted to check logstash again) and it looks like even that isn't working. I am still doing some experiments.

19

FYI...

The logging API allows for different levels of logging for different components in LS.

First do curl -XGET 'localhost:9600/_node/logging?pretty'
You see something like this:

{
  "host" : "Elastics-MacBook-Pro.local",
  "version" : "6.4.0",
  "http_address" : "127.0.0.1:9600",
  "id" : "8789409b-7126-4034-9347-de47e6ce12a9",
  "name" : "Elastics-MacBook-Pro.local",
  "loggers" : {
    "filewatch.discoverer" : "INFO",
    "filewatch.observingtail" : "INFO",
    "filewatch.sincedbcollection" : "INFO",
    "filewatch.tailmode.handlers.createinitial" : "INFO",
    "filewatch.tailmode.processor" : "INFO",
    "logstash.agent" : "INFO",
    "logstash.api.service" : "INFO",
    "logstash.codecs.json" : "INFO",
    ...
    "logstash.filters.grok" : "INFO",
    "logstash.filters.date" : "INFO",
    "logstash.inputs.file" : "INFO",
    ...
    "logstash.outputs.stdout" : "INFO",
    "logstash.pipeline" : "INFO",
    ...
    "slowlog.logstash.codecs.json" : "INFO",
    "slowlog.logstash.codecs.rubydebug" : "INFO",
    "slowlog.logstash.filters.date" : "INFO",
    "slowlog.logstash.inputs.file" : "INFO",
    "slowlog.logstash.outputs.stdout" : "INFO"
  }
}

Using the API
Turn DEBUG on for just the date and grok filters:

curl -XPUT 'localhost:9600/_node/logging?pretty' -H 'Content-Type: application/json' -d'
{
    "logstash.filters.date" : "DEBUG"
    "logstash.filters.grok" : "DEBUG"
}
'

Turn trace off:

curl -XPUT 'localhost:9600/_node/logging?pretty' -H 'Content-Type: application/json' -d'
{
    "logstash.filters.date" : "WARN"
    "logstash.filters.grok" : "WARN"
}
'

Or

curl -XPUT 'localhost:9600/_node/logging/reset?pretty'

NOTE: it might be a good idea to start LS with logging set to WARN in the logstash.yml so other logging is less verbose.

20

Use windows equivalents for curl

Any component starting with filewatch needs to be set to TRACE