Logstash file input not working

[Beuhlet_Reseau (Newbie)]

Hello,

I have a question.

Before i used filebeat to send logs to logstash. I decided (to better performance) to delete filebeat.

Now i use file input in logstash :

 file {
   path => "/home/cra_elk/*"
   type => "cra"
   #start_position => "beginning"
   #sincedb_path => "/dev/null"
 }

If i not use sincedb_path => "/dev/null" it doesn't work.

Files i try to upload are created yesterday (2017041818h). I don't understand.

[magnusbaeck (Magnus Bäck)]

Logstash thinks it has already processed the files so it's just waiting for more data to be appended to hem. Setting the sincedb_path option like that resets the saved state about the current position in the file, so that combined with start_position => "beginning" will make Logstash read the files from the top. Alternatively you can delete the current sincedb files to reset the state.

See the file input's documentation and countless previous threads on this topic.

[pablosan (Pablo)]

Also check the ignore_older setting in https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html

[Beuhlet_Reseau (Newbie)]

Ok it's work now i don't know why

I don't find sincedb file, i deleted it there is few days and i believe It has not been created since.

Thank you for your help

[Beuhlet_Reseau (Newbie)]

Furthermore

Logstash prefer whats syntaxe ? :

Method 1 :

 file {
   path => "/data/serveur_*/elkf/DC7_*.txt"
   type => "cra"
 }

Method 2 :

file {
       path => [ "/data/serveur_1/elkf/DC7_*.txt",   "/data/serveur_2/elkf/DC7_*.txt",   "/data/serveur_3/elkf/DC7_*.txt" ]
       type => "cra"
     }

Method 3

 file {
   path => "/data/serveur_1/elkf/DC7_*.txt"
   type => "cra"
   path => "/data/serveur_2/elkf/DC7_*.txt"
   type => "cra"
   path => "/data/serveur_3/elkf/DC7_*.txt"
   type => "cra"
     }

?

[magnusbaeck (Magnus Bäck)]

The first two are fine. I'd avoid the last one even if it might work.

[guyboertje]

@Beuhlet_Reseau - the glob patterns are taken from the OS - https://en.wikipedia.org/wiki/Glob_(programming)

path => "/data/serveur_[1-3]/elkf/DC7_*.txt" should work too if you want to prevent file discovery from serveur_5 for example.

FWIW: as far as I can recall, with Method 2 you can control the order of discovered files better - if you wanted, say, to read serveur_3 before serveur_1.

Other than that, follow Magnus' advice.

[Beuhlet_Reseau (Newbie)]

Yes the controle is the key ! I think turn me to

"/data/serveur_*/elkf/DC7_*.txt" because i don't need order.

However I find the upload quite bizarre. I explain me :

Log file created at 12h06,
Discover available about this file in kibana : 12h11,
Timestamp display in Discover about this file : 12h07.

It's correct but i find bizarre the gap of 5 minutes for upload. (but the timestamp is correct )

It is surely nothing but ... The main thing is that it works

Thank you both @warkolm @guyboertje.