How to parse logs on startup and merge files per folder?

Elastic Stack Logstash
1

Dear Community,

I had troubles with parsing of logs with logstash during startup

Once I add an exta line to log file -> parsed;
It seems that parsing only is triggerred once change is observer

  1. Is there a setting for the input? I've seen http polling, but is there somehting as polling time for a file not http request?
  2. I do have number of folders , each folder contains 2 files . Is that possible to parse folder by folder and concatenate fields from these 2 files into a signe json?

Folder o_1: obs.out, log.out
Folder o_2: obs.out, log.out
Json formed from timestamp from obs.out and all messages from log.out
So this will be a JSON with "timestamp" :

2

Yes, by default.

All your questions really depend on what your config looks like.

3

My current config at this points looks like this. It is currently only for a single in, single out (don't know so far how to merge two file per folder)

input {
file {
path => "C:\logstash-5.6.1\logstash-5.6.1\0_inputs\netstat_An.out"
start_position => "beginning" }
}

filter {
grok {
patterns_dir => ["C:\logstash-5.6.1\logstash-5.6.1\0_patterns"]
match => { "message" => "%{SPACE} %{WORD:methodtcp} %{SPACE} %{IP:localIp}:%{NUMBER:localport} %{SPACE} %{IP:foreignIp}:%{NUMBER:foreignport} %{SPACE} %{STATUS:status}" }
}
}

output {

file {
path => "C:\logstash-5.6.1\logstash-5.6.1\0_outputs\output.json"
codec =>
line {
format => "out:%{localIp}"
}

}

}

4

What do you mean by that?

5

I've figure out how to parse 2 files with 2 file scetion and 2 pths in Input plugin

Is that possible to somehow trigger the reading of file without adding extra lines?

Seems it work only if you add something, like new log records written to a file

Note: we try to parse logs locally from a floder on logstash start
I will certainly appreciate your advise on this

6

That is the only way.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.