Filter in input section

[Nitin_gupta]

Hi ,

i added fields in filebeat configuration to identify different types of files

My filebeat configuration

• input_type: log
  paths:
  • c:\Users\user1\Desktop\ELK\server_sftp.log.D*
    fields:
    type_log: sftp_server
• input_type: log
  paths:
  • c:\Users\user1\Desktop\ELK\server_ftp.log.D*
    fields:
    type_log: ftp_server
• input_type: log
  paths:
  • c:\Users\user1\Desktop\ELK\client.log.D*
    fields:
    type_log: client

because Server and client logs are different in format
server logs start with timestamp in brackets eg:- [2017-09-19 00:00:27.741]
whereas client logs start with timestamp without brackets eg:- 2017-09-19 00:00:27.741

My issue is , i am including codec code for multiline in logstash input {} .
is it possible to identify different types of logs and use multiline .

what i am expecting in logstash configuration is below .

• type_log is the field included from filebeat configuration(above)

input{
beats{
port=>5044
if[type_log] =~ "client" {
start_position => "beginning"
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601}"
negate => true
what => previous
}
}else if[type_log] =~ "server"{
codec => multiline {
pattern => "^[%{TIMESTAMP_ISO8601}]"
negate => true
what => previous
}
}
}
}

Please guide.

Regards
Nitin

[magnusbaeck (Magnus Bäck)]

Don't use multiline with the beats input on the Logstash side. You always want to do multiline processing as close to the source as possible.

[Nitin_gupta]

Thanks @magnusbaeck

i will enable multiline in filebeat .

Is it possible to enable multiline for different input prospectors ?

for example
different multiline pattern for

input_type: log
paths:
c:\Users\user1\Desktop\ELK\server_ftp.log.D*
fields:
type_log: ftp_server

and different pattern for

input_type: log
paths:
c:\Users\user1\Desktop\ELK\client.log.D*
fields:
type_log: client

or i need to use 2 different filebeat ??

[magnusbaeck (Magnus Bäck)]

The multiline configuration is specific to each prospector.