Can Elasticsearch extract fields from multiple log files? Can we write this code in a single file?

Elastic Stack Beats
1

Hi Friends,

I am monitoring /var/log/* in my linux box through Filebeat. Every log file have it's own style. I am getting the event like below (sample log file)
"message:b8tTpts/0��3T(:a� pts/0ts/0SAHISD+jocktramen10.33.111.11f>Te @version:1 @timestamp:January 19th 2016, 16:38:32.732 beat.hostname:NSAH-PC1169-1 beat.name:NSAH-PC1169-1 count:1 fields: - input_type:log offset:109,405 ** source:/var/log/wtmp** type:syslog host:NHCLT-PC1169-1 _id:AVJZkgL5augbAVuIOsWM _type:syslog _index:filebeat-2016.01.19 _score:"

From above event I couldn't able to make a meaningful dashboard in Kibana. I cannot Extract new fields as every log file have it's own style of logging. Can I write code in Logstash , that extract from multiple log files?

In Elasticsearch document there i found /var/log/syslog. But I didn't found in my Linux box.

Is Syslog is a software? will it take all the logfiles, and convert it into Single format i.e., Syslog format?

2

From above event I couldn't able to make a meaningful dashboard in Kibana.

/var/log/wtmp is a binary file that you won't be able to read in a meaningful way with Filebeat. Another reason not to tell Filebeat to read /var/log/* is that some events are logged in multiple files so if you read them all you'll get duplicates. Check your syslog configuration to see how things are set up. Chances are all events will go to /var/log/syslog and then there's no point in reading any other files.

I cannot Extract new fields as every log file have it's own style of logging. Can I write code in Logstash , that extract from multiple log files?

Yes. For example, grok filters can be configured to try multiple patterns until it gets a match.

Is Syslog is a software? will it take all the logfiles, and convert it into Single format i.e., Syslog format?

I'm not sure what you're asking here.

3

How to read wtmp files through filebeat and send to logstash instance?.

4

How to read wtmp files through filebeat and send to logstash instance?.

As I said earlier: /var/log/wtmp is a binary file that you won't be able to read in a meaningful way with Filebeat.