How To classify and follow logs from different log files and machines

[Julien_Hac]

Hi everyone,

I need to treat different source log file and see then in Kibana :

• Type log file - as batch log, server application log, apache server log
• Log from different machines

Here is my installation :

• First the machine send log to Redis by Filebeat,
• then Logstash indexer retreive the log and send to elasticsearch,
• then Kibana retreive the datas in elasticsearch

How can i distinguish log from apache or server app in kibana ?

How can i distinguish log from different machines ?

Of course the logs should not be merge otherwise i couldn't read the log sequentially... how to follow a type of log sequentially to track it easily ?

Thank you for help

[josephjohney]

You can distinguish each log or patterns in most of these applications

Try to use document_type in filebeat,which is ver useful in logstash ( it will be 'type'in logstash ) and will be part of the metadata.
Also use tags and fields in Filebeat.

From logstash try tocreate elasticsearch indices based on the type..

Also try to create indexes in uniform formats so that a pattern consolidating many indices could be defined in kibana

[Julien_Hac]

Thank for you answer,
So im going to add document_type for my different log file and create different elasticsearch index in some cases.

But, in order to distinguish the mulitple machines log (i have a cluster of different machines), can i use the beat.hostname ? If i use it i wouldn't need to add a redundant new field to know if the log is from host1 or host2 etc...
so i could filter on beat.hostname in kibana to identify a same kind of log with different machine.