How to solve in Kibana: Visualization of specific log entries with additional information?

Elastic Stack Kibana
1

Hi all,

We want to migrate existing script based solution into Kibana, with following features:

  • handling multiple specific(fetching with regular expressions) text log entries.
    Should be not peformant has to run only several times each day.
  • on each fetched log entry we should show specific additional related configuration information.
  • We want to visualize in a table view with customized header row and column.

How i achieve this in Kibana. Please give me a hint regarding:

  • which is in this situation the best way to process log entries?
  • How can I visualize it as described in the beginning?
2

Hi there,

Are you currently indexing your log data into Elasticsearch? If so, can you provide a sample log and the Elasticsearch mapping you're using?

Thanks!

3

Hello Lukas,

yes, we are indexing it but at this stage the important text part(position right after loglevel column information) on each line is not indexed (GREEDYDATA).
Below are some sample log rows, highlighted text parts (number and string) are entries for fetching related row. Numbers are also keys for additional Info in order to show later on in a view somewhere

ltimestamp threadid loglevel textpart
2017-03-22 T04:28:04.280 [0x7f6d92b67700] INFO - [SFE-00342]: SFE-00342 market housekeeping starts
2017-03-22 T05:57:05.271 [0x7f6d92b67700] INFO - [SFE-01350]: SFE-01350 market housekeeping starts
2017-03-22 T07:27:01.285 [0x7f70a4287700] WARN - No tick denominator available for listing: 933984583,113,0 Prov: 152
2017-03-22 T04:28:12.021 [0x7f6d92b67700] ALERT - [SFE-00342]: updateSodHKStatus SFE-00342 SODHK for market 342 terminates after
2017-03-22 T05:57:06.404 [0x7f6d92b67700] ALERT - [SFE-01350]: updateSodHKStatus SFE-01350 SODHK for market 1350 terminates after

Below is current mapping visible

Thanks in advance!

"filebeat-2017.03.21" : {
"mappings" : {
"log" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"beat" : {
"properties" : {
"hostname" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"debugtimestamp" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"host" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"input_type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"loglevel" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ltimestamp" : {
"type" : "date"
},
"message" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"offset" : {
"type" : "long"
},
"source" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"tags" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"threadid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
},

4

It looks like you're using Filebeat. Have you tried using the sample Filebeat Kibana dashboards?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.