Logstash filter with fields in the log file itself

Elastic Stack Logstash
1

I have Tomcat log file as below. If you see the log file contains Token Id, Request Id. I would like to filter out (filter in .conf file) data from this log file in such a way that i can see all the logs in one shot where TokenId = 3321 (as an example) on Kibana Discover page; similarly there will be use-case where i want to see all the logs with RequestId 2345874 on Kibana Discover page. How do we do this? I tried by writing below filter, but it did not work. Please advise.

match => { "message" => ["%{COMBINEDAPACHELOG}", "TokenId: %{NUMBER:TokenId}"] }

Tomcat_09272018.log
60.183.171.212 - - [27/Sep/2018:00:00:15 +0000] TokenId : 3331 :: RequestId:2345874 :: Portfolio Id: 28
180.186.112.151 - - [27/Sep/2018:00:00:20 +0000] TokenId: 3321 :: RequestId:2343474 :: Portfolio Id: 29
168.72.204.145 - - [27/Sep/2018:00:00:25 +0000] TokenId: 3241 :: RequestId:2345879 :: Portfolio Id: 23
84.204.225.40 - - [27/Sep/2018:00:00:30 +0000] TokenId: 3321 :: RequestId:2343474 :: Portfolio Id: 29
184.72.89.211 - - [27/Sep/2018:00:00:35 +0000] TokenId: 3331 :: RequestId:2345874 :: Portfolio Id: 28
152.213.20.112 - - [27/Sep/2018:00:00:40 +0000] TokenId: 3321 :: RequestId:2343474 :: Portfolio Id: 29
100.201.134.191 - - [27/Sep/2018:00:00:45 +0000] TokenId: 3331 :: RequestId:2345874 :: Portfolio Id: 28
196.174.214.227 - - [27/Sep/2018:00:00:50 +0000] TokenId: 3241 :: RequestId:2345879 :: Portfolio Id: 23

2

That is not a common Apache log format, so it is not surprising that it is not working. Have a look at this introductory blog which goes through how to use Logstash and build custom patterns.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.