Logstash filter with fields in the log file itself

I have Tomcat log file as below. If you see the log file contains Token Id, Request Id. I would like to filter out (filter in .conf file) data from this log file in such a way that i can see all the logs in one shot where TokenId = 3321 (as an example) on Kibana Discover page; similarly there will be use-case where i want to see all the logs with RequestId 2345874 on Kibana Discover page. How do we do this? I tried by writing below filter, but it did not work. Please advise.

match => { "message" => ["%{COMBINEDAPACHELOG}", "TokenId: %{NUMBER:TokenId}"] }

Tomcat_09272018.log
60.183.171.212 - - [27/Sep/2018:00:00:15 +0000] TokenId : 3331 :: RequestId:2345874 :: Portfolio Id: 28
180.186.112.151 - - [27/Sep/2018:00:00:20 +0000] TokenId: 3321 :: RequestId:2343474 :: Portfolio Id: 29
168.72.204.145 - - [27/Sep/2018:00:00:25 +0000] TokenId: 3241 :: RequestId:2345879 :: Portfolio Id: 23
84.204.225.40 - - [27/Sep/2018:00:00:30 +0000] TokenId: 3321 :: RequestId:2343474 :: Portfolio Id: 29
184.72.89.211 - - [27/Sep/2018:00:00:35 +0000] TokenId: 3331 :: RequestId:2345874 :: Portfolio Id: 28
152.213.20.112 - - [27/Sep/2018:00:00:40 +0000] TokenId: 3321 :: RequestId:2343474 :: Portfolio Id: 29
100.201.134.191 - - [27/Sep/2018:00:00:45 +0000] TokenId: 3331 :: RequestId:2345874 :: Portfolio Id: 28
196.174.214.227 - - [27/Sep/2018:00:00:50 +0000] TokenId: 3241 :: RequestId:2345879 :: Portfolio Id: 23

That is not a common Apache log format, so it is not surprising that it is not working. Have a look at this introductory blog which goes through how to use Logstash and build custom patterns.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.