I try to parse this dataset: https://github.com/logpai/loghub/tree/master/Android using logstash.
I have tried using grok filter but some parts of the log contains multiple templates.
example of log:
03-17 16:13:38.811 1702 2395 D WindowManager: printFreezingDisplayLogsopening app wtoken = AppWindowToken{9f4ef63 token=Token{a64f992 ActivityRecord{de9231d u0 com.tencent.qt.qtl/.activity.info.NewsDetailXmlActivity t761}}}, allDrawn= false, startingDisplayed = false, startingMoved = false, isRelaunching = false
03-17 16:13:38.819 1702 8671 D PowerManagerService: acquire lock=233570404, flags=0x1, tag="View Lock", name=com.android.systemui, ws=null, uid=10037, pid=2227
03-17 16:13:38.820 1702 8671 D PowerManagerService: ready=true,policy=3,wakefulness=1,wksummary=0x23,uasummary=0x1,bootcompleted=true,boostinprogress=false,waitmodeenable=false,mode=false,manual=38,auto=-1,adj=0.0userId=0
03-17 16:13:38.839 1702 2113 V WindowManager: Skipping AppWindowToken{df0798e token=Token{78af589 ActivityRecord{3b04890 u0 com.tencent.qt.qtl/com.tencent.video.player.activity.PlayerActivity t761}}} -- going to hide
03-17 16:13:38.859 2227 2227 D TextView: visible is system.time.showampm
My grok filter:
filter {
grok {
patterns_dir => ["./patterns"]
match => { "message" => "%{MONTHANDDAY:date}%{SPACE}%{TIME:time}%{SPACE}%{NUMBER:pid}%{SPACE}\
%{NUMBER:tid}%{SPACE}%{WORD:level}%{SPACE}%{WORD:App}:%{SPACE}%{GREEDYDATA:INFO}" }
}
}
Result: Logstash can handle log with same structure. How about the postfix with 100+ templates? Do I need to manually define grok pattern for each template or what is the correct tool for this?