Log stash filter for multi line occurrences and capture in custom field

Elastic Stack Logstash
1

We are having a huge log file in which we are applying many filter and storing it as a specific filed in that below line has, more than one occurance(3 to 5 times), In that we wanted to capture last occurance under filed name called "build_Status". But for us either we are getting nil value for build_Status or getting lots of lines next to that line Current state: job state = 'Succeded' . Please review below and suggest us
Log File example:
[2019-02-04 07:16:46Z INFO StepsRunner] Current state: job state = 'Succeeded'
Some Additional lines
[2019-02-04 07:26:46Z INFO StepsRunner] Current state: job state = 'Failed'
Some Additional lines
[2019-02-04 07:28:46Z INFO StepsRunner] Current state: job state = 'Canceled'
Some Additional lines
[2019-02-04 07:32:46Z INFO StepsRunner] Current state: job state = 'Failed'
Some Additional lines

Logstash Config file:
input {
beats {
client_inactivity_timeout => 1200
port => 5002
}
}

filter
{
if [message] =~ "Version: 2.122.1"
{
grok {
add_tag => [ "start" ]
match => { "message" => "%{TIMESTAMP_ISO8601:build_StartTime}" }
}
}
grok {
add_tag => [ "start" ]
match => { "message" => "(?<build_ErrorMessage> ERR .*)" }
}
if [message] =~ "Current state: job state"
{
grok {
add_tag => [ "start" ]
match => { "message" => "Current state: job state = '(?<build_Status>.+)'" }
}
if ![build_Status] or [build_Status] == 'nil'
{
mutate {
add_tag => [ "start" ]
add_field => {"build_Status" => "Succeded"}
}
}
}
if [message] =~ "Job completed."
{
grok {
add_tag => [ "start" ]
match => { "message" => "%{TIMESTAMP_ISO8601:build_EndTime}" }
}
}

grok {
add_tag => [ "end" ]
match => { "message" => "Job completed." }
}
if "start" in [tags] {
aggregate {
task_id => "%{source}"
code => "
map['build_Status'] = event.get('build_Status') unless event.get('build_Status').nil?
map['build_ErrorMessage'] = event.get('build_ErrorMessage') unless event.get('build_ErrorMessage').nil?
map['build_StartTime'] = event.get('build_StartTime') unless event.get('build_StartTime').nil?
map['build_EndTime'] = event.get('build_EndTime') unless event.get('build_EndTime').nil?
"
}
}

if "end" in [tags] {
aggregate {
task_id => "%{source}"
code => "
event.set('build_ErrorMessage', map['build_ErrorMessage'])
event.set('build_Status', map['build_Status'])
event.set('build_StartTime', map['build_StartTime'])
event.set('build_EndTime', map['build_EndTime'])
"
end_of_task => true
}
}
if "end" not in [tags] {
drop { }
}
mutate {
remove_field => [ "message" ]
}
mutate {
remove_tag => [ "start" ]
}

}

output {
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "tfslog-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}

FileBeat Config:

filebeat:
prospectors:

paths:

  • C:/*********** /Worker_ .log
    multiline:
    pattern: '^[.+]'
    negate: true
    match: after
    max_line: 999
    timeout: 60
    output.logstash:
    hosts: ["Logstash_Server:5002"]

OutPut:
""build_StartTime" => "2019-02-05 07:50:03Z"
"build_BuildId" => "1448697",
"build_ErrorMessage" => nil,
"beat" => {
"hostname" => "########",
"version" => "6.3.0",
"name" => "#######"
},
"build_Status" => nil,
"build_EndTime" => "2019-02-05 09:40:03Z"
Some Times build_Status contains:
"build_Status" => "'\n[2019-02-05 07:50:32Z INFO JobRun ner] Job result after all pre-job steps finish: \n[2019-02-05 07:50:32Z INFO Job Runner] Run all job steps.\n[2019-02-05 07:50:32Z INFO StepsRunner] Processing s tep: DisplayName='$/Roche.DP.NewGen/BuildTools/Dev/Cake/build.cmd', ContinueOnEr ror=False, Enabled=True\n[2019-02-05 07:50:32Z INFO ExpressionManager] Evaluatin g: succeeded()\n[2019-02-05 07:50:32Z INFO ExpressionManager] Expanded: True\n[2 019-02-05 07:50:32Z INFO ExpressionManager] Result: True\n[2019-02-05 07:50:32Z etc

2

That configuration never sets build_DefinitionName, so it will drop all events. You are showing us one configuration and asking us to help fix the behaviour of a different configuration. We cannot do that.

3

Thanks for the reply. I used or condition so if build_DefinitionName is there then it will drop else it will ignore. But as of now I removed the build_DefinitionName part from above config. Now please have a look and help us.

We need to Capture build_Status = Failed or build_Status = Succeed or build_Status = Canceled from the below log line, Which has multiple occurrences and we wanted to capture the final occurrence
[2019-02-04 07:16:46Z INFO StepsRunner] Current state: job state = 'Succeeded'

4

You are doing an aggregate with a non-existent field as task_id. That does not work.

5

No It's working for us. I guess you didn't get our issue or I am not explaining it properly anyhow we can close this

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.