Websphere multiline logs

Elastic Stack Logstash
1

This is what my log file looks like:
[3/12/18 5:28:05:364 EDT] 00000056 AutowiredAnno I org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor JSR-330 'javax.inject.Inject' annotation found and supported for autowiring

i have used the following multiline codec:

multiline{
pattern => "^DATESTAMP"
negate => true
what => previous
}

and used the following grok pattern:
grok{
match => { "message" => "%{DATESTAMP:TIMESTAMP} EDT] %{WORD:ID} %{WORD:CLASS} %{WORD:event_type} %{GREEDYDATA:MSG}" }
}
Logstash pipeline starts but i dont see any logs being parsed.

2

pattern => "^DATESTAMP"

Two problems:

  • Your log line actually begins with [ so you need to include that (escaped with a backslash).
  • You need to reference the pattern as %{DATESTAMP}.
3

hi magnus,
i made the changes as you said but the logs are not getting parsed.(the pipeline starts but nothing happens after that)
here is my full configuration:

input{
file{
path=>"C:/Users/Mahe/Desktop/internship/sample1.log"
start_position=>"beginning"
sincedb_path => "/dev/null"
codec=>multiline{
pattern => "^[%{DATESTAMP}"
negate => true
what => previous
}
}
}

filter{

grok{
		match => { "message" => "%{DATESTAMP:TIMESTAMP} EDT] %{WORD:ID} %{WORD:CLASS} %{WORD:event_type} %{GREEDYDATA:MSG}" }
}

}

output{
elasticsearch {
hosts =>"localhost"
index=>"my1.0"
document_type=>"my_logs"

 }
stdout {}

}

4

is there any problem in the grok filter

5

sincedb_path => "/dev/null"

On Windows use "nul" instead of "/dev/null".

6

hi magnus,
no change

Sending Logstash's logs to C:/logstash-5.4.1/logs which is now configured via log4j2.properties
[2018-05-23T12:05:36,500][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2018-05-23T12:05:36,508][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-05-23T12:05:36,717][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x36a1f3ad URL:http://localhost:9200/>}
[2018-05-23T12:05:36,724][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-05-23T12:05:36,853][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword"}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-05-23T12:05:36,879][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<URI::Generic:0x7d51463d URL://localhost>]}
[2018-05-23T12:05:36,961][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2018-05-23T12:05:37,552][INFO ][logstash.pipeline ] Pipeline main started
[2018-05-23T12:05:37,742][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

pipe line gets started but the logs are not parsed

7

What happens if you disable the multiline codec?

8

no change even if the multiline codec is disabled,pattern seems to work fine where i used it on the grok debugger

9

[3/12/18 5:28:05:364 EDT] 00000056 AutowiredAnno I org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor JSR-330 'javax.inject.Inject' annotation found and supported for autowiring

%{DATESTAMP:TIMESTAMP} EDT] %{WORD:ID} %{WORD:CLASS} %{WORD:event_type} %{GREEDYDATA:msg}

this is the output i get on grok debugger:
{
"TIMESTAMP": [
[
"3/12/18 5:28:05:364"
]
],
"ID": [
[
"00000056"
]
],
"CLASS": [
[
"AutowiredAnno"
]
],
"event_type": [
[
"I"
]
],
"msg": [
[
"org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor JSR-330 'javax.inject.Inject' annotation found and supported for autowiring"
]
]
}

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.