Error in structuring log file

Sripal · November 26, 2018, 6:46am

Hi all,

I'm new to ELK stack.
I'm trying to read the system error log, but i am unable to achieve. Here are my exploration.

SysErr Lof file sample:

[9/3/18 12:19:34:644 IST] 00000ff6 SystemErr [9/3/18 12:19:34:645 IST] 00000ff6 SystemErr [9/3/18 12:19:34:645 IST] 00000ff6 SystemErr [9/3/18 12:19:34:645 IST] 00000ff6 SystemErr [9/3/18 12:19:34:645 IST] 00000ff6 SystemErr [9/3/18 12:19:34:645 IST] 00000ff6 SystemErr [9/3/18 12:40:29:206 IST] 00001f17 SystemErr [9/3/18 12:41:02:344 IST] 00000ffc SystemErr [9/17/18 10:35:07:028 IST] 0000007b SystemErr [9/17/18 10:35:07:028 IST] 0000007b SystemErr [9/17/18 10:35:07:029 IST] 0000007b SystemErr [9/17/18 15:09:47:415 IST] 0000007a SystemErr java org.apache.openjpa.lib.conf.ProductDeriv [9/17/18 15:09:47:416 IST] 0000007a SystemErr [9/17/18 15:09:47:463 IST] 0000007a SystemErr [9/17/18 15:09:48:038 IST] 0000007a SystemErr With the Partitioning, OLAP, Advanced [9/17/18 15:09:48:045 IST] 0000007a SystemErr [9/17/18 15:09:48:051 IST] 0000007a SystemErr [9/17/18 15:09:49:883 IST] 0000007b SystemErr java org.apache.openjpa.lib.conf.ProductDeriv R at org.apache.cxf.jaxrs.impl.ResponseBuilderImpl.status(ResponseBuilderImpl.java:78)
R at javax.ws.rs.core.Response.status(Response.java:613)
R at com.JavaAdapterResource.logout(JavaAdapterResource.java:1778)
R at sun.reflect.GeneratedMethodAccessor488.invoke(Unknown Source)
R at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
R at java.lang.reflect.Method.invoke(Method.java:508)
R java.lang.IllegalArgumentException: Illegal status value : 0
R java.lang.IllegalArgumentException: Illegal status value : 0
R log4j:WARN No appenders could be found for logger (org.apache.cxf.common.logging.LogUtils).
R log4j:WARN Please initialize the log4j system properly.
R log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
R Some product derivations are being skipped. For information about product derivation status, run:
ations
R com.ibm.ws.persistence.WsJpaProductDerivation:java.lang.ClassNotFoundException: com.ibm.ws.persistence.WsJpaProductDerivation
R 16 jpa-unit-rdbms WARN [server.startup : 0] openjpa.Runtime - Could not create the optional validation provider. Reason returned: "A default ValidatorFactory could not be created."
R 591 jpa-unit-rdbms INFO [server.startup : 0] openjpa.jdbc.JDBC - Using dictionary class "org.apache.openjpa.jdbc.sql.OracleDictionary" (Oracle Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
Analytics and Real Application Testing options ,Oracle JDBC driver 12.1.0.2.0).
R 598 jpa-unit-rdbms INFO [server.startup : 0] openjpa.jdbc.JDBC - Connected to Oracle version 12.12 using JDBC driver Oracle JDBC driver version 12.1.0.2.0.
R 604 jpa-unit-rdbms INFO [server.startup : 0] openjpa.Runtime - Starting OpenJPA 2.4.0
R Some product derivations are being skipped. For information about product derivation status, run:
ations

My Config file

input {
file{
path => "/app/install/database/SystemErr.log"
start_position => "beginning"
}
}

filter {
grok {
match =>
{
"message", "%{SYSLOG5424SD:time} %{NOTSPACE:id1} %{WORD:errortype}\s\s\s\s %{WORD:id2}\s%{WORD:check}"
}
}

if [check] == " " {
	grok
	{
	match =>
		{
		"message", "%{WORD:id3} %{URIHOST}(%{JAVACLASS}:%{NUMBER:errorclass}) "
		}
	}
}

if [check] == "java.*" {
   grok
	{
	match =>
		{
		"message", "%{URIHOST}:%{CISCO_REASON}:%{Number:statusvalue} "
		}
	}	
}

if [check] == "log4j:*" {
   grok
	{
	match =>
		{
		"message", "log4j:WARN %{CISCO_REASON} (%{URIHOST}). "
		}
	}	
}

}

output {
stdout {}
elasticsearch{
hosts => "x.x.x.x"
index => "system_error_log_x"
}

}

Can anybody help me, how to read the entire log file. Please let me know my mistakes in log file.
Thanks in advance.

Sripal · November 26, 2018, 6:51am

My observations in log file are:

[9/3/18 12:19:34:644 IST] 00000ff6 SystemErr R (till here every thing same)
[9/17/18 15:09:48:051 IST] 0000007a SystemErr R 604 jpa-unit-rdbms INFO [server.startup : 0] openjpa.Runtime - Starting OpenJPA 2.4.0 (if its a number, then the structure is same)
[9/3/18 12:19:34:645 IST] 00000ff6 SystemErr R at java.lang.reflect.Method.invoke(Method.java:508) (this line is different)
[9/17/18 10:35:07:028 IST] 0000007b SystemErr R log4j:WARN Please initialize the log4j system properly. (this line is different)

Please anybody help me how to read the entire log with out missing any information.

Sripal · November 26, 2018, 6:53am

Is there anyother way to achieve this?
easily ?

Eniqmatic · November 26, 2018, 8:52am

Some errors and observations about what you are getting would be handy.

What are you expecting to receive in Elastic and what did you actually recieve? Any errors in the logstash log?

Sripal · December 4, 2018, 9:42am

@Eniqmatic

Please review the error

[user@server bin]$ ./logstash -f /path/to/my/logfile/syserr_log.conf
Sending Logstash logs to /app/install/logstash-6.4.2/logs which is now configured via log4j2.properties
[2018-12-04T15:10:07,753][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-12-04T15:10:08,469][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.4.2"}
[2018-12-04T15:10:14,465][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-12-04T15:10:15,000][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://x.x.x.x:9200/]}}
[2018-12-04T15:10:15,016][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://x.x.x.x:9200/, :path=>"/"}
[2018-12-04T15:10:15,245][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://x.x.x.x:9200/"}
[2018-12-04T15:10:15,305][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-12-04T15:10:15,309][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2018-12-04T15:10:15,364][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//x.x.x.x"]}
[2018-12-04T15:10:15,375][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-12-04T15:10:15,400][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-12-04T15:10:15,689][ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"#<LogStash::FilterDelegator:0x7ceffd7d @metric_events_out=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: out value:0, @metric_events_in=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: in value:0, @metric_events_time=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: duration_in_millis value:0, @id="9c030f9b6ff707c026be39d193338171a60b2c9f46176b22ee2cc685c628100a", @klass=LogStash::Filters::Grok, @metric_events=#LogStash::Instrument::NamespacedMetric:0x5fa24f36, @filter=<LogStash::Filters::Grok match=>{"message"=>"%{URIHOST}:%{CISCO_REASON}:%{Number:statusvalue} "}, id=>"9c030f9b6ff707c026be39d193338171a60b2c9f46176b22ee2cc685c628100a", enable_metric=>true, periodic_flush=>false, patterns_files_glob=>"*", break_on_match=>true, named_captures_only=>true, keep_empty_captures=>false, tag_on_failure=>["_grokparsefailure"], timeout_millis=>30000, tag_on_timeout=>"_groktimeout">>", :error=>"pattern %{Number:statusvalue} not defined", :thread=>"#<Thread:0x132abcfb run>"}
[2018-12-04T15:10:16,119][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Grok::PatternError: pattern %{Number:statusvalue} not defined>, :backtrace=>["/app/install/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in block in compile'", "org/jruby/RubyKernel.java:1292:inloop'", "/app/install/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in compile'", "/app/install/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:281:inblock in register'", "org/jruby/RubyArray.java:1734:in each'", "/app/install/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:275:inblock in register'", "org/jruby/RubyHash.java:1343:in each'", "/app/install/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:270:inregister'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:242:in register_plugin'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:253:inblock in register_plugins'", "org/jruby/RubyArray.java:1734:in each'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:253:inregister_plugins'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:595:in maybe_setup_out_plugins'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:263:instart_workers'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:200:in run'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:160:inblock in start'"], :thread=>"#<Thread:0x132abcfb run>"}
[2018-12-04T15:10:16,135][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
[user@server bin]$

Eniqmatic · December 4, 2018, 9:44am

Here is your error, you have not defined this pattern!

Sripal · December 4, 2018, 9:55am

Sripal:

input {
file{
path => "/app/install/database/SystemErr.log"
start_position => "beginning"
}
}

filter {
grok {
match =>
{
"message", "%{SYSLOG5424SD:time} %{NOTSPACE:id1} %{WORD:errortype}\s\s\s\s %{WORD:id2}\s%{WORD:check}"
}
}
if [check] == " " {
	grok
	{
	match =>
		{
		"message", "%{WORD:id3} %{URIHOST}(%{JAVACLASS}:%{NUMBER:errorclass}) "
		}
	}
}

if [check] == "java.*" {
   grok
	{
	match =>
		{
		"message", "%{URIHOST}:%{CISCO_REASON}:%{Number:statusvalue} "
		}
	}	
}

if [check] == "log4j:*" {
   grok
	{
	match =>
		{
		"message", "log4j:WARN %{CISCO_REASON} (%{URIHOST}). "
		}
	}	
}
}

output {
stdout {}
elasticsearch{
hosts => "x.x.x.x"
index => "system_error_log_x"
}

}

@Eniqmatic

Thanks much i got the error. I changed %{Number:statusvalue} to %{NUMBER:statusvalue}.

Eniqmatic · December 4, 2018, 10:07am

Good to hear, please mark as solved if fixed!

Sripal · December 4, 2018, 10:39am

@Eniqmatic Ya fine. Its not fully fixed.

I want to split the message [1/17/18 17:32:30:066 IST] 0000016f SystemErr R java.lang.IllegalArgumentException: Illegal status value : 0

if the word check match with java then it should use another gork fileter to filter the patterns.
but its not happend plz check

filter {
grok {
match =>
{
"message", "%{SYSLOG5424SD:time} %{NOTSPACE:id1} %{WORD:errortype}\s\s\s\s %{WORD:id2}\s%{WORD:check}"
}
}

if [check] == "java.*" {
grok
{
match =>
{
"message", "%{URIHOST}:%{CISCO_REASON}:%{Number:statusvalue} "
}
}
}

Eniqmatic · December 4, 2018, 10:42am

Your field says "java" yet you have done "java.*" the DOT being significant. The dot will not match your field.

Sripal · December 4, 2018, 1:58pm

@Eniqmatic Thanks for the quick response.
Its fixed the problem.
Appreciate your efforts

system · January 1, 2019, 1:58pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.