Error in structuring log file


(Sripal) #1

Hi all,

I'm new to ELK stack.
I'm trying to read the system error log, but i am unable to achieve. Here are my exploration.

SysErr Lof file sample:

[9/3/18 12:19:34:644 IST] 00000ff6 SystemErr R at org.apache.cxf.jaxrs.impl.ResponseBuilderImpl.status(ResponseBuilderImpl.java:78)
[9/3/18 12:19:34:645 IST] 00000ff6 SystemErr R at javax.ws.rs.core.Response.status(Response.java:613)
[9/3/18 12:19:34:645 IST] 00000ff6 SystemErr R at com.JavaAdapterResource.logout(JavaAdapterResource.java:1778)
[9/3/18 12:19:34:645 IST] 00000ff6 SystemErr R at sun.reflect.GeneratedMethodAccessor488.invoke(Unknown Source)
[9/3/18 12:19:34:645 IST] 00000ff6 SystemErr R at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
[9/3/18 12:19:34:645 IST] 00000ff6 SystemErr R at java.lang.reflect.Method.invoke(Method.java:508)
[9/3/18 12:40:29:206 IST] 00001f17 SystemErr R java.lang.IllegalArgumentException: Illegal status value : 0
[9/3/18 12:41:02:344 IST] 00000ffc SystemErr R java.lang.IllegalArgumentException: Illegal status value : 0
[9/17/18 10:35:07:028 IST] 0000007b SystemErr R log4j:WARN No appenders could be found for logger (org.apache.cxf.common.logging.LogUtils).
[9/17/18 10:35:07:028 IST] 0000007b SystemErr R log4j:WARN Please initialize the log4j system properly.
[9/17/18 10:35:07:029 IST] 0000007b SystemErr R log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[9/17/18 15:09:47:415 IST] 0000007a SystemErr R Some product derivations are being skipped. For information about product derivation status, run:
java org.apache.openjpa.lib.conf.ProductDerivations
[9/17/18 15:09:47:416 IST] 0000007a SystemErr R com.ibm.ws.persistence.WsJpaProductDerivation:java.lang.ClassNotFoundException: com.ibm.ws.persistence.WsJpaProductDerivation
[9/17/18 15:09:47:463 IST] 0000007a SystemErr R 16 jpa-unit-rdbms WARN [server.startup : 0] openjpa.Runtime - Could not create the optional validation provider. Reason returned: "A default ValidatorFactory could not be created."
[9/17/18 15:09:48:038 IST] 0000007a SystemErr R 591 jpa-unit-rdbms INFO [server.startup : 0] openjpa.jdbc.JDBC - Using dictionary class "org.apache.openjpa.jdbc.sql.OracleDictionary" (Oracle Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options ,Oracle JDBC driver 12.1.0.2.0).
[9/17/18 15:09:48:045 IST] 0000007a SystemErr R 598 jpa-unit-rdbms INFO [server.startup : 0] openjpa.jdbc.JDBC - Connected to Oracle version 12.12 using JDBC driver Oracle JDBC driver version 12.1.0.2.0.
[9/17/18 15:09:48:051 IST] 0000007a SystemErr R 604 jpa-unit-rdbms INFO [server.startup : 0] openjpa.Runtime - Starting OpenJPA 2.4.0
[9/17/18 15:09:49:883 IST] 0000007b SystemErr R Some product derivations are being skipped. For information about product derivation status, run:
java org.apache.openjpa.lib.conf.ProductDerivations

My Config file

input {
file{
path => "/app/install/database/SystemErr.log"
start_position => "beginning"
}
}

filter {
grok {
match =>
{
"message", "%{SYSLOG5424SD:time} %{NOTSPACE:id1} %{WORD:errortype}\s\s\s\s %{WORD:id2}\s%{WORD:check}"
}
}

if [check] == " " {
	grok
	{
	match =>
		{
		"message", "%{WORD:id3} %{URIHOST}(%{JAVACLASS}:%{NUMBER:errorclass}) "
		}
	}
}

if [check] == "java.*" {
   grok
	{
	match =>
		{
		"message", "%{URIHOST}:%{CISCO_REASON}:%{Number:statusvalue} "
		}
	}	
}

if [check] == "log4j:*" {
   grok
	{
	match =>
		{
		"message", "log4j:WARN %{CISCO_REASON} (%{URIHOST}). "
		}
	}	
}

}

output {
stdout {}
elasticsearch{
hosts => "x.x.x.x"
index => "system_error_log_x"
}

}

Can anybody help me, how to read the entire log file. Please let me know my mistakes in log file.
Thanks in advance.


(Sripal) #2

My observations in log file are:

  1. [9/3/18 12:19:34:644 IST] 00000ff6 SystemErr R (till here every thing same)
  2. [9/17/18 15:09:48:051 IST] 0000007a SystemErr R 604 jpa-unit-rdbms INFO [server.startup : 0] openjpa.Runtime - Starting OpenJPA 2.4.0 (if its a number, then the structure is same)
  3. [9/3/18 12:19:34:645 IST] 00000ff6 SystemErr R at java.lang.reflect.Method.invoke(Method.java:508) (this line is different)
  4. [9/17/18 10:35:07:028 IST] 0000007b SystemErr R log4j:WARN Please initialize the log4j system properly. (this line is different)

Please anybody help me how to read the entire log with out missing any information.


(Sripal) #3

Is there anyother way to achieve this?
easily ? :wink:


(Lewis Barclay) #4

Some errors and observations about what you are getting would be handy.

What are you expecting to receive in Elastic and what did you actually recieve? Any errors in the logstash log?


(Sripal) #5

@Eniqmatic

Please review the error

[user@server bin]$ ./logstash -f /path/to/my/logfile/syserr_log.conf
Sending Logstash logs to /app/install/logstash-6.4.2/logs which is now configured via log4j2.properties
[2018-12-04T15:10:07,753][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-12-04T15:10:08,469][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.4.2"}
[2018-12-04T15:10:14,465][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-12-04T15:10:15,000][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://x.x.x.x:9200/]}}
[2018-12-04T15:10:15,016][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://x.x.x.x:9200/, :path=>"/"}
[2018-12-04T15:10:15,245][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://x.x.x.x:9200/"}
[2018-12-04T15:10:15,305][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-12-04T15:10:15,309][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2018-12-04T15:10:15,364][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//x.x.x.x"]}
[2018-12-04T15:10:15,375][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-12-04T15:10:15,400][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-12-04T15:10:15,689][ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"#<LogStash::FilterDelegator:0x7ceffd7d @metric_events_out=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: out value:0, @metric_events_in=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: in value:0, @metric_events_time=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: duration_in_millis value:0, @id="9c030f9b6ff707c026be39d193338171a60b2c9f46176b22ee2cc685c628100a", @klass=LogStash::Filters::Grok, @metric_events=#LogStash::Instrument::NamespacedMetric:0x5fa24f36, @filter=<LogStash::Filters::Grok match=>{"message"=>"%{URIHOST}:%{CISCO_REASON}:%{Number:statusvalue} "}, id=>"9c030f9b6ff707c026be39d193338171a60b2c9f46176b22ee2cc685c628100a", enable_metric=>true, periodic_flush=>false, patterns_files_glob=>"*", break_on_match=>true, named_captures_only=>true, keep_empty_captures=>false, tag_on_failure=>["_grokparsefailure"], timeout_millis=>30000, tag_on_timeout=>"_groktimeout">>", :error=>"pattern %{Number:statusvalue} not defined", :thread=>"#<Thread:0x132abcfb run>"}
[2018-12-04T15:10:16,119][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Grok::PatternError: pattern %{Number:statusvalue} not defined>, :backtrace=>["/app/install/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in block in compile'", "org/jruby/RubyKernel.java:1292:inloop'", "/app/install/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in compile'", "/app/install/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:281:inblock in register'", "org/jruby/RubyArray.java:1734:in each'", "/app/install/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:275:inblock in register'", "org/jruby/RubyHash.java:1343:in each'", "/app/install/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:270:inregister'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:242:in register_plugin'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:253:inblock in register_plugins'", "org/jruby/RubyArray.java:1734:in each'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:253:inregister_plugins'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:595:in maybe_setup_out_plugins'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:263:instart_workers'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:200:in run'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:160:inblock in start'"], :thread=>"#<Thread:0x132abcfb run>"}
[2018-12-04T15:10:16,135][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
[user@server bin]$


(Lewis Barclay) #6

Here is your error, you have not defined this pattern!


(Sripal) #7

@Eniqmatic

Thanks much i got the error. I changed %{Number:statusvalue} to %{NUMBER:statusvalue}.


(Lewis Barclay) #8

Good to hear, please mark as solved if fixed!


(Sripal) #9

@Eniqmatic Ya fine. Its not fully fixed.

I want to split the message [1/17/18 17:32:30:066 IST] 0000016f SystemErr R java.lang.IllegalArgumentException: Illegal status value : 0

if the word check match with java then it should use another gork fileter to filter the patterns.
but its not happend plz check

filter {
grok {
match =>
{
"message", "%{SYSLOG5424SD:time} %{NOTSPACE:id1} %{WORD:errortype}\s\s\s\s %{WORD:id2}\s%{WORD:check}"
}
}

if [check] == "java.*" {
grok
{
match =>
{
"message", "%{URIHOST}:%{CISCO_REASON}:%{Number:statusvalue} "
}
}
}


(Lewis Barclay) #10

Your field says "java" yet you have done "java.*" the DOT being significant. The dot will not match your field.


(Sripal) #11

@Eniqmatic Thanks for the quick response.
Its fixed the problem.
Appreciate your efforts