LogLevel field in kibana

[shiv94]

Hello,

I want to get the log level field in kibana for all the logs

For example,

2019-01-16 13:54:55,833 INFO [org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean] (default task-29) Creating Service {http://www.GETWPWS.WBS2PRMI.com}GETWPWSService from WSDL: file:/var/acweb/properties/GtWp.wsdl
2019-01-16 13:54:55,929 ERROR [io.undertow.request] (default task-29) UT005023: Exception handling request to /agent/agentweb/tertiary_template.jsp: javax.servlet.ServletException: javax.servlet.ServletException: javax.servlet.ServletExc
eption: javax.servlet.jsp.JspException: No bean found under attribute key policyCorrespondenceDocumentList
at org.apache.struts.chain.ComposableRequestProcessor.process(ComposableRequestProcessor.java:286)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)

I need to get log level as INFO, ERROR, WARN, DEBUG ... In the above I have two entries one is INFO and other is ERROR. Is there any option to get these values for log level field?
I tried using grok filter but didn't work

grok {
match => { "message" => ["%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} (?(?:[a-zA-Z0-9]+.)*[-A-Za-z0-9$]+) %{GREEDYDATA:message}"]
}
mutate {
add_field => {"loglevel" => %{LOGLEVEL:level}}
}
}Preformatted text

[Badger]

As displayed, that is not a valid grok pattern. If you are posting a configuration please select it in the edit pane and click on </> in the toolbar above the pane.

What are you trying to achieve using the mutate filter?

[shiv94]

I am trying to pull the LOGLEVEL value. Can you suggest me better option?

[Badger]

match => { "message" => ["^%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel}"]

will get you a field on the event called loglevel. But I get a feeling you actually want more than that. You still need to fix your original post.

[shiv94]

Will loglevel will be displayed in the kibana? Like in the below image. Preformatted text

[Badger]

Well, it will add a field called loglevel to the event. If you ingest that into elasticsearch then if kibana queries elasticsearch the field will be there.

[shiv94]

Hi Badger ,
I tried this and got the below error

        filter {
         if "abcd" in [tags] {
         match => { "message" => ["^%{TIMESTAMP_ISO8601:timestamp1} %{LOGLEVEL:loglevel}"] }
         }
        }

[2019-01-16T14:57:37,361][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.4.1"}
[2019-01-16T14:57:37,623][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 68, column 8 (byte 1092) after filter {\n if "abcd" in [tags] {\n match ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:149:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:22:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:38:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:309:inblock in converge_state'"]}

timestamp1 and loglevel field should be displayed like in the below image

[Badger]

You are missing 'grok {' and the matching }

[shiv94]

Thanks and sorry for the wrong post

[shiv94]

Got it but I can see for only few events, even other contains the same format

[Badger]

OK, so what does the message field look like on one of those events that does not have loglevel?

[shiv94]

I did the changes only in one server not the entire cluster. Now, it works

Thanks a lot for you immediate response helped me alot