Hi Magnus,
I have recently started using ELK, I have been assigned to do ELK integration in my project. I have installed 7.6.1 of ELK and filebeat. I need your assistance is configuring filter for logstash to achieve task mentioned below
What I have done until now ---
-
I have created a sample microservice which will create some dummy logs and exception when specific REST endpoint url is hit
-
microservices is configured with logback xml where they will dump all the logs in specific logback folder
3 filebeat will pickup all the logs and send it to logstash
- logstash will forward the same logs to elastic search and can be viewed in Kibana
What I want to achieve ---
-
I want to visualize top- N exception names coming from the logs with respect to count ( I have attached the screenshot of kibana visualization for the same)
-
Top -N URIPATHS to be visualized in kibana wrt count
-
I wanted to see the whole stack trace as a single unit
please help....
my sample log file is as below --
2020-04-07 20:20:33.679 ERROR 41436 --- [ restartedMain] o.s.boot.SpringApplication : Application run failed
java.lang.IllegalStateException: Error processing condition on org.springframework.boot.autoconfigure.context.PropertyPlaceholderAutoConfiguration.propertySourcesPlaceholderConfigurer
at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:60) ~[spring-boot-autoconfigure-2.2.5.RELEASE.jar:2.2.5.RELEASE]
at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:108) ~[spring-context-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForBeanMethod(ConfigurationClassBeanDefinitionReader.java:184) ~[spring-context-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForConfigurationClass(ConfigurationClassBeanDefinitionReader.java:144) ~[spring-context-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitions(ConfigurationClassBeanDefinitionReader.java:120) ~[spring-context-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:331) ~[spring-context-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:236) ~[spring-context-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:275) ~[spring-context-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:95) ~[spring-context-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:706) ~[spring-context-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:532) ~[spring-context-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141) ~[spring-boot-2.2.5.RELEASE.jar:2.2.5.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747) [spring-boot-2.2.5.RELEASE.jar:2.2.5.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) [spring-boot-2.2.5.RELEASE.jar:2.2.5.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) [spring-boot-2.2.5.RELEASE.jar:2.2.5.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) [spring-boot-2.2.5.RELEASE.jar:2.2.5.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) [spring-boot-2.2.5.RELEASE.jar:2.2.5.RELEASE]
at com.ibm.product.ProductsApplication.main(ProductsApplication.java:11) [classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_241]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_241]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_241]
at java.lang.reflect.Method.invoke(Unknown Source) ~[na:1.8.0_241]
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) [spring-boot-devtools-2.2.5.RELEASE.jar:2.2.5.RELEASE]
Caused by: java.lang.IllegalStateException: Failed to introspect Class [com.ibm.product.service.KafkaConfig] from ClassLoader [org.springframework.boot.devtools.restart.classloader.RestartClassLoader@6c43b25f]
at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:481) ~[spring-core-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:358) ~[spring-core-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.util.ReflectionUtils.getUniqueDeclaredMethods(ReflectionUtils.java:414) ~[spring-core-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.lambda$getTypeForFactoryMethod$2(AbstractAutowireCapableBeanFactory.java:743) ~[spring-beans-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(Unknown Source) ~[na:1.8.0_241]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getTypeForFactoryMethod(AbstractAutowireCapableBeanFactory.java:742) ~[spring-beans-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.determineTargetType(AbstractAutowireCapableBeanFactory.java:681) ~[spring-beans-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.predictBeanType(AbstractAutowireCapableBeanFactory.java:649) ~[spring-beans-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.isFactoryBean(AbstractBeanFactory.java:1605) ~[spring-beans-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:520) ~[spring-beans-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:491) ~[spring-beans-5.2.4.RELEASE.jar:5.2.4.RELEASE]
at org.springframework.boot.autoconfigure.condition.OnBeanCondition.collectBeanNamesForType(OnBeanCondition.java:230) ~[spring-boot-autoconfigure-2.2.5.RELEASE.jar:2.2.5.RELEASE]
at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getBeanNamesForType(OnBeanCondition.java:223) ~[spring-boot-autoconfigure-2.2.5.RELEASE.jar:2.2.5.RELEASE]
at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getBeanNamesForType(OnBeanCondition.java:213) ~[spring-boot-autoconfigure-2.2.5.RELEASE.jar:2.2.5.RELEASE]
at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getMatchingBeans(OnBeanCondition.java:167) ~[spring-boot-autoconfigure-2.2.5.RELEASE.jar:2.2.5.RELEASE]
at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getMatchOutcome(OnBeanCondition.java:142) ~[spring-boot-autoconfigure-2.2.5.RELEASE.jar:2.2.5.RELEASE]
at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:47) ~[spring-boot-autoconfigure-2.2.5.RELEASE.jar:2.2.5.RELEASE]
... 22 common frames omitted
Caused by: java.lang.NoClassDefFoundError: ProducerFactory
at java.lang.Class.getDeclaredMethods0(Native Method) ~[na:1.8.0_241]
at java.lang.Class.privateGetDeclaredMethods(Unknown Source) ~[na:1.8.0_241]
at java.lang.Class.getDeclaredMethods(Unknown Source) ~[na:1.8.0_241]
at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:463) ~[spring-core-5.2.4.RELEASE.jar:5.2.4.RELEASE]
... 38 common frames omitted
Caused by: java.lang.ClassNotFoundException: ProducerFactory
at java.net.URLClassLoader.findClass(Unknown Source) ~[na:1.8.0_241]
at java.lang.ClassLoader.loadClass(Unknown Source) ~[na:1.8.0_241]
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) ~[na:1.8.0_241]
at java.lang.ClassLoader.loadClass(Unknown Source) ~[na:1.8.0_241]
at org.springframework.boot.devtools.restart.classloader.RestartClassLoader.loadClass(RestartClassLoader.java:144) ~[spring-boot-devtools-2.2.5.RELEASE.jar:2.2.5.RELEASE]
at java.lang.ClassLoader.loadClass(Unknown Source) ~[na:1.8.0_241]
... 42 common frames omitted
Below is my sample config file
input {
beats{
port => 5044
}
}
filter {
if [message] =~ "\tat" {
grok {
match => ["message", "^(\tat)"]
add_tag => ["stacktrace"]
}
}
if [message] == "ERROR"{
grok{
match => {"message" => "%TIMESTAMP_ISO8601:timestamp} % {SPACE} % {LOGLEVEL:loglevel}
%{SPACE} % {INT:pid} % {SPACE}---%{SPACE} [%{DATA:threadname}] % {SPACE} % {JAVACLASS:className}
% {SPACE} : %{SPACE} Exception - %{JAVACLASS:exceptionName} : %{SPACE} % {DATA:exceptionMesaage}
\n (?m) % {GREEDYDATA:stacktrace}"
}
}
}
output {
elasticsearch{
hosts => "http://localhost:9200"
index => "filebeat"
}
}
I am getting below error for above config file --
C:\logstash-7.6.1\logstash-7.6.1\config>
C:\logstash-7.6.1\logstash-7.6.1\config>
C:\logstash-7.6.1\logstash-7.6.1\config>logstash -f logstashtest.conf
Sending Logstash logs to C:/logstash-7.6.1/logstash-7.6.1/logs which is now configured via log4j2.properties
[2020-04-07T20:27:16,201][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-04-07T20:27:16,333][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.6.1"}
[2020-04-07T20:27:18,118][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [A-Za-z0-9_-], [ \t\r\n], "#", "=>" at line 31, column 18 (byte 564) after filter {\n\nif [message] =~ "\tat" {\n grok {\n match => ["message", "^(\tat)"]\n add_tag => ["stacktrace"]\n }\n }\nif [message] == "ERROR"{\n\tgrok{\n\t\tmatch => {"message" => "%TIMESTAMP_ISO8601:timestamp} % {SPACE} % {LOGLEVEL:loglevel}\n%{SPACE} % {INT:pid} % {SPACE}---%{SPACE} \[%{DATA:threadname}] % {SPACE} % {JAVACLASS:className}\n% {SPACE} : %{SPACE} Exception - %{JAVACLASS:exceptionName} : %{SPACE} % {DATA:exceptionMesaage}\n\n (?m) % {GREEDYDATA:stacktrace}"\n\t}\n}\t\n \n}\n \n\n\n\noutput {\n elasticsearch", :backtrace=>["C:/logstash-7.6.1/logstash-7.6.1/logstash-core/lib/logstash/compiler.rb:47:in compile_imperative'", "C:/logstash-7.6.1/logstash-7.6.1/logstash-core/lib/logstash/compiler.rb:55:in compile_graph'", "C:/logstash-7.6.1/logstash-7.6.1/logstash-core/lib/logstash/compiler.rb:17:in block in compile_sources'", "org/jruby/RubyArray.java:2580:in map'", "C:/logstash-7.6.1/logstash-7.6.1/logstash-core/lib/logstash/compiler.rb:14:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:161:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "C:/logstash-7.6.1/logstash-7.6.1/logstash-core/lib/logstash/java_pipeline.rb:27:in initialize'", "C:/logstash-7.6.1/logstash-7.6.1/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "C:/logstash-7.6.1/logstash-7.6.1/logstash-core/lib/logstash/agent.rb:326:in block in converge_state'"]}
[2020-04-07T20:27:18,590][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-04-07T20:27:23,312][INFO ][logstash.runner ] Logstash shut down.