Removing a part of every line on a multiple codec

Elastic Stack Logstash
#1

Hi

I'm combining a stack trace with a mutiline codec everything is going great my multiline codec combines evertyhing to one logstash event. But as you can see in the logstash event below i need to delete a part of the log every line is this possible with grok? ("INFO | jvm 1 | main | 2016/03/01 04:03:03.503 |" This is the part that needs to be removed.)

INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | 04:03:03,407 [SitemapGeneratorCronjob::com.test.foo.types.job.SitemapGeneratorJob] ERROR [SitemapGenerator] Error while exporting the the facet '/ -> Trio (brand) -> Series 3401 (collection)'.
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | de.hybris.platform.servicelayer.exceptions.UnknownIdentifierException: No ProductCollectionInfo found with collectionValue [Series 3401] and brandValue [Trio]
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.foo.dao.impl.DefaultCollectionDao.getCollectionBySolrValue(DefaultCollectionDao.java:77)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.foo.services.impl.DefaultfooCollectionService.getCollectionBySolrValue(DefaultfooCollectionService.java:46)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.solr.services.impl.DefaultFilterPageUrlService.generateUrl(DefaultFilterPageUrlService.java:242)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.solr.services.impl.DefaultFilterPageUrlService.fabricateUrl(DefaultFilterPageUrlService.java:122)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.sitemap.services.SitemapUrlService$7.apply(SitemapUrlService.java:117)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.sitemap.services.SitemapUrlService$7.apply(SitemapUrlService.java:1)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.sitemap.converters.LanguageTranslator.getAllTranslations(LanguageTranslator.java:41)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.sitemap.converters.LanguageTranslator.getAllTranslations(LanguageTranslator.java:27)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.sitemap.services.SitemapUrlService.generateFacetUrls(SitemapUrlService.java:114)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.sitemap.services.SitemapElementService.generateFacetElement(SitemapElementService.java:45)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.sitemap.SitemapGenerator.exportFacets(SitemapGenerator.java:106)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.sitemap.SitemapGenerator.generateSitemaps(SitemapGenerator.java:73)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.foo.types.job.SitemapGeneratorJob.performCronJob(SitemapGeneratorJob.java:44)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at de.hybris.platform.cronjob.jalo.Job.execute(Job.java:1262)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at de.hybris.platform.cronjob.jalo.Job.performImpl(Job.java:793)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at de.hybris.platform.cronjob.jalo.Job.access$1(Job.java:752)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at de.hybris.platform.cronjob.jalo.Job$JobRunable.run(Job.java:657)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at de.hybris.platform.util.threadpool.PoolableThread.run(PoolableThread.java:131)

#2

You can use grok parsing and mutate for remove fileds.
https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html

#3

I'm using a multiline codec who puts everything of my stacktrace in to one field.
So it is impossible to remove fields i think.

Is it possible to make an array of your fields with your multiline codec?
Like for example, is it posibble to make an array of log_level where then would be INFO for every line of my stacktrace?

#4

So if you have all stacktrace in to one filed you can use GROK and parsed this filed to other fields - its simple.
You can test your groks patterns here:
https://grokdebug.herokuapp.com

Your next step will be droping all events with other value for log_level than INFO.

#5

But i also have fields without a stacktrace.

My grok works to get all the fields out of the log file.

But now i need to delete a part of that line if i have a stacktrace.

You suggest to do a new grok on the field of my stacktrace?
But there are also normal error messages in that don't have a stacktrace.

I don't see how that is possible?

Here is an example of a log event without a stacktrace.

INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | 04:03:03,407 [SitemapGeneratorCronjob::com.test.foo.types.job.SitemapGeneratorJob] ERROR [SitemapGenerator] Error while exporting the the facet '/ -> Trio (brand) -> Series 3401 (collection)'.

When i have a stacktrace i have this in my stacktrace field: [SitemapGeneratorCronjob::com.test.foo.types.job.SitemapGeneratorJob] ERROR [SitemapGenerator] Error while exporting the the facet '/ -> Trio (brand) -> Series 3401 (collection)'.
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | de.hybris.platform.servicelayer.exceptions.UnknownIdentifierException: No ProductCollectionInfo found with collectionValue [Series 3401] and brandValue [Trio]
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.foo.dao.impl.DefaultCollectionDao.getCollectionBySolrValue(DefaultCollectionDao.java:77)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.foo.services.impl.DefaultfooCollectionService.getCollectionBySolrValue(DefaultfooCollectionService.java:46)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.solr.services.impl.DefaultFilterPageUrlService.generateUrl(DefaultFilterPageUrlService.java:242)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.solr.services.impl.DefaultFilterPageUrlService.fabricateUrl(DefaultFilterPageUrlService.java:122)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.sitemap.services.SitemapUrlService$7.apply(SitemapUrlService.java:117)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.sitemap.services.SitemapUrlService$7.apply(SitemapUrlService.java:1)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.sitemap.converters.LanguageTranslator.getAllTranslations(LanguageTranslator.java:41)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.sitemap.converters.LanguageTranslator.getAllTranslations(LanguageTranslator.java:27)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.sitemap.services.SitemapUrlService.generateFacetUrls(SitemapUrlService.java:114)

While i actually want this:
[SitemapGeneratorCronjob::com.test.foo.types.job.SitemapGeneratorJob] ERROR [SitemapGenerator] Error while exporting the the facet '/ -> Trio (brand) -> Series 3401 (collection)'.
de.hybris.platform.servicelayer.exceptions.UnknownIdentifierException: No ProductCollectionInfo found with collectionValue [Series 3401] and brandValue [Trio]
at com.test.foo.dao.impl.DefaultCollectionDao.getCollectionBySolrValue(DefaultCollectionDao.java:77)
at com.test.foo.services.impl.DefaultfooCollectionService.getCollectionBySolrValue(DefaultfooCollectionService.java:46)
at com.test.solr.services.impl.DefaultFilterPageUrlService.generateUrl(DefaultFilterPageUrlService.java:242)
at com.test.solr.services.impl.DefaultFilterPageUrlService.fabricateUrl(DefaultFilterPageUrlService.java:122)
at com.test.sitemap.services.SitemapUrlService$7.apply(SitemapUrlService.java:117)
at com.test.sitemap.services.SitemapUrlService$7.apply(SitemapUrlService.java:1)
at com.test.sitemap.converters.LanguageTranslator.getAllTranslations(LanguageTranslator.java:41)
at com.test.sitemap.converters.LanguageTranslator.getAllTranslations(LanguageTranslator.java:27)
at com.test.sitemap.services.SitemapUrlService.generateFacetUrls(SitemapUrlService.java:114)

#6

I'd look into the mutate filter's gsub option.

1 Like
#7

Indeed with gsub i was able to fix it. Thanks for the help!

#8