Removing a part of every line on a multiple codec

jervdv · March 9, 2016, 8:01am

Hi

I'm combining a stack trace with a mutiline codec everything is going great my multiline codec combines evertyhing to one logstash event. But as you can see in the logstash event below i need to delete a part of the log every line is this possible with grok? ("INFO | jvm 1 | main | 2016/03/01 04:03:03.503 |" This is the part that needs to be removed.)

michalterbert · March 9, 2016, 8:06am

You can use grok parsing and mutate for remove fileds.
https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html

jervdv · March 9, 2016, 8:11am

I'm using a multiline codec who puts everything of my stacktrace in to one field.
So it is impossible to remove fields i think.

Is it possible to make an array of your fields with your multiline codec?
Like for example, is it posibble to make an array of log_level where then would be INFO for every line of my stacktrace?

michalterbert · March 9, 2016, 8:20am

So if you have all stacktrace in to one filed you can use GROK and parsed this filed to other fields - its simple.
You can test your groks patterns here:
https://grokdebug.herokuapp.com

Your next step will be droping all events with other value for log_level than INFO.

jervdv · March 9, 2016, 8:34am

But i also have fields without a stacktrace.

My grok works to get all the fields out of the log file.

But now i need to delete a part of that line if i have a stacktrace.

You suggest to do a new grok on the field of my stacktrace?
But there are also normal error messages in that don't have a stacktrace.

I don't see how that is possible?

Here is an example of a log event without a stacktrace.

INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | 04:03:03,407 [SitemapGeneratorCronjob::com.test.foo.types.job.SitemapGeneratorJob] ERROR [SitemapGenerator] Error while exporting the the facet '/ -> Trio (brand) -> Series 3401 (collection)'.

When i have a stacktrace i have this in my stacktrace field: [SitemapGeneratorCronjob::com.test.foo.types.job.SitemapGeneratorJob] ERROR [SitemapGenerator] Error while exporting the the facet '/ -> Trio (brand) -> Series 3401 (collection)'.
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | de.hybris.platform.servicelayer.exceptions.UnknownIdentifierException: No ProductCollectionInfo found with collectionValue [Series 3401] and brandValue [Trio]
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.foo.dao.impl.DefaultCollectionDao.getCollectionBySolrValue(DefaultCollectionDao.java:77)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.foo.services.impl.DefaultfooCollectionService.getCollectionBySolrValue(DefaultfooCollectionService.java:46)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.solr.services.impl.DefaultFilterPageUrlService.generateUrl(DefaultFilterPageUrlService.java:242)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.solr.services.impl.DefaultFilterPageUrlService.fabricateUrl(DefaultFilterPageUrlService.java:122)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.sitemap.services.SitemapUrlService$7.apply(SitemapUrlService.java:117)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.sitemap.services.SitemapUrlService$7.apply(SitemapUrlService.java:1)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.sitemap.converters.LanguageTranslator.getAllTranslations(LanguageTranslator.java:41)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.sitemap.converters.LanguageTranslator.getAllTranslations(LanguageTranslator.java:27)
INFO | jvm 1 | main | 2016/03/01 04:03:03.503 | at com.test.sitemap.services.SitemapUrlService.generateFacetUrls(SitemapUrlService.java:114)

While i actually want this:
[SitemapGeneratorCronjob::com.test.foo.types.job.SitemapGeneratorJob] ERROR [SitemapGenerator] Error while exporting the the facet '/ -> Trio (brand) -> Series 3401 (collection)'.
de.hybris.platform.servicelayer.exceptions.UnknownIdentifierException: No ProductCollectionInfo found with collectionValue [Series 3401] and brandValue [Trio]
at com.test.foo.dao.impl.DefaultCollectionDao.getCollectionBySolrValue(DefaultCollectionDao.java:77)
at com.test.foo.services.impl.DefaultfooCollectionService.getCollectionBySolrValue(DefaultfooCollectionService.java:46)
at com.test.solr.services.impl.DefaultFilterPageUrlService.generateUrl(DefaultFilterPageUrlService.java:242)
at com.test.solr.services.impl.DefaultFilterPageUrlService.fabricateUrl(DefaultFilterPageUrlService.java:122)
at com.test.sitemap.services.SitemapUrlService$7.apply(SitemapUrlService.java:117)
at com.test.sitemap.services.SitemapUrlService$7.apply(SitemapUrlService.java:1)
at com.test.sitemap.converters.LanguageTranslator.getAllTranslations(LanguageTranslator.java:41)
at com.test.sitemap.converters.LanguageTranslator.getAllTranslations(LanguageTranslator.java:27)
at com.test.sitemap.services.SitemapUrlService.generateFacetUrls(SitemapUrlService.java:114)

magnusbaeck · March 9, 2016, 8:45pm

I'd look into the mutate filter's gsub option.

jervdv · March 15, 2016, 9:52am

Indeed with gsub i was able to fix it. Thanks for the help!