# Drop log messages that don't contain absolute path to application data **URL:** \ **Category:** Logstash\ **Created:** [November 14, 2016, 5:07am UTC](https://discuss.elastic.co/t/drop-log-messages-that-dont-contain-absolute-path-to-application-data/65930 "2016-11-14T05:07:46Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![triceras](https://avatars.discourse-cdn.com/v4/letter/t/7ea924/32.png) [@triceras](https://discuss.elastic.co/u/triceras)\ **Post date:** [November 14, 2016, 5:07am UTC](https://discuss.elastic.co/t/drop-log-messages-that-dont-contain-absolute-path-to-application-data/65930/1 "2016-11-14T05:07:46Z")
Hi Forum, The application I am working with outputs several log messages as below. The message logged by the application displays the root path until the absolute path where the application resides in reverse order. I would like to keep only the message on the top as it contains the absolute path to the application data and drop the other messages below as they are not relevant. How can I achieve such thing using logstash ? I am new to ELK and any help would be really appreciated. > ``` > [02:26:59 9.5.2016] submitted:/testsuite/ACCESS_prod/sge/rtds4/complete/alter > [02:26:59 9.5.2016] submitted:/testsuite/ACCESS_prod/sge/rtds4/complete > [02:26:59 9.5.2016] submitted:/testsuite/ACCESS_prod/sge/rtds4 > [02:26:59 9.5.2016] submitted:/testsuite/ACCESS_prod/sge > [02:26:59 9.5.2016] submitted:/testsuite/ACCESS_prod > [02:26:59 9.5.2016] submitted:/testsuite > [02:26:59 9.5.2016] submitted:/ > > ``` Thank you. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 14, 2016, 6:49am UTC](https://discuss.elastic.co/t/drop-log-messages-that-dont-contain-absolute-path-to-application-data/65930/2 "2016-11-14T06:49:53Z")
Is there anything unique about /testsuite/ACCESS\_prod/sge/rtds4/complete/alter compared to the other paths? Is it a file rather than a directory? Does it contain any particular files that aren't found elsewhere? ---
**Author:** ![triceras](https://avatars.discourse-cdn.com/v4/letter/t/7ea924/32.png) [@triceras](https://discuss.elastic.co/u/triceras)\ **Post date:** [November 14, 2016, 11:20am UTC](https://discuss.elastic.co/t/drop-log-messages-that-dont-contain-absolute-path-to-application-data/65930/3 "2016-11-14T11:20:08Z")
Hi Magnus, This is the application absolute path. It's a directory and contains a different variety of ascii files that are genareted by the application. The other messages containing the directories above the app path can de discarted and I would like to know how can I do that using a regexp in logstash. Thank you. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 14, 2016, 11:24am UTC](https://discuss.elastic.co/t/drop-log-messages-that-dont-contain-absolute-path-to-application-data/65930/4 "2016-11-14T11:24:29Z")
You could write a ruby filter to do something like this: - If the message begins with "submitted:/", - extract the path, - check if the files contains a particular set of files, - if not, delete the event. Most of this stuff could be done with standard filters, but checking for the existence of files needs to be done with a ruby filter. ---
**Author:** ![triceras](https://avatars.discourse-cdn.com/v4/letter/t/7ea924/32.png) [@triceras](https://discuss.elastic.co/u/triceras)\ **Post date:** [November 16, 2016, 3:45am UTC](https://discuss.elastic.co/t/drop-log-messages-that-dont-contain-absolute-path-to-application-data/65930/5 "2016-11-16T03:45:37Z")
This is not quite the way I want to achieve this. To use a ruby filter to analyse every single line in a ~ 20M log file will hamper the server performance significantly. Is there any way to use the multiline filter plugin to put all the lines into a single array then chop all the other indexes in the array except the first one ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 16, 2016, 7:53am UTC](https://discuss.elastic.co/t/drop-log-messages-that-dont-contain-absolute-path-to-application-data/65930/6 "2016-11-16T07:53:45Z")
How many 20 MB log files do you need to process per second? Doing it with a multiline codec might work. If the current line matches (if we ignore the timestamp prefix for the purpose of this example) `submitted:/[a-z]`, join with the next line? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 14, 2016, 7:54am UTC](https://discuss.elastic.co/t/drop-log-messages-that-dont-contain-absolute-path-to-application-data/65930/7 "2016-12-14T07:54:11Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.