# Logstash Pipeline issue (I'm an idiot or something) **URL:** \ **Category:** Logstash\ **Created:** [December 17, 2020, 11:50pm UTC](https://discuss.elastic.co/t/logstash-pipeline-issue-im-an-idiot-or-something/259042 "2020-12-17T23:50:12Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![SigmazGFX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sigmazgfx/32/53534_2.png) [@SigmazGFX](https://discuss.elastic.co/u/SigmazGFX)\ **Post date:** [December 17, 2020, 11:50pm UTC](https://discuss.elastic.co/t/logstash-pipeline-issue-im-an-idiot-or-something/259042/1 "2020-12-17T23:50:12Z")
Hey gang. I am working with our dev team and they are dumping custom logging into the windows events Using winlogbeat to ship the log entries into logstash i am trying to set up a central pipeline to handle the different api logs. I am having trouble splitting up the data and it's all ending up in param1. The windows log data is coming in like this: ``` Identifier: c27948fe-af6e-42e2-9db6-338f1d48f9e0 IdentifierType: Test LogEvent ApplicationName: LoggingNuget-Tester Severity: Debug LogType: Name: Error talking to database Description: Something went wrong in the operation. FullStackTrace: Unable to find the specified file. AuditCreatedDate: 12/17/2020 4:55:56 PM ``` Using the beats input and regular winlogs flow right through no problem to elasticsearch My filter looks like this: ```auto if [winlog][channel] == "LoggingAPI" { if [message] =~ "Identifier: " { grok { match => { "message" => "Identifier: %{DATA:log.identifier}" } } } if [message] =~ "IdentifierType: " { grok { match => { "message" => "IdentifierType: %{DATA:log.identifierType}" } } } if [message] =~ "ApplicationName: " { grok { match => { "message" => "ApplicationName: %{DATA:log.applicationName}" } } } if [message] =~ "Severity: " { grok { match => { "message" => "Severity: %{DATA:log.severity}" } } } if [message] =~ "LogType: " { grok { match => { "message" => "LogType: %{DATA:log.logType}" } } } if [message] =~ "Name: " { grok { match => { "message" => "Name: %{DATA:log.name}" } } } if [message] =~ "Description: " { grok { match => { "message" => "Description: %{DATA:log.description}" } } } if [message] =~ "FullStackTrace: " { grok { match => { "message" => "FullStackTrace: %{GREEDYDATA:log.fullStackTrace}" } } } mutate{ add_tag => ["logging-api-via-logstash"]} } ``` The only line that makes it into it's own field is the last one "FullStackTrace" (of course that brings along the darn AuditCreatedDate If I try to use GREEDYDATA on the other lines it pulls the next line or more of data. I don't think I understand on how to get the value but stop at the new line. These logging events should never take up more than one line .. except of course the fullstacktrace. What am I doing wrong here? Thanks ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [December 18, 2020, 12:11am UTC](https://discuss.elastic.co/t/logstash-pipeline-issue-im-an-idiot-or-something/259042/2 "2020-12-18T00:11:36Z")
DATA can match a lot or a little, or even nothing at all. Try enabling the grok option `keep_empty_captures => true`. If you then find you get the fields that you expect but they are empty strings then try changing the patterns to things like `"message" => "Identifier: %{DATA:log.identifier}$"` to force DATA to consume the rest of the line. Then decide if you want to retain keep\_empty\_captures (i.e., do you want a LogType field for your example data). If that is your issue then it definitely does not indicate you are an idiot. It took me years to understand the greedy/not-greedy distinction for a Ruby regexp. As far as I know, the other regexp languages I have used for decades (ed, sh, csh, awk, and others) are basically greedy by default. GREEDYDATA will match as much as possible. DATA will match as little as possible, and if DATA is at the end of the pattern it can return a positive match for zero characters. You need a way to force DATA to consume the rest of the line. The $ I used above does that. Another way of consuming the line is a custom pattern that matches zero or more not-newline `"message" => "Identifier: (?[^\n]*)"`. Note that log.identifier will be a field with a period in its name. If your intention was to create an object like ``` "log": { "identifier": "...", "name": "...", ... } ``` then you would need to use [log][identifier] etc. ---
**Author:** ![SigmazGFX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sigmazgfx/32/53534_2.png) [@SigmazGFX](https://discuss.elastic.co/u/SigmazGFX)\ **Post date:** [December 18, 2020, 12:16am UTC](https://discuss.elastic.co/t/logstash-pipeline-issue-im-an-idiot-or-something/259042/3 "2020-12-18T00:16:05Z")
Thanks Badger. I'll try this out now ---
**Author:** ![SigmazGFX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sigmazgfx/32/53534_2.png) [@SigmazGFX](https://discuss.elastic.co/u/SigmazGFX)\ **Post date:** [December 18, 2020, 12:29am UTC](https://discuss.elastic.co/t/logstash-pipeline-issue-im-an-idiot-or-something/259042/4 "2020-12-18T00:29:17Z")
Ok great! I was able to get the results you detailed. and exactly as you detailed.. the first result were the fields albeit empty.. once I added the $'s they were populated with the expected data. I'll still need to do a bit of tweaking but I thank you very VERY much for steering me the right way.. Unfortunately I have to wear many hats and my grok hat is waaaay too small and I don't wear it all that often. I'll be back for more soon I'm sure. ---
**Author:** ![SigmazGFX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sigmazgfx/32/53534_2.png) [@SigmazGFX](https://discuss.elastic.co/u/SigmazGFX)\ **Post date:** [December 18, 2020, 1:15am UTC](https://discuss.elastic.co/t/logstash-pipeline-issue-im-an-idiot-or-something/259042/5 "2020-12-18T01:15:15Z")
Looking good, Thanks again! ```auto log.application_name LoggingNuget-Tester log.audit_created_date 12/17/2020 8:03:01 PM log.description Help, I'm trapped in this testing program! log.full_stack_trace Unable to find the specified file. log.identifier 8e8ba83f-3664-4e5b-ad74-433a0a81fffd log.identifier_type Test LogEvent log.level information log.name LoggingNuget-Tester log.severity Trace message Identifier: 8e8ba83f-3664-4e5b-ad74-433a0a81fffd IdentifierType: Test LogEvent ApplicationName: LoggingNuget-Tester Severity: Trace LogType: Name: Standard operation Description: Help, I'm trapped in this testing program! FullStackTrace: Unable to find the specified file. AuditCreatedDate: 12/17/2020 8:03:01 PM ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 15, 2021, 1:15am UTC](https://discuss.elastic.co/t/logstash-pipeline-issue-im-an-idiot-or-something/259042/6 "2021-01-15T01:15:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.