Split string into array

Elastic Stack Logstash
1

Hi.

What I'm trying to do is accessing a nested string in "doc" called "message" (doc.message?), and this string contains a couple of things, including a case specification. That specification is what I am trying to extract and create another object variable under "doc" with (as in "doc.case"). It is not always the same text inside the string of course, but neither is the case always in the same "section" of the string. I have tried to use split{}, both within and outside of mutate{}, and neither worked. The logfile itself is in xml formatting.

Here is the code:
input { beats { port => '5044' } } filter { xml { source => 'message' target => 'doc' } split { field => 'message' } mutate { remove_field => 'message' } } output { elasticsearch { hosts => 'http://localhost:9200' index => 'logfiles' document_type => 'commandusage' } }

And here is the logfile itself:
"doc": { "AppDomain": "DefaultDomain [1]", "Message": """3D OpenND Mode: VRT Images in stack: 211 (max limit: 2000) Viewport size: (896, 1078) Date: 20171022 14:13 Session GUID: a3bda1f1-d1f7-4fde-a3f0-7e8a926f9bfc Server: Local Case: 5b44490f-202a-439f-a7f1-57f2be400ede""", "User": "User123", "Categories": "Cat1, Cat2", "Level": "Info", "Thread": "[1]", "Method": "Method.123", "Host": "Host1", "Time": "2018-03-04T10:10:59.3353115+00:00" },

(Sorry about the looks, I can't seem to get the editing right)

2

You're probably looking for split { field => '[doc][message]' }, see https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references.

3

Thank you for the quick answer.

Tried it, but it complained about the field being a "NilClass" type, which isn't splittable.

4

That indicates that the field doesn't exist. Dump the raw event with a stdout { codec => rubydebug } output.

5

The field shows up in the dump, and a "_split_type_failure" tag shows up on the files, along with the previous error message.

6

The field shows up in the dump

Well, please show it to us.

7

Of course.

[2018-03-09T09:06:24,055][WARN ][logstash.filters.split   ] Only String and Array types are splittable. field:[_source][doc][Message] is of type = NilClass

{
"tags" => [
[0] "beats_input_codec_plain_applied",
[1] "_split_type_failure"
],
"beat" => {
"version" => "6.2.2",
"hostname" => "LocalPC",
"name" => "LocalPC"
},
"prospector" => {
"type" => "log"
},
"host" => "LocalPC",
"offset" => 386,
"@timestamp" => 2018-03-09T08:06:22.293Z,
"@version" => "1",
"source" => "C:\log.slf",
"doc" => {
"Level" => "Info",
"Host" => "Host1",
"Method" => "Company.Utilities.DefaultLogs.CommandUsageLog::Info",
"User" => "User",
"Categories" => "client.performance.commands, common.assembly.utilities",
"AppDomain" => "DefaultDomain [1]",
"Message" => "c6411347-96cf-4330-b3e5-310d523adbba\tSave Image\t3D\tTime (ms):\t20\tCase:\t6e5e2d3a-ce1c-453c-93ff-4e9ce67936fe",
"Thread" => "[1]",
"Time" => "2018-01-16T19:00:24.6785488+00:00"
}
}

8

Field names are case sensitive so use [doc][Message]. I missed the uppercase M earlier, sorry.

9

No worries. Despite trying that earlier, this time it worked. It doesn't give any error messages, but neither does it create the [doc][Case] I want either, of course. Any thoughts on that?

10

Splitting doesn't do that for you. I suggest you use a grok or dissect filter to process the string and extract the fields you want.

11

Right, thanks for the help!

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.