Split string into array

(Lucas Horn) #1


What I'm trying to do is accessing a nested string in "doc" called "message" (doc.message?), and this string contains a couple of things, including a case specification. That specification is what I am trying to extract and create another object variable under "doc" with (as in "doc.case"). It is not always the same text inside the string of course, but neither is the case always in the same "section" of the string. I have tried to use split{}, both within and outside of mutate{}, and neither worked. The logfile itself is in xml formatting.

Here is the code:
input { beats { port => '5044' } } filter { xml { source => 'message' target => 'doc' } split { field => 'message' } mutate { remove_field => 'message' } } output { elasticsearch { hosts => 'http://localhost:9200' index => 'logfiles' document_type => 'commandusage' } }

And here is the logfile itself:
"doc": { "AppDomain": "DefaultDomain [1]", "Message": """3D OpenND Mode: VRT Images in stack: 211 (max limit: 2000) Viewport size: (896, 1078) Date: 20171022 14:13 Session GUID: a3bda1f1-d1f7-4fde-a3f0-7e8a926f9bfc Server: Local Case: 5b44490f-202a-439f-a7f1-57f2be400ede""", "User": "User123", "Categories": "Cat1, Cat2", "Level": "Info", "Thread": "[1]", "Method": "Method.123", "Host": "Host1", "Time": "2018-03-04T10:10:59.3353115+00:00" },

(Sorry about the looks, I can't seem to get the editing right)

(Magnus Bäck) #2

You're probably looking for split { field => '[doc][message]' }, see https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references.

(Lucas Horn) #3

Thank you for the quick answer.

Tried it, but it complained about the field being a "NilClass" type, which isn't splittable.

(Magnus Bäck) #4

That indicates that the field doesn't exist. Dump the raw event with a stdout { codec => rubydebug } output.

(Lucas Horn) #5

The field shows up in the dump, and a "_split_type_failure" tag shows up on the files, along with the previous error message.

(Magnus Bäck) #6

The field shows up in the dump

Well, please show it to us.

(Lucas Horn) #7

Of course.

[2018-03-09T09:06:24,055][WARN ][logstash.filters.split   ] Only String and Array types are splittable. field:[_source][doc][Message] is of type = NilClass

"tags" => [
[0] "beats_input_codec_plain_applied",
[1] "_split_type_failure"
"beat" => {
"version" => "6.2.2",
"hostname" => "LocalPC",
"name" => "LocalPC"
"prospector" => {
"type" => "log"
"host" => "LocalPC",
"offset" => 386,
"@timestamp" => 2018-03-09T08:06:22.293Z,
"@version" => "1",
"source" => "C:\log.slf",
"doc" => {
"Level" => "Info",
"Host" => "Host1",
"Method" => "Company.Utilities.DefaultLogs.CommandUsageLog::Info",
"User" => "User",
"Categories" => "client.performance.commands, common.assembly.utilities",
"AppDomain" => "DefaultDomain [1]",
"Message" => "c6411347-96cf-4330-b3e5-310d523adbba\tSave Image\t3D\tTime (ms):\t20\tCase:\t6e5e2d3a-ce1c-453c-93ff-4e9ce67936fe",
"Thread" => "[1]",
"Time" => "2018-01-16T19:00:24.6785488+00:00"

(Magnus Bäck) #8

Field names are case sensitive so use [doc][Message]. I missed the uppercase M earlier, sorry.

(Lucas Horn) #9

No worries. Despite trying that earlier, this time it worked. It doesn't give any error messages, but neither does it create the [doc][Case] I want either, of course. Any thoughts on that?

(Magnus Bäck) #10

Splitting doesn't do that for you. I suggest you use a grok or dissect filter to process the string and extract the fields you want.

(Lucas Horn) #11

Right, thanks for the help!

(system) #12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.