Move subarrays to document root

Elastic Stack Logstash
1

Greetings
My incoming messages contain subarrays in the following format:

com.openexchange.ajax.action
com.openexchange.ajax.module
com.openexchange.database.schema
com.openexchange.grizzly.method
com.openexchange.grizzly.queryString
com.openexchange.grizzly.remoteAddress
com.openexchange.grizzly.remotePort
com.openexchange.grizzly.requestURI
com.openexchange.grizzly.serverName
com.openexchange.grizzly.servletPath
com.openexchange.grizzly.session
com.openexchange.grizzly.threadName
com.openexchange.grizzly.userAgent
com.openexchange.login.authId
com.openexchange.login.client
com.openexchange.login.clientIp
com.openexchange.login.login
com.openexchange.login.userAgent
com.openexchange.login.version
com.openexchange.mail.accountId
com.openexchange.mail.host
com.openexchange.mail.login
com.openexchange.mail.session
com.openexchange.request.trackingId
com.openexchange.session.authId
com.openexchange.session.clientId
com.openexchange.session.contextId
com.openexchange.session.sessionId
com.openexchange.session.userId
com.openexchange.session.userName

However, I would like to strip off the com.openexchange part of each field. Is that possible? Preferrably without having to list the ajax, database, etc. subfields.
I tried this filter, but it doesn't move anything:

mutate {
    rename => { "com.openexchange.grizzly" => "grizzly" }
    rename => { "com.openexchange.mail"    => "mail" }
    rename => { "com.openexchange.request" => "request" }
    rename => { "com.openexchange.database"=> "database" }
    rename => { "com.openexchange.ajax"    => "ajax" }
    rename => { "com.openexchange.login"   => "login" }
    rename => { "com.openexchange.session" => "session" }
}

I also tried a Ruby filter, but it creates fields with null in them:

ruby {
    code => '
        event.set("[grizzly]", event.get("[com][openexchange][grizzly]"))
        event.set("[mail]", event.get("[com][openexchange][mail]"))
        event.set("[request]", event.get("[com][openexchange][request]"))
        event.set("[database]", event.get("[com][openexchange][database]"))
        event.set("[ajax]", event.get("[com][openexchange][ajax]"))
        event.set("[login]", event.get("[com][openexchange][login]"))
        event.set("[session]", event.get("[com][openexchange][session]"))
    '
}

Of course, renaming individual leafs will work, but that's tedious, and not general in case I add more in the log source.

2 
    ruby { code => 'event.get("[com][openexchange]").each { | k, v| event.set(k, v) }' }
    mutate { remove_field => [ "[com][openexchange]", "com" ] }
3

Thanks, but that didn't do it. Now I'm getting

Ruby exception occurred: undefined method `each' for nil:NilClass

However, the fields should exist, I certainly do see them in Kibana.

4

What exactly do you see in Kibana. Are they arrays? Can you copy and paste from the JSON tab?

5

Here is a part of one document that I can see in Kibana:

{
  "_index": "clog-mail-2018.08.10",
  "_type": "doc",
  "_id": "SqKWJGUBcUjz8ltKlwa6",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2018-08-10T16:06:08.891Z",
    "@version": 1,
    "com.openexchange.grizzly.method": "GET",
    "com.openexchange.grizzly.threadName": "OXWorker-0001932",
    "com.openexchange.grizzly.remotePort": "18716",
    ...
    "tags": [
      "oxbe",
      "open-xchange",
      "beats_input_codec_plain_applied",
      "_rubyexception",
    ]
  },
  "fields": {
    "@timestamp": [
      "2018-08-10T16:06:08.891Z"
    ]
  }
}

So, my mistake, they don't seem to be actual arrays. I apologise for the confusion.

6

OK, so you have dots in your field names. That is unsupported, and I know there are things that do not work with such fields.

Take a look at the de_dot filter. Note that it explicitly warns that it is expensive to run. But this would transform that into something that the ruby filter would work on.

de_dot { nested => true }
1 Like
7

Excellent, that works! However, there are some clashes with fields that already exist in the document (e.g. requests already exists as a string). What would be the easiest way to store all of the subfields into an ox field rather than the document root?

8

A slight change to the ruby filter

ruby { code => 'event.get("[com][openexchange]").each { | k, v| event.set("[ox][" + k + "]", v) }' }
1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.