Dynamically bring nested fields to root level

Elastic Stack Logstash
1

I need to extract fields, full_name, roles, email and username from below output

"Mika.singh" => {
         "metadata" => {},
        "full_name" => "mika singh",
            "roles" => [
            [0] "teaml",
            [1] "team-interface",
            [2] "dev",
            [3] "support-read"
        ],
            "email" => "mika.singh@xyz",
          "enabled" => true,
         "username" => "mika.singh"
    },
"logstash_user" => {
     "metadata" => {},
    "full_name" => "",
        "roles" => [
        [0] "logstash_system",
        [1] "logstash_admin"
    ],
        "email" => "",
      "enabled" => true,
     "username" => "logstash_user"
},

But as you can see top level field is different every time, I tried with ruby filter but none of them worked,

Please help me on this.

2

So you have a pair of hashes called logstash_user and Mika.singh. What is the structure of the data in which they occur?

3

I am trying to pull kibana users through logstash pipeline and store it in index

input {
  http_poller {
    urls => {
      kibana_users => {
        # Supports all options supported by ruby's Manticore HTTP client
        method => get
        user => "elastic"
        password => "xyz123"
        url => "https://10.2.3.20:9200/_security/user"
        headers => {
          Accept => "application/json"
        }
     }
    }
    truststore => "/usr/share/logstash/downloaded_truststore.jks"
    truststore_password => "xyz123"
    request_timeout => 60
    # Supports "cron", "every", "at" and "in" schedules by rufus scheduler
    schedule => { cron => "* * * * * UTC"}
    codec => "json"
    # A hash of request metadata info (timing, response headers, etc.) will be sent here
    metadata_target => "http_poller_metadata"
  }
}
4

You are saying you want to change the structure of your data, by moving fields to the top-level, but you have not explained what the existing structure of your data is. We cannot answer a question that has not been asked.

5

Sorry Badger, But I am trying to pull list of users inside Kibana through user Api and store in an index to create visulization , for that I need fields
like
username, email, roles and full name

I am using curl command to pull the users list.

curl -X GET --cacert /usr/share/logstash/ca.pem -u elastic:xyz123 https://10.2.3.20:9200/_security/user

Getting below results from query

 
{"elastic":{"username":"elastic","roles":["superuser"],"full_name":null,"email":null,"metadata":{"_reserved":true},"enabled":true},"kibana":{"username":"kibana","roles":["kibana_system"],"full_name":null,"email":null,"metadata":{"_reserved":true,"_deprecated_reason":"Please use the [kibana_system] user instead.","_deprecated":true},"enabled":true},"kibana_system":{"username":"kibana_system","roles":["kibana_system"],"full_name":null,"email":null,"metadata":{"_reserved":true},"enabled":true}
6

You could try something like

    ruby {
        code => '
            users = []
            event.to_hash.each { |k, v|
                if v.is_a? Hash
                    users << v
                    event.remove(k)
                end
            }
            event.set("users", users)
        '
    }
    split { field => "users" }
    ruby {
        code => '
            event.remove("users").each { |k, v|
                event.set(k,v)
            }
        '
    }