Map nested fields of object from Logstash to Elasticsearch

[kejsi]

Map nested fields of object from Logstash to Elasticsearch

I have an object which looks like this:

"dummyObj":"{"id":6,"nrs":[1,2,3,4]}"

I supply this object from Log4j2 using JsonLayout. Moreover, I have created an index in Elasticsearch like this:

{
"mappings": {
        "dummy": {
                "properties": { 
                        "parsedDummyObj": {
                            "type": "nested",
                            "properties": {
                                "id": { "type": "text" },
                                "numbers": { "type": "text" }
                            }
                        }
                }                                                                                          
        }
}
}

Now, in Logstash I do this:

filter {

  json {
        source => "dummyObj"
        target => "parsedDummyObj"
        remove_field=>["dummyObj"]
    }
}

However, I get the error: object mapping [parsedDummyObj] can't be changed from nested to non-nested"

My question is: what type of filter/mapping should I use, for the json (dummyObj), to be added successfully in Elasticsearch?

PS: I have researched, and I know the answer may implicitly exist out there, but I am extremely too knew to Elastic and hope for your understanding.

[dadoonet (David Pilato)]

You should print what is generated by Logstash in the output part with a stdout plugin.
You will see that probably your parsedDummyObj is looking like:

"parsedDummyObj": {
  //
}

Instead of:

"parsedDummyObj": [{
  //
}]

The later is expected for nested documents.

Why did you define parsedDummyObj as nested in your mapping?

[kejsi]

Hey @dadoonet thanks so much for the reply!

I want expandable fields in Kibana, and I am fully aware that it is not supported by default, that's why I came across the KNQL plugin .

So I want something like this in the screenshot, for the parsedDummyObj:

[kejsi]

As for the right syntax for the nested types, I also came across this (which apparently is only a json string not an array):

[dadoonet (David Pilato)]

That does not answer to my question:

Why did you define parsedDummyObj as nested in your mapping?

I still don't understand what is your business requirement for this.

Don't think of the plugin you are using. Just think about your use case first. Describe objects as you want them to be.

[kejsi]

Thank you @dadoonet !

The requirement is to visualize custom properties and custom objects in Kibana, so they can appear on the left of the dashboard like all searchable fields:

(The gray area next to the blue and red area).

So my object is simple for testing purposes, and has only two fields, but it could have had another object as a field.

So what I want is: if I define something as "nested", I want the fields that are within to:

1. Be expanded in the dashboard (as in the screenshot in the previous answer)
2. Eventually to be shown in the gray area so that I can use the zoom button to search for them

Please let me know if you need further details!

Many thanks!

[dadoonet (David Pilato)]

@kejsi

The requirement is to visualize custom properties and custom objects in Kibana

I'm sorry but IMHO this is not a business requirement. This is a technical implementation or solution.

But anyway, let me describe what nested is in elasticsearch.

Nested objects are used to index structures like:

{
  "text": "hello",
  "objects": [{
    "foo": "bar",
    "x": 1
  }, {
    "foo": "baz",
    "x": 2
  }]
}

When you index without using a nested type in mapping, you end up indexing a document like:

{
  "text": "hello",
  "objects.foo": ["bar", "baz"],
  "objects.x": [1, 2]
}

When you use nested you are indexing actually 3 documents in Lucene behind the scene:

{
  "text": "hello"
}
{
  "objects.foo": "bar",
  "objects.x": 1
}
{
  "objects.foo": "baz",
  "objects.x": 1
}

So using nested in the mapping is "only" a way to describe which index implementation you want.

It's not related to any graphical representation in Kibana for one single sub object.

I hope this clarifies.
In your case, do not define parsedDummyObj as a nested type and you should be good.

[kejsi]

Thanks for sharing your knowledge @dadoonet its really helpful!

Yes, you are right, if I get away with the "nested" prop, I get to output the object. (see screenshot)

.

But here, I have all the fields scattered, it goes without saying that it would be more convenient for users to have the object as expandable, rather than show each field like this.

What do you think in this case?

[dadoonet (David Pilato)]

For anything related to UI I'd ask in #kibana forum instead.