# Map nested fields of object from Logstash to Elasticsearch

**URL:** https://discuss.elastic.co/t/map-nested-fields-of-object-from-logstash-to-elasticsearch/163869
**Category:** Elasticsearch
**Created:** [January 11, 2019, 8:54am UTC](https://discuss.elastic.co/t/map-nested-fields-of-object-from-logstash-to-elasticsearch/163869 "2019-01-11T08:54:13Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![kejsi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kejsi/32/39670_2.png) [@kejsi](https://discuss.elastic.co/u/kejsi)
#### Post date: [January 11, 2019, 8:54am UTC](https://discuss.elastic.co/t/map-nested-fields-of-object-from-logstash-to-elasticsearch/163869/1 "2019-01-11T08:54:13Z")

</div>

> <https://stackoverflow.com/questions/54142349/map-nested-fields-of-object-from-logstash-to-elasticsearch>

I have an object which looks like this:

```auto
"dummyObj":"{"id":6,"nrs":[1,2,3,4]}"

```

I supply this object from Log4j2 using JsonLayout. Moreover, I have created an index in Elasticsearch like this:

```auto
{
"mappings": {
        "dummy": {
                "properties": { 
                        "parsedDummyObj": {
                            "type": "nested",
                            "properties": {
                                "id": { "type": "text" },
                                "numbers": { "type": "text" }
                            }
                        }
                }                                                                                          
        }
}
}

```

Now, in Logstash I do this:

```auto
filter {

  json {
        source => "dummyObj"
        target => "parsedDummyObj"
        remove_field=>["dummyObj"]
    }
}

```

However, I get the error: **object mapping [parsedDummyObj] can't be changed from nested to non-nested"**

My question is: what type of filter/mapping should I use, for the json (dummyObj), to be added successfully in Elasticsearch?

PS: I have researched, and I know the answer may implicitly exist out there, but I am extremely too knew to Elastic and hope for your understanding.

---

<div class="post-metadata">

### Author: ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [January 11, 2019, 10:15am UTC](https://discuss.elastic.co/t/map-nested-fields-of-object-from-logstash-to-elasticsearch/163869/2 "2019-01-11T10:15:19Z")

</div>

You should print what is generated by Logstash in the output part with a stdout plugin.  
You will see that probably your `parsedDummyObj` is looking like:

```auto
"parsedDummyObj": {
  //
}

```

Instead of:

```auto
"parsedDummyObj": [{
  //
}]

```

The later is expected for nested documents.

Why did you define `parsedDummyObj` as `nested` in your mapping?

---

<div class="post-metadata">

### Author: ![kejsi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kejsi/32/39670_2.png) [@kejsi](https://discuss.elastic.co/u/kejsi)
#### Post date: [January 11, 2019, 10:18am UTC](https://discuss.elastic.co/t/map-nested-fields-of-object-from-logstash-to-elasticsearch/163869/3 "2019-01-11T10:18:36Z")

</div>

Hey @dadoonet thanks so much for the reply!

I want expandable fields in Kibana, and I am fully aware that it is not supported by default, that's why I came across the [KNQL plugin](https://ppadovani.github.io/knql_plugin/knql/) .

So I want something like this in the screenshot, for the parsedDummyObj:

 ![exapandable](https://us1.discourse-cdn.com/elastic/original/3X/d/c/dc9ae101464cc3a0c73ee8f598cdd502b9f4b047.jpeg)

---

<div class="post-metadata">

### Author: ![kejsi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kejsi/32/39670_2.png) [@kejsi](https://discuss.elastic.co/u/kejsi)
#### Post date: [January 11, 2019, 10:20am UTC](https://discuss.elastic.co/t/map-nested-fields-of-object-from-logstash-to-elasticsearch/163869/4 "2019-01-11T10:20:59Z")

</div>

As for the right syntax for the nested types, I also came across this (which apparently is only a json string not an array):

 ![unexpandable](https://us1.discourse-cdn.com/elastic/original/3X/6/8/689f07c99ded46c925fe40b4dc895b63070bdc60.png)

---

<div class="post-metadata">

### Author: ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [January 11, 2019, 10:31am UTC](https://discuss.elastic.co/t/map-nested-fields-of-object-from-logstash-to-elasticsearch/163869/5 "2019-01-11T10:31:53Z")

</div>

That does not answer to my question:

> Why did you define `parsedDummyObj` as `nested` in your mapping?

I still don't understand what is your business requirement for this.

Don't think of the plugin you are using. Just think about your use case first. Describe objects as you want them to be.

---

<div class="post-metadata">

### Author: ![kejsi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kejsi/32/39670_2.png) [@kejsi](https://discuss.elastic.co/u/kejsi)
#### Post date: [January 11, 2019, 10:43am UTC](https://discuss.elastic.co/t/map-nested-fields-of-object-from-logstash-to-elasticsearch/163869/6 "2019-01-11T10:43:58Z")

</div>

Thank you @dadoonet !

The requirement is to visualize custom properties and custom objects in Kibana, so they can appear on the left of the dashboard like all searchable fields:

 ![Selection_008](https://us1.discourse-cdn.com/elastic/original/3X/9/0/905c9aaa29841e4366989f633c954bfaff04f3bf.png) (The gray area next to the blue and red area).

So my object is simple for testing purposes, and has only two fields, but it could have had another object as a field.

So what I want is: if I define something as "nested", I want the fields that are within to:

1. Be expanded in the dashboard (as in the screenshot in the previous answer)
2. Eventually to be shown in the gray area so that I can use the zoom button to search for them

Please let me know if you need further details!

Many thanks!

---

<div class="post-metadata">

### Author: ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [January 11, 2019, 11:07am UTC](https://discuss.elastic.co/t/map-nested-fields-of-object-from-logstash-to-elasticsearch/163869/7 "2019-01-11T11:07:27Z")

</div>

> [@kejsi](#):
>
> The requirement is to visualize custom properties and custom objects in Kibana

I'm sorry but IMHO this is not a business requirement. This is a technical implementation or solution.

But anyway, let me describe what `nested` is in elasticsearch.

Nested objects are used to index structures like:

```auto
{
  "text": "hello",
  "objects": [{
    "foo": "bar",
    "x": 1
  }, {
    "foo": "baz",
    "x": 2
  }]
}

```

When you index without using a nested type in mapping, you end up indexing a document like:

```auto
{
  "text": "hello",
  "objects.foo": ["bar", "baz"],
  "objects.x": [1, 2]
}

```

When you use nested you are indexing actually 3 documents in Lucene behind the scene:

```auto
{
  "text": "hello"
}
{
  "objects.foo": "bar",
  "objects.x": 1
}
{
  "objects.foo": "baz",
  "objects.x": 1
}

```

So using `nested` in the mapping is "only" a way to describe which index implementation you want.

It's not related to any graphical representation in Kibana for one single sub object.

I hope this clarifies.  
In your case, do not define `parsedDummyObj` as a `nested` type and you should be good.

---

<div class="post-metadata">

### Author: ![kejsi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kejsi/32/39670_2.png) [@kejsi](https://discuss.elastic.co/u/kejsi)
#### Post date: [January 11, 2019, 11:29am UTC](https://discuss.elastic.co/t/map-nested-fields-of-object-from-logstash-to-elasticsearch/163869/8 "2019-01-11T11:29:44Z")

</div>

Thanks for sharing your knowledge @dadoonet its really helpful!

Yes, you are right, if I get away with the "nested" prop, I get to output the object. (see screenshot)

 ![Selection_011](https://us1.discourse-cdn.com/elastic/original/3X/4/a/4a52988d32e85e0c58d7f5f969584684a222539c.png) .

But here, I have all the fields scattered, it goes without saying that it would be more convenient for users to have the object as expandable, rather than show each field like this.

What do you think in this case?

---

<div class="post-metadata">

### Author: ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [January 11, 2019, 11:55am UTC](https://discuss.elastic.co/t/map-nested-fields-of-object-from-logstash-to-elasticsearch/163869/9 "2019-01-11T11:55:34Z")

</div>

For anything related to UI I'd ask in #kibana forum instead.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 8, 2019, 11:55am UTC](https://discuss.elastic.co/t/map-nested-fields-of-object-from-logstash-to-elasticsearch/163869/10 "2019-02-08T11:55:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
