# Define field as nested type using Logstash

**URL:** <https://discuss.elastic.co/t/define-field-as-nested-type-using-logstash/242289>\
**Category:** Logstash\
**Created:** [July 23, 2020, 6:50am UTC](https://discuss.elastic.co/t/define-field-as-nested-type-using-logstash/242289 "2020-07-23T06:50:13Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yoorae\_Kim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yoorae_kim/32/56310_2.png) [@Yoorae\_Kim](https://discuss.elastic.co/u/Yoorae_Kim)\
**Post date:** [July 23, 2020, 6:50am UTC](https://discuss.elastic.co/t/define-field-as-nested-type-using-logstash/242289/1 "2020-07-23T06:50:14Z")

</div>

Hello,  
I am currently shipping my data to Elasticsearch using Logstash.  
The inner-hit query I am trying to perform is returning an error saying

```auto
"failed to create query: [nested] nested object under path [transactions] is not of nested type"

```

Thus, I've been trying to define the 'transactions' field as a nested field using Logstash filters and output APIs, but I'm keep running into sorts of errors.

```auto
mutate {
    convert => { "transactions" => "nested" }
  }

```

returns

```auto
:message=>"Could not execute action: PipelineAction::Reload<main>, action_result: false", :backtrace=>nil}

```

and creating a mapping template and calling it at the output following the blog post of

> **[​Little Logstash Lessons: Using Logstash to help create an Elasticsearch...](https://www.elastic.co/blog/logstash_lesson_elasticsearch_mapping)**
>
> How to use Logstash together with Elasticsearch to create custom mapping templates. Improve your Elasticsearch storage and performance!

returns

```auto
Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'http://localhost:9200/_template/ether_template'", 

```

this error.

Is there any way to define a mapping type as 'nested' using Logstash?

---

<div class="post-metadata">

**Author:** ![Jenni](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jenni/32/29684_2.png) [@Jenni](https://discuss.elastic.co/u/Jenni)\
**Post date:** [July 23, 2020, 7:12am UTC](https://discuss.elastic.co/t/define-field-as-nested-type-using-logstash/242289/2 "2020-07-23T07:12:53Z")

</div>

You can not _convert_ something else to `nested` because `nested` is a datatype in Elasticsearch that represents an array of one or more objects. In Logstash this would be an array of one or more hashes – fields with subfields. I guess your `transactions` was not an array, but only one element with subfields when it appeared for the first time. So the dynamic mapping mapped it as `object` (data type for only one object) instead of `nested`.

Your problem with installing the template might have to do with the template not being compatible with your ES version. That tutorial is from 2017 and uses document types that have been deprecated for a while.

---

<div class="post-metadata">

**Author:** ![Yoorae\_Kim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yoorae_kim/32/56310_2.png) [@Yoorae\_Kim](https://discuss.elastic.co/u/Yoorae_Kim)\
**Post date:** [July 23, 2020, 7:21am UTC](https://discuss.elastic.co/t/define-field-as-nested-type-using-logstash/242289/3 "2020-07-23T07:21:53Z")

</div>

Transactions field has multiple elements with multiple subfields to itself.

For example,  
I edited transactions field using ruby filter this way:

```auto
transactions_num.times do |index|
        event.set("[transactions][#{index}][gasprice_int]", event.get("[transactions][#{index}][gasPrice]").to_i(16))
        event.set("[transactions][#{index}][transactionIndex]", event.get("[transactions][#{index}][transactionIndex]").to_i(16))

```

It should be mapped as nested because it's an array of multiple objects.

Would you please check if I did my inner hit query the right way?

```auto
GET etherbeat-8.0.0-2020.07.23/_search
{
  "_source": {
    "includes": ["*"],
    "excludes": ["transactions"]
  },
  "query": {
    "nested": {
      "path": "transactions", 
      "inner_hits": {        
        "_source": [
          "address"
        ]
      },
      "query": {
        "bool": {
          "must": [
            {
              "term": {
                "transactions.address": "0xaaaaaaaaa"
              }
            }
          ]
        }
      }
    }
  }
}

```

Thank you so much btw T.T

---

<div class="post-metadata">

**Author:** ![Yoorae\_Kim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yoorae_kim/32/56310_2.png) [@Yoorae\_Kim](https://discuss.elastic.co/u/Yoorae_Kim)\
**Post date:** [July 23, 2020, 7:26am UTC](https://discuss.elastic.co/t/define-field-as-nested-type-using-logstash/242289/4 "2020-07-23T07:26:31Z")

</div>

Each block contains 100+ transactions objects under transactions field, and I want the query to only return single transaction that matches the address query instead of the whole document which is a block itself that contains 100+ transactions.

Is there any way to do this without using inner hit?

---

<div class="post-metadata">

**Author:** ![Jenni](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jenni/32/29684_2.png) [@Jenni](https://discuss.elastic.co/u/Jenni)\
**Post date:** [July 23, 2020, 7:27am UTC](https://discuss.elastic.co/t/define-field-as-nested-type-using-logstash/242289/5 "2020-07-23T07:27:29Z")

</div>

What does `GET etherbeat-8.0.0-2020.07.23/_mapping` say?

---

<div class="post-metadata">

**Author:** ![Yoorae\_Kim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yoorae_kim/32/56310_2.png) [@Yoorae\_Kim](https://discuss.elastic.co/u/Yoorae_Kim)\
**Post date:** [July 23, 2020, 7:30am UTC](https://discuss.elastic.co/t/define-field-as-nested-type-using-logstash/242289/6 "2020-07-23T07:30:32Z")

</div>

It's really long so I only pasted the transactions part which goes like

```auto
"transactions" : {
          "properties" : {
            "address" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "blockHash" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
             and blah blah blah 

```

so yeah I think it is dynamically mapped as an object with multiple fields as you said.  
I don't know why because transactions field contains multiple elements.  
I don't know how I would define transactions field as nested, or perform the query as the way I want to.

---

<div class="post-metadata">

**Author:** ![Jenni](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jenni/32/29684_2.png) [@Jenni](https://discuss.elastic.co/u/Jenni)\
**Post date:** [July 23, 2020, 7:48am UTC](https://discuss.elastic.co/t/define-field-as-nested-type-using-logstash/242289/7 "2020-07-23T07:48:06Z")

</div>

I just realized that the dynamic mapping generally doesn't assign the nested type. It only creates the data type object. Object fields with multiple entries might look the same, but have a different internal representation than nested:  
[https://www.elastic.co/de/blog/managing-relations-inside-elasticsearch](https://www.elastic.co/de/blog/managing-relations-inside-elasticsearch)

You should probably try to _PUT_ the template in Elasticsearch and check what errors the API throws and then correct the template accordingly.

---

<div class="post-metadata">

**Author:** ![Yoorae\_Kim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yoorae_kim/32/56310_2.png) [@Yoorae\_Kim](https://discuss.elastic.co/u/Yoorae_Kim)\
**Post date:** [July 23, 2020, 8:54am UTC](https://discuss.elastic.co/t/define-field-as-nested-type-using-logstash/242289/8 "2020-07-23T08:54:28Z")

</div>

I fixed the errors by following the template format for version 7 on GitHub :[https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/templates/ecs-disabled/elasticsearch-7x.json](https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/templates/ecs-disabled/elasticsearch-7x.json)

and the uploaded template 'transactions' field seems to be of a nested type:

```auto
GET _template/ether_template/

```

This returns

```auto
"transactions" : {
          "type" : "nested",
          "properties" : {

```

and Logstash is outputting with the template

```auto
[2020-07-23T17:35:30,557][INFO][logstash.outputs.elasticsearch][main] Using mapping template from {:path=>"/usr/local/etc/logstash/ether-mappings.json"}

```

but my actual index mapping is still not mapped as nested type......

Logstash is giving me a warning saying

```auto
[2020-07-23T17:35:30,430][WARN][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}

```

---

<div class="post-metadata">

**Author:** ![Jenni](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jenni/32/29684_2.png) [@Jenni](https://discuss.elastic.co/u/Jenni)\
**Post date:** [July 23, 2020, 9:02am UTC](https://discuss.elastic.co/t/define-field-as-nested-type-using-logstash/242289/9 "2020-07-23T09:02:32Z")

</div>

You'll have to create a new index. The old one won't be affected by the template.

---

<div class="post-metadata">

**Author:** ![Yoorae\_Kim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yoorae_kim/32/56310_2.png) [@Yoorae\_Kim](https://discuss.elastic.co/u/Yoorae_Kim)\
**Post date:** [July 23, 2020, 9:06am UTC](https://discuss.elastic.co/t/define-field-as-nested-type-using-logstash/242289/10 "2020-07-23T09:06:39Z")

</div>

Yep... deleting the index and restarting the Logstash fixed it.  
TYSM!!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 20, 2020, 9:06am UTC](https://discuss.elastic.co/t/define-field-as-nested-type-using-logstash/242289/11 "2020-08-20T09:06:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.