Define field as nested type using Logstash

[Yoorae_Kim]

Hello,
I am currently shipping my data to Elasticsearch using Logstash.
The inner-hit query I am trying to perform is returning an error saying

"failed to create query: [nested] nested object under path [transactions] is not of nested type"

Thus, I've been trying to define the 'transactions' field as a nested field using Logstash filters and output APIs, but I'm keep running into sorts of errors.

mutate {
    convert => { "transactions" => "nested" }
  }

returns

:message=>"Could not execute action: PipelineAction::Reload<main>, action_result: false", :backtrace=>nil}

and creating a mapping template and calling it at the output following the blog post of

​Little Logstash Lessons: Using Logstash to help create an Elasticsearch...

returns

Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'http://localhost:9200/_template/ether_template'", 

this error.

Is there any way to define a mapping type as 'nested' using Logstash?

[Jenni]

You can not convert something else to nested because nested is a datatype in Elasticsearch that represents an array of one or more objects. In Logstash this would be an array of one or more hashes – fields with subfields. I guess your transactions was not an array, but only one element with subfields when it appeared for the first time. So the dynamic mapping mapped it as object (data type for only one object) instead of nested.

Your problem with installing the template might have to do with the template not being compatible with your ES version. That tutorial is from 2017 and uses document types that have been deprecated for a while.

[Yoorae_Kim]

Transactions field has multiple elements with multiple subfields to itself.

For example,
I edited transactions field using ruby filter this way:

transactions_num.times do |index|
        event.set("[transactions][#{index}][gasprice_int]", event.get("[transactions][#{index}][gasPrice]").to_i(16))
        event.set("[transactions][#{index}][transactionIndex]", event.get("[transactions][#{index}][transactionIndex]").to_i(16))

It should be mapped as nested because it's an array of multiple objects.

Would you please check if I did my inner hit query the right way?

GET etherbeat-8.0.0-2020.07.23/_search
{
  "_source": {
    "includes": [ "*" ],
    "excludes": [ "transactions" ]
  },
  "query": {
    "nested": {
      "path": "transactions", 
      "inner_hits": {        
        "_source": [
          "address"
        ]
      },
      "query": {
        "bool": {
          "must": [
            {
              "term": {
                "transactions.address": "0xaaaaaaaaa"
              }
            }
          ]
        }
      }
    }
  }
}

Thank you so much btw T.T

[Yoorae_Kim]

Each block contains 100+ transactions objects under transactions field, and I want the query to only return single transaction that matches the address query instead of the whole document which is a block itself that contains 100+ transactions.

Is there any way to do this without using inner hit?

[Jenni]

What does GET etherbeat-8.0.0-2020.07.23/_mapping say?

[Yoorae_Kim]

It's really long so I only pasted the transactions part which goes like

"transactions" : {
          "properties" : {
            "address" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "blockHash" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
             and blah blah blah 

so yeah I think it is dynamically mapped as an object with multiple fields as you said.
I don't know why because transactions field contains multiple elements.
I don't know how I would define transactions field as nested, or perform the query as the way I want to.

[Jenni]

I just realized that the dynamic mapping generally doesn't assign the nested type. It only creates the data type object. Object fields with multiple entries might look the same, but have a different internal representation than nested:
https://www.elastic.co/de/blog/managing-relations-inside-elasticsearch

You should probably try to PUT the template in Elasticsearch and check what errors the API throws and then correct the template accordingly.

[Yoorae_Kim]

I fixed the errors by following the template format for version 7 on GitHub :https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/templates/ecs-disabled/elasticsearch-7x.json

and the uploaded template 'transactions' field seems to be of a nested type:

GET _template/ether_template/

This returns

"transactions" : {
          "type" : "nested",
          "properties" : {

and Logstash is outputting with the template

[2020-07-23T17:35:30,557][INFO ][logstash.outputs.elasticsearch][main] Using mapping template from {:path=>"/usr/local/etc/logstash/ether-mappings.json"}

but my actual index mapping is still not mapped as nested type......

Logstash is giving me a warning saying

[2020-07-23T17:35:30,430][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}

[Jenni]

You'll have to create a new index. The old one won't be affected by the template.

[Yoorae_Kim]

Yep... deleting the index and restarting the Logstash fixed it.
TYSM!!