Query not returning document - mapping issue?

[jong99]

I'm having problems when querying for a document and my search is not working. It's probably something simple such as an issue with my mapping. The queries below were generated by Kibana but I've simplified below to try to isolate the problem. I've also omitted a few extraneous fields for privacy but I hope it doesn't affect the issue here.

Note: I'm using Elasticsearch 6.8.4

Document I'm trying to locate:

{
    "event": "buildlog",
    "data": {
      "sTags": "gentest_v6",
    },
    "@timestamp": "2019-12-13T12:31:23.207Z",
    "type": "resourceanalysis"
}

Mapping Template

{
  "order": 0,
  "version": 50001,
  "index_patterns": [
    "myindexname"
  ],
  "settings": {
    "index": {
      "codec": "default",
      "mapping": {
        "total_fields": {
          "limit": "5000"
        }
      },
      "refresh_interval": "30s",
      "number_of_shards": "3",
      "priority": "5",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "event": {
      "dynamic_templates": [
        {
          "objs": {
            "path_match": "data.*",
            "mapping": {
              "type": "nested"
            },
            "match_mapping_type": "object"
          }
        },
        {
          "s_prefix": {
            "match_pattern": "regex",
            "path_match": "data.*",
            "mapping": {
              "ignore_above": 10000,
              "type": "keyword"
            },
            "match": "^s.+$"
          }
        },
        {
          "d_prefix_double": {
            "match_pattern": "regex",
            "path_match": "data.*",
            "mapping": {
              "type": "double"
            },
            "match_mapping_type": "double",
            "match": "^d.+$"
          }
        },
        {
          "f_prefix_float": {
            "match_pattern": "regex",
            "path_match": "data.*",
            "mapping": {
              "type": "double"
            },
            "match_mapping_type": "double",
            "match": "^f.+$"
          }
        },
        {
          "i_prefix_integer": {
            "match_pattern": "regex",
            "path_match": "data.*",
            "mapping": {
              "type": "long"
            },
            "match_mapping_type": "long",
            "match": "^i.+$"
          }
        },
        {
          "l_prefix_long": {
            "match_pattern": "regex",
            "path_match": "data.*",
            "mapping": {
              "type": "long"
            },
            "match_mapping_type": "long",
            "match": "^l.+$"
          }
        },
        {
          "b_prefix": {
            "match_pattern": "regex",
            "path_match": "data.*",
            "mapping": {
              "ignore_above": 10000,
              "type": "boolean"
            },
            "match": "^b.+$"
          }
        }
      ],
      "properties": {
        "data": {
          "type": "nested"
        },
        "build": {
          "type": "keyword"
        },
        "event": {
          "type": "keyword"
        },
        "type": {
          "type": "keyword"
        },
        "user": {
          "type": "keyword"
        }
      }
    }
  },
  "aliases": {}
}

Mapping of index

{
  "myindexname": {
    "mappings": {
      "event": {
        "dynamic_templates": [
          {
            "objs": {
              "path_match": "data.*",
              "match_mapping_type": "object",
              "mapping": {
                "type": "nested"
              }
            }
          },
          {
            "s_prefix": {
              "match": "^s.+$",
              "path_match": "data.*",
              "match_pattern": "regex",
              "mapping": {
                "ignore_above": 10000,
                "type": "keyword"
              }
            }
          },
          {
            "d_prefix_double": {
              "match": "^d.+$",
              "path_match": "data.*",
              "match_mapping_type": "double",
              "match_pattern": "regex",
              "mapping": {
                "type": "double"
              }
            }
          },
          {
            "f_prefix_float": {
              "match": "^f.+$",
              "path_match": "data.*",
              "match_mapping_type": "double",
              "match_pattern": "regex",
              "mapping": {
                "type": "double"
              }
            }
          },
          {
            "i_prefix_integer": {
              "match": "^i.+$",
              "path_match": "data.*",
              "match_mapping_type": "long",
              "match_pattern": "regex",
              "mapping": {
                "type": "long"
              }
            }
          },
          {
            "l_prefix_long": {
              "match": "^l.+$",
              "path_match": "data.*",
              "match_mapping_type": "long",
              "match_pattern": "regex",
              "mapping": {
                "type": "long"
              }
            }
          },
          {
            "b_prefix": {
              "match": "^b.+$",
              "path_match": "data.*",
              "match_pattern": "regex",
              "mapping": {
                "ignore_above": 10000,
                "type": "boolean"
              }
            }
          }
        ],
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "data": {
            "type": "nested",
            "properties": {
              "sTags": {
                "type": "keyword",
                "ignore_above": 10000
              }
            }
          },
          "event": {
            "type": "keyword"
          },
          "type": {
            "type": "keyword"
          }
        }
      }
    }
  }
}

When I query directly for the document by ID then I find the document:

Query that successfully returns the document, searching by ID

{
  "version": true,
  "size": 500,
  "query": {
    "bool": {
      "must": [
        {
          "match_phrase": {
            "_id": {
              "query": "YzEJZ28B0juYBJ2odwa3"
            }
          }
        }
      ]
    }
  }
}

Response

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 9,
    "successful": 9,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 1,
    "hits": [
      {
        "_index": "myindexname",
        "_type": "event",
        "_id": "YzEJZ28B0juYBJ2odwa3",
        "_version": 2,
        "_score": 1,
        "_source": {
          "event": "buildlog",
          "data": {
            "sTags": "gentest_v6"
          },
          "@timestamp": "2019-12-13T12:31:23.207Z",
          "type": "resourceanalysis"
        }
      }
    ]
  }
}

But when I start to filter by other fields my queries do not return the document:

Querying for another field is NOT returning document??

{
  "version": true,
  "size": 500,
  "query": {
    "bool": {
      "must": [
        {
          "match_phrase": {
            "data.sTags": {
              "query": "gentest_v6"
            }
          }
        },
        {
          "match_phrase": {
            "_id": {
              "query": "YzEJZ28B0juYBJ2odwa3"
            }
          }
        }
      ]
    }
  }
}

Response:

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 9,
    "successful": 9,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 0,
    "max_score": null,
    "hits": [

    ]
  }
}

Any ideas why this second query does not return the document? It's probably something simple I'm missing!

Thanks!

[William_Brafford]

Hi,

I have one suggestion based on a quick look at what you've posted. Your index mapping has data as a nested type. I believe that means that you need to use a nested query for such types.

From the Nested datatype docs:

Because nested documents are indexed as separate documents, they can only be accessed within the scope of the nested query, the nested / reverse_nested aggregations, or nested inner hits.

I am thinking that something like this might work:

{
  "version": true,
  "size": 500,
  "query": {
    "bool": {
      "must": [
        {
          "nested": {
            "path": "data",
            "query": {
              "match_phrase": {
                "data.sTags": {
                  "query": "gentest_v6"
                }
              }
            }
          }
        },
        {
          "match_phrase": {
            "_id": {
              "query": "YzEJZ28B0juYBJ2odwa3"
            }
          }
        }
      ]
    }
  }
}

This is quick, untested advice, so if it doesn't work, please let me know, and I will do some investigation on a 6.8.4 cluster.

-William

[jong99]

I think you're right that it's the nested property that's doing it. I struggled to get the mapping template to do what I wanted and now reading the docs I think nested is not quite what I'm after. Not quite sure why I put it in there to begin with but I think I was trying to fix something else.

I've removed the nested from the mapping and it seems to work at least from a simple test. Thanks for your help!

[William_Brafford]

That makes sense. nested can be a little strange, but I think the docs do a pretty good job of explaining the main case when you'd want to use it. Looking back at your example, you'll probably be fine without it. Good luck moving forward!

-William