# Search with custom analyzer returns no results

**URL:** <https://discuss.elastic.co/t/search-with-custom-analyzer-returns-no-results/156751>\
**Category:** Elasticsearch\
**Created:** [November 14, 2018, 10:31pm UTC](https://discuss.elastic.co/t/search-with-custom-analyzer-returns-no-results/156751 "2018-11-14T22:31:10Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jpresley](https://avatars.discourse-cdn.com/v4/letter/j/b38774/32.png) [@jpresley](https://discuss.elastic.co/u/jpresley)\
**Post date:** [November 14, 2018, 10:31pm UTC](https://discuss.elastic.co/t/search-with-custom-analyzer-returns-no-results/156751/1 "2018-11-14T22:31:11Z")

</div>

We have fix protocol log messages that look like "8=FIX.4.4|9=67|35=5|49=OMNIMKT001|56=GEMINIMKT|34=357821|52=20181109-23:00:59.800|10=056|". We would like to search for strings such as "35=5". To do this, I created a custom analyzer in my proof of concept

```
PUT /tokenizer-test-2018.11.14/
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "tokenizer": "my_tokenizer"
        }
      },
      "tokenizer": {
        "my_tokenizer": {
          "type": "pattern",
          "pattern": "\\|"
        }
      }
    }
  }
}

```

I populated the index with a couple of messages:

```
POST /tokenizer-test-2018.11.14/_doc
{
  "@timestamp": "2018-11-4T11:26:45 -0800",
  "message": "8=FIX.4.4|9=67|35=5|49=OMNIMKT001|56=GEMINIMKT|34=357821|52=20181109-23:00:59.800|10=056|"
}

POST /tokenizer-test-2018.11.14/_doc
{
  "@timestamp": "2018-11-4T11:26:45 -0800",
  "message": "8=FIX.4.4|9=67|35=0|49=OMNIMKT001|56=GEMINIMKT|34=357821|52=20181109-23:00:59.800|10=056|"
}

```

I used the analyze api to test the tokenization occurs as expected, where 35=5 is a token.

However when I search

```
GET /tokenizer-test-2018.11.14/_search
{
  "query": {
    "match": {
      "message": {
        "query": "35=5",
        "analyzer": "my_analyzer"
      }
    }
  }
}

```

I get 0 hits. When I remove the analyzer from the search, I get both the document with 35=0 (which we don't want) and 35=5, which we do want.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 15, 2018, 8:08am UTC](https://discuss.elastic.co/t/search-with-custom-analyzer-returns-no-results/156751/2 "2018-11-15T08:08:30Z")

</div>

It looks like a list of key-value pairs. Why not parse the data into different fields at ingest, e.g. using the Logstash kv filter or an ingest pipeline kv processor? This would allow you to search as well as aggregate over the data.

---

<div class="post-metadata">

**Author:** ![jpresley](https://avatars.discourse-cdn.com/v4/letter/j/b38774/32.png) [@jpresley](https://discuss.elastic.co/u/jpresley)\
**Post date:** [November 15, 2018, 2:44pm UTC](https://discuss.elastic.co/t/search-with-custom-analyzer-returns-no-results/156751/3 "2018-11-15T14:44:00Z")

</div>

That's on our roadmap. We just don't have the resources to do it right now.

Since I asked my question, I was able to get what I needed by creating a template before ingesting the data.

```
PUT /_template/tokenizer-test-template/
{
  "index_patterns": "tokenizer-test-*",
  "mappings": {
    "_doc": {
      "dynamic": "strict",
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "message": {
          "type": "text",
          "analyzer": "my_analyzer"
        }
      }
    }
  },
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "tokenizer": "my_tokenizer"
        }
      },
      "tokenizer": {
        "my_tokenizer": {
          "type": "pattern",
          "pattern": "\\|"
        }
      }
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 13, 2018, 2:44pm UTC](https://discuss.elastic.co/t/search-with-custom-analyzer-returns-no-results/156751/4 "2018-12-13T14:44:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.