# Analyzer producing tokens having position \`-1\`

**URL:** <https://discuss.elastic.co/t/analyzer-producing-tokens-having-position-1/74613>\
**Category:** Elasticsearch\
**Created:** [February 10, 2017, 7:29am UTC](https://discuss.elastic.co/t/analyzer-producing-tokens-having-position-1/74613 "2017-02-10T07:29:01Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pyppe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pyppe/32/3531_2.png) [@Pyppe](https://discuss.elastic.co/u/Pyppe)\
**Post date:** [February 10, 2017, 7:29am UTC](https://discuss.elastic.co/t/analyzer-producing-tokens-having-position-1/74613/1 "2017-02-10T07:29:01Z")

</div>

Hi!

We've got an analyzer using a custom tokenizer:

```auto
"analyzer" : {
  "my_analyzer" : {
    "tokenizer" : "my_custom_tokenizer",
    "filter" : ["lowercase"]
  }
}

```

Custom tokenizer being as follows (in Scala code):

```auto
class MyCustomTokenizer extends Tokenizer {

  val termAtt = addAttribute(classOf[CharTermAttribute])
  val offsetAtt = addAttribute(classOf[OffsetAttribute])

  @volatile var tokens: Iterator[String] = null
  @volatile var startOffset = 0

  override def reset(): Unit = {
    super.reset()
    tokens = null
    startOffset = 0
  }

  override def incrementToken(): Boolean = {
    if (tokens == null) {
      // set it up
      val in = new BufferedReader(input)
      val lines = Iterator.continually(in.readLine).takeWhile(_ != null)
      val inputStr = lines.mkString(" ") // turn lines to whitespace

      tokens = tokenizeIntoIterator(inputStr) // So this is our function that turns text into Iterator[String]
    }

    if (tokens.hasNext) {
      val token = tokens.next
      val endOffset = startOffset + token.length

      termAtt.setEmpty()
      termAtt.append(token)
      offsetAtt.setOffset(startOffset, endOffset)
      startOffset = endOffset

      true
    } else false
  }

}

```

Now, the weird thing is that using this `my_analyzer` we start to get `"position": "-1"` after a while.

At first, it works as expected:

```auto
curl -XGET 'localhost:9200/example/_analyze?pretty=1&analyzer=my_analyzer' -d 'foo bar'
{
  "tokens" : [ {
    "token" : "foo",
    "start_offset" : 0,
    "end_offset" : 3,
    "type" : "word",
    "position" : 0
  }, {
    "token" : "bar",
    "start_offset" : 3,
    "end_offset" : 6,
    "type" : "word",
    "position" : 1
  } ]
}

```

But pretty soon we start to get responses where position is always `-1`.

```auto
curl -XGET 'localhost:9200/example/_analyze?pretty=1&analyzer=my_analyzer' -d 'foo bar'
{
  "tokens" : [ {
    "token" : "foo",
    "start_offset" : 0,
    "end_offset" : 3,
    "type" : "word",
    "position" : -1
  }, {
    "token" : "bar",
    "start_offset" : 3,
    "end_offset" : 6,
    "type" : "word",
    "position" : -1
  } ]
}

```

What is going on here? Do we have some bug in the `MyCustomTokenizer`? What kind of affect does that `-1` have in searches? We're using Elasticsearch `2.3.2`.

Any insights would be greatly appreciated. 🙂

---

<div class="post-metadata">

**Author:** ![Pyppe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pyppe/32/3531_2.png) [@Pyppe](https://discuss.elastic.co/u/Pyppe)\
**Post date:** [February 10, 2017, 7:46am UTC](https://discuss.elastic.co/t/analyzer-producing-tokens-having-position-1/74613/2 "2017-02-10T07:46:21Z")

</div>

Hmmm....  
It seems that setting `PositionIncrementAttribute` to `1` on `reset()` seems to have fixed the behavior:

```auto
override def reset(): Unit = {
  super.reset()
  tokens = null
  startOffset = 0
  Option(getAttribute(classOf[PositionIncrementAttribute])).
    foreach(_.setPositionIncrement(1))
}

```

Go figure... 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 10, 2017, 7:46am UTC](https://discuss.elastic.co/t/analyzer-producing-tokens-having-position-1/74613/3 "2017-03-10T07:46:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.