# Inconsistent behaviour of the UAX URL email tokenizer?

**URL:** <https://discuss.elastic.co/t/inconsistent-behaviour-of-the-uax-url-email-tokenizer/267493>\
**Category:** Elasticsearch\
**Created:** [March 17, 2021, 11:59am UTC](https://discuss.elastic.co/t/inconsistent-behaviour-of-the-uax-url-email-tokenizer/267493 "2021-03-17T11:59:05Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![jakopako](https://avatars.discourse-cdn.com/v4/letter/j/9de053/32.png) [@jakopako](https://discuss.elastic.co/u/jakopako)\
**Post date:** [March 17, 2021, 11:59am UTC](https://discuss.elastic.co/t/inconsistent-behaviour-of-the-uax-url-email-tokenizer/267493/1 "2021-03-17T11:59:05Z")

</div>

The following two queries extract urls in different ways and I am wondering whether this is a bug. In the first case the ' following the url is part of the extracted token whereas in the second case it is not. This seems to depend on whether the url ends with a path or with search parameters or more specifically, whether the url contains a ? or not. This has been tested with version 7.11.2 in the Dev Tools section of Kibana.

**First case:**

```auto
GET /_analyze
{
  "tokenizer" : "uax_url_email",
  "text" : "src='http://example.com/?id=100'"
}

```

Response:

```auto
{
  "tokens" : [
    {
      "token" : "src",
      "start_offset" : 0,
      "end_offset" : 3,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "http://example.com/?id=100'",
      "start_offset" : 5,
      "end_offset" : 32,
      "type" : "<URL>",
      "position" : 1
    }
  ]
}

```

**Second case:**

```auto
GET /_analyze
{
  "tokenizer" : "uax_url_email",
  "text" : "src='http://example.com/path/to/script.js'"
}

```

Response:

```auto
{
  "tokens" : [
    {
      "token" : "src",
      "start_offset" : 0,
      "end_offset" : 3,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "http://example.com/path/to/script.js",
      "start_offset" : 5,
      "end_offset" : 41,
      "type" : "<URL>",
      "position" : 1
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 14, 2021, 11:59am UTC](https://discuss.elastic.co/t/inconsistent-behaviour-of-the-uax-url-email-tokenizer/267493/2 "2021-04-14T11:59:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.