# Prevent thai tokenizer from tokenizing hashtag

**URL:** <https://discuss.elastic.co/t/prevent-thai-tokenizer-from-tokenizing-hashtag/320225>\
**Category:** Elasticsearch\
**Created:** [December 1, 2022, 8:19am UTC](https://discuss.elastic.co/t/prevent-thai-tokenizer-from-tokenizing-hashtag/320225 "2022-12-01T08:19:03Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![TFP](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tfp/32/113843_2.png) [@TFP](https://discuss.elastic.co/u/TFP)\
**Post date:** [December 1, 2022, 8:19am UTC](https://discuss.elastic.co/t/prevent-thai-tokenizer-from-tokenizing-hashtag/320225/1 "2022-12-01T08:19:03Z")

</div>

I'm currently using the thai tokenizer and wanted to preserve hashtag words but the tokenizer keeps on removing the hashtag symbol.

ES version: 7.16.2

```auto
GET /_analyze
{
	"tokenizer": "thai",
	"text": "#รายการพ #hashtag"
}

```

Response

```auto
{
	"tokens": [
		{
			"token": "รายการ",
			"start_offset": 1,
			"end_offset": 7,
			"type": "word",
			"position": 0
		},
		{
			"token": "พ",
			"start_offset": 7,
			"end_offset": 8,
			"type": "word",
			"position": 1
		},
		{
			"token": "hashtag",
			"start_offset": 10,
			"end_offset": 17,
			"type": "word",
			"position": 2
		}
	]
}

```

I expect it to be like

```auto
{
	"tokens": [
		{
			"token": "#รายการพ",
			"start_offset": 1,
			"end_offset": 7,
			"type": "word",
			"position": 0
		},
		{
			"token": "#hashtag",
			"start_offset": 10,
			"end_offset": 17,
			"type": "word",
			"position": 2
		}
	]
}

```

I tried using char\_filter to replace all hashtags with some placeholder and replace it on filter to hashtag symbol again. But the result is not what expected.

```auto
{
	"tokenizer": "thai",
	"filter": [
		{
			"pattern": "hashtagplaceholder([^\\s*]+)",
			"type": "pattern_replace",
			"replacement": "#$1"
		}
	],
	"char_filter": [
		{
			"pattern": "#([^\\s*]+)",
			"type": "pattern_replace",
			"replacement": "hashtagplaceholder$1"
		}
	],
	"text": "#รายการพ #hashtag"
}

```

Response:

```auto
{
	"tokens": [
		{
			"token": "hashtagplaceholder",
			"start_offset": 0,
			"end_offset": 7,
			"type": "word",
			"position": 0
		},
		{
			"token": "รายการ",
			"start_offset": 7,
			"end_offset": 7,
			"type": "word",
			"position": 1
		},
		{
			"token": "พ",
			"start_offset": 7,
			"end_offset": 8,
			"type": "word",
			"position": 2
		},
		{
			"token": "#hashtag",
			"start_offset": 9,
			"end_offset": 17,
			"type": "word",
			"position": 3
		}
	]
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 29, 2022, 8:19am UTC](https://discuss.elastic.co/t/prevent-thai-tokenizer-from-tokenizing-hashtag/320225/2 "2022-12-29T08:19:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.