# Extract Hashtags and Mentions into separate fields **URL:** \ **Category:** Elasticsearch\ **Created:** [December 13, 2021, 1:34pm UTC](https://discuss.elastic.co/t/extract-hashtags-and-mentions-into-separate-fields/291656 "2021-12-13T13:34:54Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![prakhar\_nigam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prakhar_nigam/32/98826_2.png) [@prakhar\_nigam](https://discuss.elastic.co/u/prakhar_nigam)\ **Post date:** [December 13, 2021, 1:34pm UTC](https://discuss.elastic.co/t/extract-hashtags-and-mentions-into-separate-fields/291656/1 "2021-12-13T13:34:54Z")
I am doing a DIY Tweet Sentiment analyser, I have an index of tweets like these, ```auto "_source" : { "id" : 26930655, "status" : 1, "title" : "Hereโ€™s 5 underrated #BTC and realistic crypto accounts that everyone should follow: @Quinnvestments , @JacobOracle , @jevauniedaye , @ginsbergonomics , @InspoCrypto", "hashtags" : null, "created_at" : 1622390229, "category" : null, "language" : 50 }, { "id" : 22521897, "status" : 1, "title" : "#bulls gonna overtake the #bears soon #ATH coming #ALTSEASON #BSCGem #eth #btc #memecoin #100xgems #satyasanatan ๐Ÿ™๐Ÿšฉ๐Ÿšฉ๐Ÿ‡ฎ๐Ÿ‡ณ""", "hashtags" : null, "created_at" : 1620045296, "category" : null, "language" : 50 } ``` There Mappings and settings are like ```auto "sentiment-en" : { "mappings" : { "properties" : { "category" : { "type" : "text" }, "created_at" : { "type" : "integer" }, "hashtags" : { "type" : "text" }, "id" : { "type" : "long" }, "language" : { "type" : "integer" }, "status" : { "type" : "integer" }, "title" : { "type" : "text", "fields" : { "raw" : { "type" : "keyword" }, "raw_text" : { "type" : "text" }, "stop" : { "type" : "text", "index_options" : "docs", "analyzer" : "stop_words_filter" }, "syn" : { "type" : "text", "index_options" : "docs", "analyzer" : "synonyms_filter" } }, "index_options" : "docs", "analyzer" : "all_ok_filter" } } } } } "settings" : { "index" : { "number_of_shards" : "10", "provided_name" : "sentiment-en", "creation_date" : "1627975717560", "analysis" : { "filter" : { "stop_words" : { "type" : "stop", "stopwords" : [] }, "synonyms" : { "type" : "synonym", "synonyms" : [] } }, "analyzer" : { "stop_words_filter" : { "filter" : ["stop_words"], "tokenizer" : "standard" }, "synonyms_filter" : { "filter" : ["synonyms"], "tokenizer" : "standard" }, "all_ok_filter" : { "filter" : ["stop_words", "synonyms"], "tokenizer" : "standard" } } }, "number_of_replicas" : "0", "uuid" : "Q5yDYEXHSM-5kvyLGgsYYg", "version" : { "created" : "7090199" } } ``` Now the problem is i want to extract all the Hashtags and mentions in a seprate field. What i want as O/P ```auto "id" : 26930655, "status" : 1, "title" : "Hereโ€™s 5 underrated #BTC and realistic crypto accounts that everyone should follow: @Quinnvestments , @JacobOracle , @jevauniedaye , @ginsbergonomics , @InspoCrypto", "hashtags" : BTC, "created_at" : 1622390229, "category" : null, "language" : 50 }, { "id" : 22521897, "status" : 1, "title" : "#bulls gonna overtake the #bears soon #ATH coming #ALTSEASON #BSCGem #eth #btc #memecoin #100xgems #satyasanatan ๐Ÿ™๐Ÿšฉ๐Ÿšฉ๐Ÿ‡ฎ๐Ÿ‡ณ""", "hashtags" : bulls,bears,ATH, ALTSEASON, BSCGem, eth , btc, memecoin, 100xGem, satyasanatan "created_at" : 1620045296, "category" : null, "language" : 50 } ``` I wanted to try reindex API but ruby codes not works there and i can not use ruby script in filter plugin as mentioned in [blog](https://discuss.elastic.co/t/isolating-hashtags-in-twitter-feed/125532). or is there any tokenizer which can be handy in this case. I have a limitation that i can not use logstash here i am inserting data using API calls ---
**Author:** ![cbuescher](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cbuescher/32/60402_2.png) [@cbuescher](https://discuss.elastic.co/u/cbuescher)\ **Post date:** [December 14, 2021, 10:03am UTC](https://discuss.elastic.co/t/extract-hashtags-and-mentions-into-separate-fields/291656/2 "2021-12-14T10:03:10Z")
Try using two separate analyzers for the hashtag / mentions fields. Both would probably need to tokenize on whitespace, but then you can add token filters (e.g. maybe the [conditional filter](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-condition-tokenfilter.html)) that only let the hashtag or mention through and discard the rest. Hope that helps. ---
**Author:** ![prakhar\_nigam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prakhar_nigam/32/98826_2.png) [@prakhar\_nigam](https://discuss.elastic.co/u/prakhar_nigam)\ **Post date:** [December 16, 2021, 6:48am UTC](https://discuss.elastic.co/t/extract-hashtags-and-mentions-into-separate-fields/291656/3 "2021-12-16T06:48:50Z")
Thanks Christoph for your answer and it seems like a very feasible approach. I am very newbie to painless script i tried to write something like ths to extract hashtags,which results in all the tokens. Can you point me in the right direction ```auto GET /_analyze { "tokenizer": "whitespace", "filter": [ { "type": "condition", "filter": ["lowercase"], "script": { "source": "token.getTerm().toString().startsWith('#')" } } ], "text": "Crypto bull run in #BTC #ETH I was ๐Ÿค“ damn #ICL2020#ICL" } Error : "caused_by" : { "type" : "illegal_argument_exception", "reason" : "method [java.lang.CharSequence, indexOf/1] not found" } ``` `` Can i use Java Patterns Something like this in painless script to filter words `TAG_PATTERN = Pattern.compile("(?:^|\\s|[\\p{Punct}&&[^/]])(#[\\p{L}0-9-_]+)")` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 13, 2022, 6:49am UTC](https://discuss.elastic.co/t/extract-hashtags-and-mentions-into-separate-fields/291656/4 "2022-01-13T06:49:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.