# How to create tokens from the text data

**URL:** <https://discuss.elastic.co/t/how-to-create-tokens-from-the-text-data/212824>\
**Category:** Elasticsearch\
**Created:** [December 23, 2019, 8:55am UTC](https://discuss.elastic.co/t/how-to-create-tokens-from-the-text-data/212824 "2019-12-23T08:55:41Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pratik\_Hulji](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pratik_hulji/32/59612_2.png) [@Pratik\_Hulji](https://discuss.elastic.co/u/Pratik_Hulji)\
**Post date:** [December 23, 2019, 8:55am UTC](https://discuss.elastic.co/t/how-to-create-tokens-from-the-text-data/212824/1 "2019-12-23T08:55:41Z")

</div>

Hi guys, I have converted the pdf into base64 format and now i want to convert the text data into tokens. Can you please guide me.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 23, 2019, 9:12am UTC](https://discuss.elastic.co/t/how-to-create-tokens-from-the-text-data/212824/2 "2019-12-23T09:12:03Z")

</div>

What is the use case?

---

<div class="post-metadata">

**Author:** ![Pratik\_Hulji](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pratik_hulji/32/59612_2.png) [@Pratik\_Hulji](https://discuss.elastic.co/u/Pratik_Hulji)\
**Post date:** [December 23, 2019, 9:15am UTC](https://discuss.elastic.co/t/how-to-create-tokens-from-the-text-data/212824/3 "2019-12-23T09:15:09Z")

</div>

I have to extract the data from the resume

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 23, 2019, 9:27am UTC](https://discuss.elastic.co/t/how-to-create-tokens-from-the-text-data/212824/4 "2019-12-23T09:27:18Z")

</div>

What do you want to do with the extracted data?

---

<div class="post-metadata">

**Author:** ![Pratik\_Hulji](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pratik_hulji/32/59612_2.png) [@Pratik\_Hulji](https://discuss.elastic.co/u/Pratik_Hulji)\
**Post date:** [December 23, 2019, 9:31am UTC](https://discuss.elastic.co/t/how-to-create-tokens-from-the-text-data/212824/5 "2019-12-23T09:31:33Z")

</div>

This is the small task assigned to me. Further it will be used for analysis by the other team.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 23, 2019, 1:46pm UTC](https://discuss.elastic.co/t/how-to-create-tokens-from-the-text-data/212824/6 "2019-12-23T13:46:39Z")

</div>

What kind of analysis?

Anyway, you can use the \_analyze API and use its output. With that you might be able to do the rest.

---

<div class="post-metadata">

**Author:** ![Pratik\_Hulji](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pratik_hulji/32/59612_2.png) [@Pratik\_Hulji](https://discuss.elastic.co/u/Pratik_Hulji)\
**Post date:** [December 24, 2019, 5:51am UTC](https://discuss.elastic.co/t/how-to-create-tokens-from-the-text-data/212824/7 "2019-12-24T05:51:28Z")

</div>

Yes, I used \_analyze it takes this two parameters "tokenizer": "", "text": "".  
But my approach is different.

1. Convert pdf into base64 to read the text data present in pdf.
2. Filter the stop words and convert the text data into tokens.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 24, 2019, 6:12am UTC](https://discuss.elastic.co/t/how-to-create-tokens-from-the-text-data/212824/8 "2019-12-24T06:12:54Z")

</div>

Yes. That's what I wrote. Use the \_analyze API to do step 2.

---

<div class="post-metadata">

**Author:** ![Pratik\_Hulji](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pratik_hulji/32/59612_2.png) [@Pratik\_Hulji](https://discuss.elastic.co/u/Pratik_Hulji)\
**Post date:** [December 24, 2019, 6:19am UTC](https://discuss.elastic.co/t/how-to-create-tokens-from-the-text-data/212824/9 "2019-12-24T06:19:11Z")

</div>

Okay Thank you

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 21, 2020, 6:19am UTC](https://discuss.elastic.co/t/how-to-create-tokens-from-the-text-data/212824/10 "2020-01-21T06:19:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.