# Index pre-analyzed text by sending the actual terms/tokens?
**URL:** https://discuss.elastic.co/t/index-pre-analyzed-text-by-sending-the-actual-terms-tokens/254989
**Category:** Elasticsearch
**Created:** [November 10, 2020, 11:03pm UTC](https://discuss.elastic.co/t/index-pre-analyzed-text-by-sending-the-actual-terms-tokens/254989 "2020-11-10T23:03:49Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@johann-petrak](https://discuss.elastic.co/u/johann-petrak)
#### Post date: [November 10, 2020, 11:03pm UTC](https://discuss.elastic.co/t/index-pre-analyzed-text-by-sending-the-actual-terms-tokens/254989/1 "2020-11-10T23:03:49Z")
Is it possible to somehow pass on an already pre-analyzed stream of terms/tokens to an index for a field?
I already have my text broken up into tokens, I have the offsets, I can produce everything an analyzer could produce, but better. Is there a way to pass this on to the index somehow?
---
### Author:  [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)
#### Post date: [November 12, 2020, 11:23am UTC](https://discuss.elastic.co/t/index-pre-analyzed-text-by-sending-the-actual-terms-tokens/254989/2 "2020-11-12T11:23:19Z")
Hi Johann,
We don't have a raw token stream e.g. JSON format for tokens but we do allow for "clever" external text processors to add arbitrary "annotations" that overlay on the tokens elasticsearch produces.
See the [annotated text](https://www.elastic.co/blog/search-for-things-not-strings-with-the-annotated-text-plugin) plugin.
---
### Author:  [@johann-petrak](https://discuss.elastic.co/u/johann-petrak)
#### Post date: [November 12, 2020, 11:42am UTC](https://discuss.elastic.co/t/index-pre-analyzed-text-by-sending-the-actual-terms-tokens/254989/3 "2020-11-12T11:42:27Z")
Thank you for that info which is a bit disappointing because just adding the tokens I have would be zero effort. When I had a look at the annotated text plugin it looks like it would be a huge effort to understand how to implement my own plugin for doing what I want and integrate it correctly.
It would really be a huge help if there was a plugin that would essentially support something like a "rawtoken" type of field which could then contain an array of tokens as the annotated text plugin produces.
Are there resources, documentation or help for how to implement such a plugin?
---
### Author:  [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)
#### Post date: [November 12, 2020, 11:52am UTC](https://discuss.elastic.co/t/index-pre-analyzed-text-by-sending-the-actual-terms-tokens/254989/4 "2020-11-12T11:52:43Z")
I imagine JSON would be a pretty verbose way of passing a tokenized stream, especially with the offset and position information.
You then have to worry about elasticsearch functions that rely on re-tokenizing from the original text - they assume the analysis functions are "in the box" and not something you have to consult external processes for.
The most obvious need for an analyzer is in tokenizing search terms the user provides but others include some highlighters and the significant\_text aggregation.
It would be easier adding a Java Analyzer that reproduces the functionality you need.
Out of interest - what functionality do you feel is missing that requires external processing?
---
### Author:  [@johann-petrak](https://discuss.elastic.co/u/johann-petrak)
#### Post date: [November 12, 2020, 1:46pm UTC](https://discuss.elastic.co/t/index-pre-analyzed-text-by-sending-the-actual-terms-tokens/254989/5 "2020-11-12T13:46:01Z")
This is about two separate issues really:
1. which process does the tokenization of documents when during the processing of data
I am working on NLP, so I already have a lot of code and infrastructure to do tokenization, entity recognition, linking, disambiguation etc. This is code often implemented in non-Java languages like Python e.g. the output of a Neural Net. The processing is done in a completely separate job from when I want to actually store the data in ES.
1. what should the token stream look like for some text
I want to be able to use the tokens from my processing in 1) but also be able to add arbitrary overlapping token sequences - similar to what the annotated text plugin does, but for more realistic NLP annotations, where e.g. a protein name and a gene name can both overlap in various ways, or a movie title and a person name could overlap etc. The annotated text plugin is not flexible enough for this.
Since I want to be able to query for these entities in position as it is possible with the annotated text plugin, I cannot just use separate keyword fields for that.
For the query phase, it would be much easier to create a normal custom analyzer to tokenize the query, so for such a field where I would like to be able to send just the token stream, it would still be useful to be able to specify a custom analyzer for the query phase.
Highlighting is not too important for this usecase as all querying and processing of the query result would be done by a program which would then be able to process the results accordingly.
But anyway: how would one best get started understanding everything that is necessary to implement an analyzer? Is there a tutorial or similar for that somewhere?
---
### Author:  [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)
#### Post date: [November 12, 2020, 2:17pm UTC](https://discuss.elastic.co/t/index-pre-analyzed-text-by-sending-the-actual-terms-tokens/254989/6 "2020-11-12T14:17:52Z")
> [@johann-petrak](#):
>
> The annotated text plugin is not flexible enough for this.
A piece of text can have multiple annotations e.g
`[Death of Stalin](Movie&Person&Event) was released...`
If you're able to resolve overlapping tokens to the longest text span you can bundle them as above to describe the text span. Not ideal but anything else complicates the markup syntax.
> [@johann-petrak](#):
>
> how would one best get started understanding everything that is necessary to implement an analyzer? Is there a tutorial or similar for that somewhere?
I suspect the easiest thing to do is copy one of the [existing Analysis plugin'](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis.html)s source code as a template.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 10, 2020, 2:18pm UTC](https://discuss.elastic.co/t/index-pre-analyzed-text-by-sending-the-actual-terms-tokens/254989/7 "2020-12-10T14:18:06Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.