Field \"message\" was indexed without position data; cannot run PhraseQuery

Elastic Stack Elasticsearch
1

Hi,

I have an ELK configuration, which works pretty good, however, when I do some queries like this:

message:"text with spaces"

I receive the next error:

field \"message\" was indexed without position data; cannot run PhraseQuery

These kind of queries would not raise an errors:

message:Storing the user information
OR
message:"login_ok"

Here is the mapping:

          "message": {
                "store": true,
                "type": "string"
            },

Any thoughts?

Regards,

2

You need to enable term positions, which are disabled by default:

https://www.elastic.co/guide/en/elasticsearch/reference/current/term-vector.html

If you enable positions, then term frequencies are also enabled, which is
generally the desired behavior, but sometimes not.

Ivan

3

What @Ivan suggests will work, but it will be a storage and performance hit to Elasticsearch if you change that.

Are you trying to search for exact text? Or just certain words?

If you have not overridden the default template management, or indexed to names other than logstash-YYYY.MM.DD, then you can do exact text searches with the message.raw field.

4

@ivan Thanks, updated the template.

@theuntergeek Thanks for the hint, however my template is updated and *.raw is not there anymore. As amount of data is not big, I would go with indexing the positions and offsets.

Regards,

5

@Ivan

Should it take effect immediately, after the new index would be created?

My current mapping is the next:

 "message" : {
            "term_vector" : "with_positions_offsets",
            "index" : "analyzed",
            "store" : true,
            "type" : "string"
          }

However, when I do the same search using Kibana, within the timeframe "last 15 minutes", I receive the same error:
1.

Courier Fetch: 20 of 920 shards failed.

"reason": "QueryPhaseExecutionException[[logstash-2015.11.06][5]: query[filtered((message:\"starting the new registration process\"))->BooleanFilter(+cache(@timestamp:[1446801679498 TO 1446802579498]))],from[0],size[2500],sort[<custom:\"@timestamp\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@761db3ae>!]: Query Failed [Failed to execute main query]]; nested: IllegalStateException[field \"message\" was indexed without position data; cannot run PhraseQuery (term=starting)]; "

Should it work? Or it will start to work only when the old "indexes" would be removed?

Regards,

6

The next new index is necessary. The mapping for message on the current index is already fixed.

7

Sorry, still didn't got it.

I have updated the mapping yesterday, and tried to search "today" - within the interval of 15 minutes, so, in theory it should search by the new mapping. However, the search was not done.

The index is already "new", but the error is the same.

8

I am not a Kibana user, can you try executing the query directly against
the cluster? Are you using time based indices and is the query against all
indices?

Double check the mapping using the API, not by what you think you might
have used.

Ivan

9

There's a difference between term vectors with positions and just enabling term frequencies and positions.

ie setting

positions: false

in your mapping would make phrase queries not work.

Term vectors are another, unrelated data structure that keep an uninverted (ie regular looking) view of the documents used in various advanced applications of Elasticsearch.

10

Sorry, I'm not quite sure that I understood.

Could you say, in particular, should this mapping:

Solve the

field \"message\" was indexed without position data;

issue? Or I should put the term_vector option to the different value?