# Analyzer disappears after server restart (includes gist)
**URL:** https://discuss.elastic.co/t/analyzer-disappears-after-server-restart-includes-gist/8840
**Category:** Elasticsearch
**Created:** [August 23, 2012, 10:24pm UTC](https://discuss.elastic.co/t/analyzer-disappears-after-server-restart-includes-gist/8840 "2012-08-23T22:24:20Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@Kevin\_Lawrence](https://discuss.elastic.co/u/Kevin_Lawrence)
#### Post date: [August 23, 2012, 10:24pm UTC](https://discuss.elastic.co/t/analyzer-disappears-after-server-restart-includes-gist/8840/1 "2012-08-23T22:24:20Z")
I have an analyzer:
"analyzer" : {
"lowercase\_keyword" : {
"type" : "custom",
"tokenizer" : "keyword",
"filter" : ["lowercase", "trim"]
}
}
that is referenced in a mapping:
"location\_countries" : {
"properties" : {
"country" : {
"type" : "string",
"analyzer" : "lowercase\_keyword"
}
}
}
And when I use the 'country' field in a filter or a facet, the field is
(correctly) treated as a keyword.
curl -XGET 'localhost:9200/clinical\_trials/\_search?pretty=true' -d '
{
"query" : {
"term" : { "brief\_title" : "dermatitis" }
},
"filter" : {
"term" : { "country" : "united states" }
},
"facets" : {
"tag" : {
"terms" : { "field" : "country" }
}
}
}
'
facet results:
"facets" : {
"tag" : {
"\_type" : "terms",
"missing" : 0,
"total" : 1,
"other" : 0,
"terms" : [ {
"term" : "united states",
"count" : 1
} ]
}
Everything works fine until the machine gets rebooted or the Elastic
Search service gets restarted. After a restart, all my filters stop working
as if the analyzer does not exist.
The same query against the same data results in:
"facets" : {
"tag" : {
"\_type" : "terms",
"missing" : 0,
"total" : 2,
"other" : 0,
"terms" : [ {
"term" : "united",
"count" : 1
}, {
"term" : "states",
"count" : 1
} ]
}
If I query the \_settings/\_mappings of my index, the analyzer and mappings
are still defined correctly but the analyzer seems to have no effect.
What am I doing wrong?
Thanks in advance,
Kevin
git://gist.github.com/3442562.git
--
---
### Author:  [@Kevin\_Lawrence](https://discuss.elastic.co/u/Kevin_Lawrence)
#### Post date: [August 23, 2012, 10:27pm UTC](https://discuss.elastic.co/t/analyzer-disappears-after-server-restart-includes-gist/8840/2 "2012-08-23T22:27:55Z")
I get the same behavior in multiple versions of ES, including today's
0.19.9.
Kevin
On Thursday, August 23, 2012 3:24:20 PM UTC-7, Kevin Lawrence wrote:
> I have an analyzer:
>
> "analyzer" : {
> "lowercase\_keyword" : {
> "type" : "custom",
> "tokenizer" : "keyword",
> "filter" : ["lowercase", "trim"]
> }
> }
>
> that is referenced in a mapping:
>
> "location\_countries" : {
> "properties" : {
> "country" : {
> "type" : "string",
> "analyzer" : "lowercase\_keyword"
> }
> }
> }
>
> And when I use the 'country' field in a filter or a facet, the field is
> (correctly) treated as a keyword.
>
> curl -XGET 'localhost:9200/clinical\_trials/\_search?pretty=true' -d '
> {
> "query" : {
> "term" : { "brief\_title" : "dermatitis" }
> },
> "filter" : {
> "term" : { "country" : "united states" }
> },
> "facets" : {
> "tag" : {
> "terms" : { "field" : "country" }
> }
> }
> }
> '
>
> facet results:
>
> "facets" : {
> "tag" : {
> "\_type" : "terms",
> "missing" : 0,
> "total" : 1,
> "other" : 0,
> "terms" : [ {
> "term" : "united states",
> "count" : 1
> } ]
> }
>
> Everything works fine until the machine gets rebooted or the Elastic
> Search service gets restarted. After a restart, all my filters stop working
> as if the analyzer does not exist.
>
> The same query against the same data results in:
> "facets" : {
> "tag" : {
> "\_type" : "terms",
> "missing" : 0,
> "total" : 2,
> "other" : 0,
> "terms" : [ {
> "term" : "united",
> "count" : 1
> }, {
> "term" : "states",
> "count" : 1
> } ]
> }
>
> If I query the \_settings/\_mappings of my index, the analyzer and mappings
> are still defined correctly but the analyzer seems to have no effect.
>
> What am I doing wrong?
>
> Thanks in advance,
>
> Kevin
>
> git://gist.github.com/3442562.git
--
---
### Author:  [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)
#### Post date: [August 24, 2012, 6:34am UTC](https://discuss.elastic.co/t/analyzer-disappears-after-server-restart-includes-gist/8840/3 "2012-08-24T06:34:59Z")
Hi Kevin
As I said yesterday, please gist a COMPLETE curl recreation, ie
something that we can copy and paste to make it work, from index
creation onwards. Snippets of information as below are insufficient.
Also, please gist your elasticsearch config, as that may have some
impact.
ta
clint
On Thu, 2012-08-23 at 15:27 -0700, Kevin Lawrence wrote:
> I get the same behavior in multiple versions of ES, including today's
> 0.19.9.
>
> Kevin
>
> On Thursday, August 23, 2012 3:24:20 PM UTC-7, Kevin Lawrence wrote:
> I have an analyzer:
>
> ```
> "analyzer" : {
> "lowercase_keyword" : {
> "type" : "custom",
> "tokenizer" : "keyword",
> "filter" : ["lowercase", "trim"]
> }
> }
>
>
> that is referenced in a mapping:
>
>
> "location_countries" : {
> "properties" : {
> "country" : {
> "type" : "string",
> "analyzer" : "lowercase_keyword"
> }
> }
> }
>
>
> And when I use the 'country' field in a filter or a facet, the
> field is (correctly) treated as a keyword.
>
>
> curl -XGET
> 'localhost:9200/clinical_trials/_search?pretty=true' -d '
> {
> "query" : {
> "term" : { "brief_title" : "dermatitis" }
> },
> "filter" : {
> "term" : { "country" : "united states" }
> },
> "facets" : {
> "tag" : {
> "terms" : { "field" : "country" }
> }
> }
> }
> '
>
>
>
> facet results:
>
>
> "facets" : {
> "tag" : {
> "_type" : "terms",
> "missing" : 0,
> "total" : 1,
> "other" : 0,
> "terms" : [ {
> "term" : "united states",
> "count" : 1
> } ]
> }
>
>
>
>
> Everything works fine until the machine gets rebooted or the
> Elastic Search service gets restarted. After a restart, all my
> filters stop working as if the analyzer does not exist.
>
>
> The same query against the same data results in:
> "facets" : {
> "tag" : {
> "_type" : "terms",
> "missing" : 0,
> "total" : 2,
> "other" : 0,
> "terms" : [ {
> "term" : "united",
> "count" : 1
> }, {
> "term" : "states",
> "count" : 1
> } ]
> }
>
>
>
>
> If I query the _settings/_mappings of my index, the analyzer
> and mappings are still defined correctly but the analyzer
> seems to have no effect.
>
> What am I doing wrong?
>
> Thanks in advance,
>
> Kevin
>
>
>
> git://gist.github.com/3442562.git
>
> ```
>
> --
--
---
### Author:  [@Kevin\_Lawrence](https://discuss.elastic.co/u/Kevin_Lawrence)
#### Post date: [August 24, 2012, 4:57pm UTC](https://discuss.elastic.co/t/analyzer-disappears-after-server-restart-includes-gist/8840/4 "2012-08-24T16:57:28Z")
Thanks Clint,
I added my config file to the gist at:
>
>
> There are more than three files. show original
but the only things changed from the default are path.data and cluster.name.
Kevin
On Thursday, August 23, 2012 11:34:59 PM UTC-7, Clinton Gormley wrote:
> Hi Kevin
>
> As I said yesterday, please gist a COMPLETE curl recreation, ie
> something that we can copy and paste to make it work, from index
> creation onwards. Snippets of information as below are insufficient.
>
> Also, please gist your elasticsearch config, as that may have some
> impact.
>
> ta
>
> clint
>
> On Thu, 2012-08-23 at 15:27 -0700, Kevin Lawrence wrote:
>
> > I get the same behavior in multiple versions of ES, including today's
> > 0.19.9.
> >
> > Kevin
> >
> > On Thursday, August 23, 2012 3:24:20 PM UTC-7, Kevin Lawrence wrote:
> > I have an analyzer:
> >
> > ```
> > "analyzer" : {
> > "lowercase_keyword" : {
> > "type" : "custom",
> > "tokenizer" : "keyword",
> > "filter" : ["lowercase", "trim"]
> > }
> > }
> >
> >
> > that is referenced in a mapping:
> >
> >
> > "location_countries" : {
> > "properties" : {
> > "country" : {
> > "type" : "string",
> > "analyzer" : "lowercase_keyword"
> > }
> > }
> > }
> >
> >
> > And when I use the 'country' field in a filter or a facet, the
> > field is (correctly) treated as a keyword.
> >
> >
> > curl -XGET
> > 'localhost:9200/clinical_trials/_search?pretty=true' -d '
> > {
> > "query" : {
> > "term" : { "brief_title" : "dermatitis" }
> > },
> > "filter" : {
> > "term" : { "country" : "united states" }
> > },
> > "facets" : {
> > "tag" : {
> > "terms" : { "field" : "country" }
> > }
> > }
> > }
> > '
> >
> >
> >
> > facet results:
> >
> >
> > "facets" : {
> > "tag" : {
> > "_type" : "terms",
> > "missing" : 0,
> > "total" : 1,
> > "other" : 0,
> > "terms" : [ {
> > "term" : "united states",
> > "count" : 1
> > } ]
> > }
> >
> >
> >
> >
> > Everything works fine until the machine gets rebooted or the
> > Elastic Search service gets restarted. After a restart, all my
> > filters stop working as if the analyzer does not exist.
> >
> >
> > The same query against the same data results in:
> > "facets" : {
> > "tag" : {
> > "_type" : "terms",
> > "missing" : 0,
> > "total" : 2,
> > "other" : 0,
> > "terms" : [ {
> > "term" : "united",
> > "count" : 1
> > }, {
> > "term" : "states",
> > "count" : 1
> > } ]
> > }
> >
> >
> >
> >
> > If I query the _settings/_mappings of my index, the analyzer
> > and mappings are still defined correctly but the analyzer
> > seems to have no effect.
> >
> > What am I doing wrong?
> >
> > Thanks in advance,
> >
> > Kevin
> >
> >
> >
> > git://gist.github.com/3442562.git
> >
> > ```
> >
> > --
--
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [August 24, 2012, 6:37pm UTC](https://discuss.elastic.co/t/analyzer-disappears-after-server-restart-includes-gist/8840/5 "2012-08-24T18:37:56Z")
It appears that issue is that an analyzer that is created as part of a
create index request is not being saved. I am not familiar with the
index metadata, but it appears that analyzers are not persisted and
are re-read entirely from the yml file on startup. Haven't tried your
gist, but it seems correct.
The immediate workaround would be to define the analyzer in
elasticseach.yml. Persisting analyzer information to the index
metadata should be supported IMHO. I would open an issue and see what
Shay thinks.
Ivan
On Fri, Aug 24, 2012 at 9:57 AM, Kevin Lawrence [kevin@diamond-sky.com](mailto:kevin@diamond-sky.com) wrote:
> Thanks Clint,
>
> I added my config file to the gist at:
>
> [ElasticSearch forgets my analyzer · GitHub](https://gist.github.com/3442562)
>
> but the only things changed from the default are path.data and cluster.name.
>
> Kevin
>
> On Thursday, August 23, 2012 11:34:59 PM UTC-7, Clinton Gormley wrote:
>
> > Hi Kevin
> >
> > As I said yesterday, please gist a COMPLETE curl recreation, ie
> > something that we can copy and paste to make it work, from index
> > creation onwards. Snippets of information as below are insufficient.
> >
> > Also, please gist your elasticsearch config, as that may have some
> > impact.
> >
> > ta
> >
> > clint
> >
> > On Thu, 2012-08-23 at 15:27 -0700, Kevin Lawrence wrote:
> >
> > > I get the same behavior in multiple versions of ES, including today's
> > > 0.19.9.
> > >
> > > Kevin
> > >
> > > On Thursday, August 23, 2012 3:24:20 PM UTC-7, Kevin Lawrence wrote:
> > > I have an analyzer:
> > >
> > > ```
> > > "analyzer" : {
> > > "lowercase_keyword" : {
> > > "type" : "custom",
> > > "tokenizer" : "keyword",
> > > "filter" : ["lowercase", "trim"]
> > > }
> > > }
> > >
> > > that is referenced in a mapping:
> > >
> > > "location_countries" : {
> > > "properties" : {
> > > "country" : {
> > > "type" : "string",
> > > "analyzer" : "lowercase_keyword"
> > > }
> > > }
> > > }
> > >
> > > And when I use the 'country' field in a filter or a facet, the
> > > field is (correctly) treated as a keyword.
> > >
> > > curl -XGET
> > > 'localhost:9200/clinical_trials/_search?pretty=true' -d '
> > > {
> > > "query" : {
> > > "term" : { "brief_title" : "dermatitis" }
> > > },
> > > "filter" : {
> > > "term" : { "country" : "united states" }
> > > },
> > > "facets" : {
> > > "tag" : {
> > > "terms" : { "field" : "country" }
> > > }
> > > }
> > > }
> > > '
> > >
> > > facet results:
> > >
> > > "facets" : {
> > > "tag" : {
> > > "_type" : "terms",
> > > "missing" : 0,
> > > "total" : 1,
> > > "other" : 0,
> > > "terms" : [ {
> > > "term" : "united states",
> > > "count" : 1
> > > } ]
> > > }
> > >
> > > Everything works fine until the machine gets rebooted or the
> > > Elastic Search service gets restarted. After a restart, all my
> > > filters stop working as if the analyzer does not exist.
> > >
> > > The same query against the same data results in:
> > > "facets" : {
> > > "tag" : {
> > > "_type" : "terms",
> > > "missing" : 0,
> > > "total" : 2,
> > > "other" : 0,
> > > "terms" : [ {
> > > "term" : "united",
> > > "count" : 1
> > > }, {
> > > "term" : "states",
> > > "count" : 1
> > > } ]
> > > }
> > >
> > > If I query the _settings/_mappings of my index, the analyzer
> > > and mappings are still defined correctly but the analyzer
> > > seems to have no effect.
> > >
> > > What am I doing wrong?
> > >
> > > Thanks in advance,
> > >
> > > Kevin
> > >
> > > git://gist.github.com/3442562.git
> > >
> > > ```
> > >
> > > --
>
> --
--
---
### Author:  [@Kevin\_Lawrence](https://discuss.elastic.co/u/Kevin_Lawrence)
#### Post date: [August 24, 2012, 7:37pm UTC](https://discuss.elastic.co/t/analyzer-disappears-after-server-restart-includes-gist/8840/6 "2012-08-24T19:37:00Z")
Thanks Ivan.
I suspect it's a bit more complex than that. When I query the \_settings for
my index after the restart, I still see my custom analyzer but the analyzer
is not used.
I plan to download the source for ES this afternoon to see if I can figure
it out.
Kevin
On Friday, August 24, 2012 11:37:56 AM UTC-7, Ivan Brusic wrote:
> It appears that issue is that an analyzer that is created as part of a
> create index request is not being saved. I am not familiar with the
> index metadata, but it appears that analyzers are not persisted and
> are re-read entirely from the yml file on startup. Haven't tried your
> gist, but it seems correct.
>
> The immediate workaround would be to define the analyzer in
> elasticseach.yml. Persisting analyzer information to the index
> metadata should be supported IMHO. I would open an issue and see what
> Shay thinks.
>
> Ivan
>
> On Fri, Aug 24, 2012 at 9:57 AM, Kevin Lawrence \<[ke...@diamond-sky.com](mailto:ke...@diamond-sky.com)\\>
> wrote:
>
> > Thanks Clint,
> >
> > I added my config file to the gist at:
> >
> > [ElasticSearch forgets my analyzer · GitHub](https://gist.github.com/3442562)
> >
> > but the only things changed from the default are path.data and
> > cluster.name.
> >
> > Kevin
> >
> > On Thursday, August 23, 2012 11:34:59 PM UTC-7, Clinton Gormley wrote:
> >
> > > Hi Kevin
> > >
> > > As I said yesterday, please gist a COMPLETE curl recreation, ie
> > > something that we can copy and paste to make it work, from index
> > > creation onwards. Snippets of information as below are insufficient.
> > >
> > > Also, please gist your elasticsearch config, as that may have some
> > > impact.
> > >
> > > ta
> > >
> > > clint
> > >
> > > On Thu, 2012-08-23 at 15:27 -0700, Kevin Lawrence wrote:
> > >
> > > > I get the same behavior in multiple versions of ES, including today's
> > > > 0.19.9.
> > > >
> > > > Kevin
> > > >
> > > > On Thursday, August 23, 2012 3:24:20 PM UTC-7, Kevin Lawrence wrote:
> > > > I have an analyzer:
> > > >
> > > > ```
> > > > "analyzer" : {
> > > > "lowercase_keyword" : {
> > > > "type" : "custom",
> > > > "tokenizer" : "keyword",
> > > > "filter" : ["lowercase", "trim"]
> > > > }
> > > > }
> > > >
> > > > that is referenced in a mapping:
> > > >
> > > > "location_countries" : {
> > > > "properties" : {
> > > > "country" : {
> > > > "type" : "string",
> > > > "analyzer" : "lowercase_keyword"
> > > > }
> > > > }
> > > > }
> > > >
> > > > And when I use the 'country' field in a filter or a facet,
> > > >
> > > > ```
>
> the
>
> > > > ```
> > > > field is (correctly) treated as a keyword.
> > > >
> > > > curl -XGET
> > > > 'localhost:9200/clinical_trials/_search?pretty=true' -d '
> > > > {
> > > > "query" : {
> > > > "term" : { "brief_title" : "dermatitis" }
> > > > },
> > > > "filter" : {
> > > > "term" : { "country" : "united states" }
> > > > },
> > > > "facets" : {
> > > > "tag" : {
> > > > "terms" : { "field" : "country" }
> > > > }
> > > > }
> > > > }
> > > > '
> > > >
> > > > facet results:
> > > >
> > > > "facets" : {
> > > > "tag" : {
> > > > "_type" : "terms",
> > > > "missing" : 0,
> > > > "total" : 1,
> > > > "other" : 0,
> > > > "terms" : [ {
> > > > "term" : "united states",
> > > > "count" : 1
> > > > } ]
> > > > }
> > > >
> > > > Everything works fine until the machine gets rebooted or the
> > > > Elastic Search service gets restarted. After a restart, all
> > > >
> > > > ```
>
> my
>
> > > > ```
> > > > filters stop working as if the analyzer does not exist.
> > > >
> > > > The same query against the same data results in:
> > > > "facets" : {
> > > > "tag" : {
> > > > "_type" : "terms",
> > > > "missing" : 0,
> > > > "total" : 2,
> > > > "other" : 0,
> > > > "terms" : [ {
> > > > "term" : "united",
> > > > "count" : 1
> > > > }, {
> > > > "term" : "states",
> > > > "count" : 1
> > > > } ]
> > > > }
> > > >
> > > > If I query the _settings/_mappings of my index, the analyzer
> > > > and mappings are still defined correctly but the analyzer
> > > > seems to have no effect.
> > > >
> > > > What am I doing wrong?
> > > >
> > > > Thanks in advance,
> > > >
> > > > Kevin
> > > >
> > > > git://gist.github.com/3442562.git
> > > >
> > > > ```
> > > >
> > > > --
> >
> > --
--
---
### Author:  [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)
#### Post date: [August 25, 2012, 9:42am UTC](https://discuss.elastic.co/t/analyzer-disappears-after-server-restart-includes-gist/8840/7 "2012-08-25T09:42:20Z")
Hi Kevin
> I suspect it's a bit more complex than that. When I query the
> \_settings for my index after the restart, I still see my custom
> analyzer but the analyzer is not used.
The problem is not that the analyzer is disappearing - it's that you
have two fields called 'country' with different mappings.
So after indexing, it is finding the keyword 'country' field first.
After restarting it is finding the analyzed 'country' field first.
If you change your query to the following, you will see that it works
correctly:
curl -XGET 'localhost:9200/clinical\_trials/\_search?pretty=true' -d '
{
"query" : {
"term" : { "brief\_title" : "dermatitis" }
},
"filter" : {
"term" : { "location\_countries.country" : "united states" }
},
"facets" : {
"tag" : {
"terms" : { "field" : "location\_countries.country" }
}
}
}
clint
--
---
### Author:  [@Kevin\_Lawrence](https://discuss.elastic.co/u/Kevin_Lawrence)
#### Post date: [August 27, 2012, 4:48pm UTC](https://discuss.elastic.co/t/analyzer-disappears-after-server-restart-includes-gist/8840/8 "2012-08-27T16:48:59Z")
Thanks Clint!
Trying that now.
Kevin
On Saturday, August 25, 2012 2:42:20 AM UTC-7, Clinton Gormley wrote:
> Hi Kevin
>
> > I suspect it's a bit more complex than that. When I query the
> > \_settings for my index after the restart, I still see my custom
> > analyzer but the analyzer is not used.
>
> The problem is not that the analyzer is disappearing - it's that you
> have two fields called 'country' with different mappings.
>
> So after indexing, it is finding the keyword 'country' field first.
> After restarting it is finding the analyzed 'country' field first.
>
> If you change your query to the following, you will see that it works
> correctly:
>
> curl -XGET 'localhost:9200/clinical\_trials/\_search?pretty=true' -d '
> {
> "query" : {
> "term" : { "brief\_title" : "dermatitis" }
> },
> "filter" : {
> "term" : { "location\_countries.country" : "united states" }
> },
> "facets" : {
> "tag" : {
> "terms" : { "field" : "location\_countries.country" }
> }
> }
> }
>
> clint
--
---
### Author:  [@Kevin\_Lawrence](https://discuss.elastic.co/u/Kevin_Lawrence)
#### Post date: [August 27, 2012, 6:02pm UTC](https://discuss.elastic.co/t/analyzer-disappears-after-server-restart-includes-gist/8840/9 "2012-08-27T18:02:00Z")
You are my hero, Clint! That worked.
Thanks for your patience with me.
Kevin
On Monday, August 27, 2012 9:48:59 AM UTC-7, Kevin Lawrence wrote:
> Thanks Clint!
>
> Trying that now.
>
> Kevin
>
> On Saturday, August 25, 2012 2:42:20 AM UTC-7, Clinton Gormley wrote:
>
> > Hi Kevin
> >
> > > I suspect it's a bit more complex than that. When I query the
> > > \_settings for my index after the restart, I still see my custom
> > > analyzer but the analyzer is not used.
> >
> > The problem is not that the analyzer is disappearing - it's that you
> > have two fields called 'country' with different mappings.
> >
> > So after indexing, it is finding the keyword 'country' field first.
> > After restarting it is finding the analyzed 'country' field first.
> >
> > If you change your query to the following, you will see that it works
> > correctly:
> >
> > curl -XGET 'localhost:9200/clinical\_trials/\_search?pretty=true' -d '
> > {
> > "query" : {
> > "term" : { "brief\_title" : "dermatitis" }
> > },
> > "filter" : {
> > "term" : { "location\_countries.country" : "united states" }
> > },
> > "facets" : {
> > "tag" : {
> > "terms" : { "field" : "location\_countries.country" }
> > }
> > }
> > }
> >
> > clint
--
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:14am UTC](https://discuss.elastic.co/t/analyzer-disappears-after-server-restart-includes-gist/8840/10 "2017-07-06T03:14:56Z")