Index JSON and stopwords with apostrophe

Elastic Stack Elasticsearch
1

Hey guys, I'm trying to set a custom list of stopwords in the JSON that
creates an index.

Like this: "stopwords" : ["i'm", "you're" .... ] . But when I check the
index metadata (through HEAD plugin), it seems it is parsing it wrong. It
states: index.analysis.filter.myStopFilter..stopwords.179: im, youre

I tried escaping, but ' should not be escaped inside JSON (it gives syntax
error)

I've tried searching for "i'm" and it finds results so it is not using it
right.

Any ideas?

Felipe Hummel

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

This works for me:

$ curl -XPOST localhost:9200/test -d '
{
"settings": {
"analysis": {
"analyzer": {
"default": {
"type": "standard",
"stopwords": ["i'"'"'m"]
}
}
}
}
}'

$ curl localhost:9200/test/_analyze?text=i'm
{"tokens":}

Maybe there is some escaping issue when you're putting the mappings? What
do your index settings/mappings look like?

On Friday, February 8, 2013 8:25:53 PM UTC, Felipe Hummel wrote:

Hey guys, I'm trying to set a custom list of stopwords in the JSON that
creates an index.

Like this: "stopwords" : ["i'm", "you're" .... ] . But when I check the
index metadata (through HEAD plugin), it seems it is parsing it wrong. It
states: index.analysis.filter.myStopFilter..stopwords.179: im, youre

I tried escaping, but ' should not be escaped inside JSON (it gives syntax
error)

I've tried searching for "i'm" and it finds results so it is not using it
right.

Any ideas?

Felipe Hummel

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

On second glance that's obviously the case! Might help to give your full
index creation post body and the responses you get for settings and
mappings.

On Friday, 8 February 2013 23:48:51 UTC, hazzadous wrote:

This works for me:

$ curl -XPOST localhost:9200/test -d '
{
"settings": {
"analysis": {
"analyzer": {
"default": {
"type": "standard",
"stopwords": ["i'"'"'m"]
}
}
}
}
}'

$ curl localhost:9200/test/_analyze?text=i'm
{"tokens":}

Maybe there is some escaping issue when you're putting the mappings? What
do your index settings/mappings look like?

On Friday, February 8, 2013 8:25:53 PM UTC, Felipe Hummel wrote:

Hey guys, I'm trying to set a custom list of stopwords in the JSON that
creates an index.

Like this: "stopwords" : ["i'm", "you're" .... ] . But when I check the
index metadata (through HEAD plugin), it seems it is parsing it wrong. It
states: index.analysis.filter.myStopFilter..stopwords.179: im, youre

I tried escaping, but ' should not be escaped inside JSON (it gives
syntax error)

I've tried searching for "i'm" and it finds results so it is not using it
right.

Any ideas?

Felipe Hummel

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

The settings were giving me this:
"index.analysis.filter.myStopFilter..stopwords.179": "im, youre"

With your way it gives me the correct form:

"index.analysis.filter.buskTokenFilter.stopwords.0":"i'm",

"index.analysis.filter.buskTokenFilter.stopwords.1":"you're",

Why that works? What is you'"'"'re ?

Thanks!

Felipe Hummel

On Friday, February 8, 2013 4:25:53 PM UTC-4, Felipe Hummel wrote:

Hey guys, I'm trying to set a custom list of stopwords in the JSON that
creates an index.

Like this: "stopwords" : ["i'm", "you're" .... ] . But when I check the
index metadata (through HEAD plugin), it seems it is parsing it wrong. It
states: index.analysis.filter.myStopFilter..stopwords.179: im, youre

I tried escaping, but ' should not be escaped inside JSON (it gives syntax
error)

I've tried searching for "i'm" and it finds results so it is not using it
right.

Any ideas?

Felipe Hummel

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

I got the syntax
from bash - How to escape single quotes within single quoted strings - Stack Overflow

I hadn't seen it before but good to know! Far from obvious.

On Saturday, 9 February 2013 17:43:17 UTC, Felipe Hummel wrote:

The settings were giving me this:
"index.analysis.filter.myStopFilter..stopwords.179": "im, youre"

With your way it gives me the correct form:

"index.analysis.filter.buskTokenFilter.stopwords.0":"i'm",

"index.analysis.filter.buskTokenFilter.stopwords.1":"you're",

Why that works? What is you'"'"'re ?

Thanks!

Felipe Hummel

On Friday, February 8, 2013 4:25:53 PM UTC-4, Felipe Hummel wrote:

Hey guys, I'm trying to set a custom list of stopwords in the JSON that
creates an index.

Like this: "stopwords" : ["i'm", "you're" .... ] . But when I check the
index metadata (through HEAD plugin), it seems it is parsing it wrong. It
states: index.analysis.filter.myStopFilter..stopwords.179: im, youre

I tried escaping, but ' should not be escaped inside JSON (it gives
syntax error)

I've tried searching for "i'm" and it finds results so it is not using it
right.

Any ideas?

Felipe Hummel

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

On Sat, 2013-02-09 at 16:41 -0800, Harry Waye wrote:

I got the syntax
from bash - How to escape single quotes within single quoted strings - Stack Overflow

I hadn't seen it before but good to know! Far from obvious.

You can also escape single quotes in JSON as \u0027

clint

On Saturday, 9 February 2013 17:43:17 UTC, Felipe Hummel wrote:
The settings were giving me this:
"index.analysis.filter.myStopFilter..stopwords.179": "im,
youre"

    With your way it gives me the correct form:
    
    
    "index.analysis.filter.buskTokenFilter.stopwords.0":"i'm",
    
    "index.analysis.filter.buskTokenFilter.stopwords.1":"you're",
    
    
    Why that works? What is you'"'"'re ?
    
    
    Thanks!
    
    
    Felipe Hummel
    
    
    On Friday, February 8, 2013 4:25:53 PM UTC-4, Felipe Hummel
    wrote:
            Hey guys, I'm trying to set a custom list of stopwords
            in the JSON that creates an index.
            
            Like this: "stopwords" : ["i'm", "you're" .... ] . But
            when I check the index metadata (through HEAD plugin),
            it seems it is parsing it wrong. It states:
            index.analysis.filter.myStopFilter..stopwords.179: im,
            youre
            
            I tried escaping, but ' should not be escaped inside
            JSON (it gives syntax error) 
            
            
            I've tried searching for "i'm" and it finds results so
            it is not using it right.
            
            Any ideas?
            
            Felipe Hummel

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Ah, much less obscure/shell specific.

On Sunday, 10 February 2013 14:33:21 UTC, Clinton Gormley wrote:

On Sat, 2013-02-09 at 16:41 -0800, Harry Waye wrote:

I got the syntax
from
bash - How to escape single quotes within single quoted strings - Stack Overflow

I hadn't seen it before but good to know! Far from obvious.

You can also escape single quotes in JSON as \u0027

clint

On Saturday, 9 February 2013 17:43:17 UTC, Felipe Hummel wrote:
The settings were giving me this:
"index.analysis.filter.myStopFilter..stopwords.179": "im,
youre"

    With your way it gives me the correct form: 
    
    
    "index.analysis.filter.buskTokenFilter.stopwords.0":"i'm", 
    
    "index.analysis.filter.buskTokenFilter.stopwords.1":"you're", 
    
    
    Why that works? What is you'"'"'re ? 
    
    
    Thanks! 
    
    
    Felipe Hummel 
    
    
    On Friday, February 8, 2013 4:25:53 PM UTC-4, Felipe Hummel 
    wrote: 
            Hey guys, I'm trying to set a custom list of stopwords 
            in the JSON that creates an index. 
            
            Like this: "stopwords" : ["i'm", "you're" .... ] . But 
            when I check the index metadata (through HEAD plugin), 
            it seems it is parsing it wrong. It states: 
            index.analysis.filter.myStopFilter..stopwords.179: im, 
            youre 
            
            I tried escaping, but ' should not be escaped inside 
            JSON (it gives syntax error) 
            
            
            I've tried searching for "i'm" and it finds results so 
            it is not using it right. 
            
            Any ideas? 
            
            Felipe Hummel

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.