How to add filter on searching data on index using java?

Sam_4 · January 16, 2012, 6:18am

Hi

I have used following mapping to define custom analyzer at index level
used mapping:

gist.github.com

https://gist.github.com/sameek/1619313

gistfile1.json

{"index" : { "_index" : "test1", "_type" : "type1", "_id" : "1", "_analyzer":"myanalyzer" } }
{ "type1" : { "name" : "my name is john" } }
{ "myanalyzer" : {"type" : "custom", "tokenizer" :"whitespace","filter" :["myfilter1"]}}
{"myfilter1":{"type":"stop" "stopwords_path":"D:\resources\stopwords_eng.txt" ,"ignore_case":true}}
{"create" : { "_index" : "test1", "_type" : "type1", "_id" : "1" } }
{ "type1" : { "name" : "my name is john" } }

stopwords_eng.txt

is
the 
an

I am using following java code to search data:
String indexName="test1";
String indexType="type1";
String data="is";

        QueryStringQueryBuilder querystring = new

QueryStringQueryBuilder(data);
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchType.DEFAULT)
.setQuery(querystring.analyzer("whitespace")).setSize(100).execute().actionGet();

I have question to you how to define stop word filter and
stopwords_path setting on this search query.please help me.

Thanks

kimchy · January 17, 2012, 9:24am

You don't define it on the query, you define it in the index configuration
with a custom analyzer, and then use that analyzer either by making it the
default one, or specifically applying it in the mapping to specific fields.

On Mon, Jan 16, 2012 at 8:18 AM, sam mishra.sameek@gmail.com wrote:

Hi

I have used following mapping to define custom analyzer at index level
used mapping:

Custom analyzer Mapping · GitHub

I am using following java code to search data:
String indexName="test1";
String indexType="type1";
String data="is";
       QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(data);
SearchResponse searchResponse =

client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchType.DEFAULT)

.setQuery(querystring.analyzer("whitespace")).setSize(100).execute().actionGet();

I have question to you how to define stop word filter and
stopwords_path setting on this search query.please help me.

Thanks

Sam_4 · January 17, 2012, 10:58am

thanks for reply.could u explain how to define index configuration
with a custom analyzer and also how to make it default one. i have
tried but i did not found any document how to define custom analyzer
with rabbitmq river and make it default.please refer any document or
solution,which help me.

On Jan 17, 2:24 pm, Shay Banon kim...@gmail.com wrote:

You don't define it on the query, you define it in the index configuration
with a custom analyzer, and then use that analyzer either by making it the
default one, or specifically applying it in the mapping to specific fields.

On Mon, Jan 16, 2012 at 8:18 AM, sam mishra.sam...@gmail.com wrote:

Hi

I have used following mapping to define custom analyzer at index level
used mapping:

Custom analyzer Mapping · GitHub

I am using following java code to search data:
String indexName="test1";
String indexType="type1";
String data="is";
       QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(data);
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchTyp e.DEFAULT)

.setQuery(querystring.analyzer("whitespace")).setSize(100).execute().action Get();

I have question to you how to define stop word filter and
stopwords_path setting on this search query.please help me.

Thanks

kimchy · January 17, 2012, 11:29am

When you define the rabbitmq river, you define which index it is going to
index into. Before you define the river, you will need to make sure to
create that index with the relevant mappings you plan to use. Default
analyzer can be defined simply by naming it "default".

Analysis Docs:
Elasticsearch Platform — Find real-time answers at scale | Elastic.
Create Index API (with mapping):
Elasticsearch Platform — Find real-time answers at scale | Elastic
.
Associating an analyzer with a field:
Elasticsearch Platform — Find real-time answers at scale | Elastic.

On Tue, Jan 17, 2012 at 12:58 PM, sam mishra.sameek@gmail.com wrote:

thanks for reply.could u explain how to define index configuration
with a custom analyzer and also how to make it default one. i have
tried but i did not found any document how to define custom analyzer
with rabbitmq river and make it default.please refer any document or
solution,which help me.

On Jan 17, 2:24 pm, Shay Banon kim...@gmail.com wrote:
You don't define it on the query, you define it in the index
configuration
with a custom analyzer, and then use that analyzer either by making it
the
default one, or specifically applying it in the mapping to specific
fields.

On Mon, Jan 16, 2012 at 8:18 AM, sam mishra.sam...@gmail.com wrote:

Hi

I have used following mapping to define custom analyzer at index level
used mapping:

Custom analyzer Mapping · GitHub

I am using following java code to search data:
String indexName="test1";
String indexType="type1";
String data="is";
       QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(data);
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchTyp
e.DEFAULT)

.setQuery(querystring.analyzer("whitespace")).setSize(100).execute().action
Get();

I have question to you how to define stop word filter and
stopwords_path setting on this search query.please help me.

Thanks

Sam_4 · January 17, 2012, 12:15pm

Thanks for quick reply. i have created gist for used mapping:

River Mapping · GitHub

when i try to execute it got following exception:

org.elasticsearch.common.jackson.JsonParseException: Unexpected
character ('"' (code 34)): was expecting comma to separate OBJECT
entries
at [Source: [B@1171e30; line: 1, column: 336]
at
org.elasticsearch.common.jackson.JsonParser._constructError(JsonParser.java:
1291)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:
385)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:
306)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:
312)
at
org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:
46)
at org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:
121)
at
org.elasticsearch.client.action.bulk.BulkRequestBuilder.add(BulkRequestBuilder.java:
84)
at org.elasticsearch.river.rabbitmq.RabbitmqRiver
$Consumer.run(RabbitmqRiver.java:216)
at java.lang.Thread.run(Thread.java:662)

please help where the problem in my mapping.

Thanks

On Jan 17, 4:29 pm, Shay Banon kim...@gmail.com wrote:

When you define the rabbitmq river, you define which index it is going to
index into. Before you define the river, you will need to make sure to
create that index with the relevant mappings you plan to use. Default
analyzer can be defined simply by naming it "default".

Analysis Docs:Elasticsearch Platform — Find real-time answers at scale | Elastic.
Create Index API (with mapping):Elasticsearch Platform — Find real-time answers at scale | Elastic...
.
Associating an analyzer with a field:Elasticsearch Platform — Find real-time answers at scale | Elastic.

On Tue, Jan 17, 2012 at 12:58 PM, sam mishra.sam...@gmail.com wrote:

thanks for reply.could u explain how to define index configuration
with a custom analyzer and also how to make it default one. i have
tried but i did not found any document how to define custom analyzer
with rabbitmq river and make it default.please refer any document or
solution,which help me.

On Jan 17, 2:24 pm, Shay Banon kim...@gmail.com wrote:

You don't define it on the query, you define it in the index
configuration
with a custom analyzer, and then use that analyzer either by making it
the
default one, or specifically applying it in the mapping to specific
fields.

On Mon, Jan 16, 2012 at 8:18 AM, sam mishra.sam...@gmail.com wrote:

Hi

I have used following mapping to define custom analyzer at index level
used mapping:

Custom analyzer Mapping · GitHub

I am using following java code to search data:
String indexName="test1";
String indexType="type1";
String data="is";
       QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(data);
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchTyp
e.DEFAULT)

.setQuery(querystring.analyzer("whitespace")).setSize(100).execute().action
Get();

I have question to you how to define stop word filter and
stopwords_path setting on this search query.please help me.

Thanks

Sam_4 · January 18, 2012, 5:47am

I resolved the above syntax error and now it is not giving me any
exception but again the mapping is not working as expected.

Tested Output:
I included "inc" in the stop words file but when I searched "inc" it
should give me 0 hits as the stop word filter, filters the stop words
from the index but I am getting 1 hit as the text I have indexed
contain "inc".
This made me sure that the custom analyzer is not configured
correctly.

I updated the above mapping and now I am using this mapping
River Mapping · GitHub but I am not sure
that this is completely correct.Please tell me where I am going wrong
and what changes are needed to be made.

Thanks
On Jan 17, 5:15 pm, sam mishra.sam...@gmail.com wrote:

Thanks for quick reply. i have created gist for used mapping:

River Mapping · GitHub

when i try to execute it got following exception:

org.elasticsearch.common.jackson.JsonParseException: Unexpected
character ('"' (code 34)): was expecting comma to separate OBJECT
entries
at [Source: [B@1171e30; line: 1, column: 336]
at
org.elasticsearch.common.jackson.JsonParser._constructError(JsonParser.java :
1291)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._reportError(Js onParserMinimalBase.java:
385)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._reportUnexpect edChar(JsonParserMinimalBase.java:
306)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser.nextToken(Utf8Stream Parser.java:
312)
at
org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXCo ntentParser.java:
46)
at org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:
121)
at
org.elasticsearch.client.action.bulk.BulkRequestBuilder.add(BulkRequestBuil der.java:
84)
at org.elasticsearch.river.rabbitmq.RabbitmqRiver
$Consumer.run(RabbitmqRiver.java:216)
at java.lang.Thread.run(Thread.java:662)

please help where the problem in my mapping.

Thanks

On Jan 17, 4:29 pm, Shay Banon kim...@gmail.com wrote:

When you define the rabbitmq river, you define which index it is going to
index into. Before you define the river, you will need to make sure to
create that index with the relevant mappings you plan to use. Default
analyzer can be defined simply by naming it "default".

Analysis Docs:Elasticsearch Platform — Find real-time answers at scale | Elastic.
Create Index API (with mapping):Elasticsearch Platform — Find real-time answers at scale | Elastic...
.
Associating an analyzer with a field:Elasticsearch Platform — Find real-time answers at scale | Elastic.

On Tue, Jan 17, 2012 at 12:58 PM, sam mishra.sam...@gmail.com wrote:

thanks for reply.could u explain how to define index configuration
with a custom analyzer and also how to make it default one. i have
tried but i did not found any document how to define custom analyzer
with rabbitmq river and make it default.please refer any document or
solution,which help me.

On Jan 17, 2:24 pm, Shay Banon kim...@gmail.com wrote:

You don't define it on the query, you define it in the index
configuration
with a custom analyzer, and then use that analyzer either by making it
the
default one, or specifically applying it in the mapping to specific
fields.

On Mon, Jan 16, 2012 at 8:18 AM, sam mishra.sam...@gmail.com wrote:

Hi

I have used following mapping to define custom analyzer at index level
used mapping:

Custom analyzer Mapping · GitHub

I am using following java code to search data:
String indexName="test1";
String indexType="type1";
String data="is";
       QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(data);
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchTyp
e.DEFAULT)

.setQuery(querystring.analyzer("whitespace")).setSize(100).execute().action
Get();

I have question to you how to define stop word filter and
stopwords_path setting on this search query.please help me.

Thanks

Sam_4 · January 18, 2012, 9:39am

It is disappointing to me.i didn't find mapping details when define
custom analyzer with rabbitmq,but i did not getting any helpful
response from your side.elastic provided documentation is quite
confusing. please any one help me to get correct mapping for both
index_analyzer and search_analyzer.i not able to find correct mapping.

On Jan 18, 10:47 am, sam mishra.sam...@gmail.com wrote:

I resolved the above syntax error and now it is not giving me any
exception but again the mapping is not working as expected.

Tested Output:
I included "inc" in the stop words file but when I searched "inc" it
should give me 0 hits as the stop word filter, filters the stop words
from the index but I am getting 1 hit as the text I have indexed
contain "inc".
This made me sure that the custom analyzer is not configured
correctly.

I updated the above mapping and now I am using this mappinghttps://gist.github.com/1626452but I am not sure
that this is completely correct.Please tell me where I am going wrong
and what changes are needed to be made.

Thanks
On Jan 17, 5:15 pm, sam mishra.sam...@gmail.com wrote:

Thanks for quick reply. i have created gist for used mapping:

River Mapping · GitHub

when i try to execute it got following exception:

org.elasticsearch.common.jackson.JsonParseException: Unexpected
character ('"' (code 34)): was expecting comma to separate OBJECT
entries
at [Source: [B@1171e30; line: 1, column: 336]
at
org.elasticsearch.common.jackson.JsonParser._constructError(JsonParser.java :
1291)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._reportError(Js onParserMinimalBase.java:
385)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._reportUnexpect edChar(JsonParserMinimalBase.java:
306)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser.nextToken(Utf8Stream Parser.java:
312)
at
org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXCo ntentParser.java:
46)
at org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:
121)
at
org.elasticsearch.client.action.bulk.BulkRequestBuilder.add(BulkRequestBuil der.java:
84)
at org.elasticsearch.river.rabbitmq.RabbitmqRiver
$Consumer.run(RabbitmqRiver.java:216)
at java.lang.Thread.run(Thread.java:662)

please help where the problem in my mapping.

Thanks

On Jan 17, 4:29 pm, Shay Banon kim...@gmail.com wrote:

When you define the rabbitmq river, you define which index it is going to
index into. Before you define the river, you will need to make sure to
create that index with the relevant mappings you plan to use. Default
analyzer can be defined simply by naming it "default".

Analysis Docs:Elasticsearch Platform — Find real-time answers at scale | Elastic.
Create Index API (with mapping):Elasticsearch Platform — Find real-time answers at scale | Elastic...
.
Associating an analyzer with a field:Elasticsearch Platform — Find real-time answers at scale | Elastic.

On Tue, Jan 17, 2012 at 12:58 PM, sam mishra.sam...@gmail.com wrote:

thanks for reply.could u explain how to define index configuration
with a custom analyzer and also how to make it default one. i have
tried but i did not found any document how to define custom analyzer
with rabbitmq river and make it default.please refer any document or
solution,which help me.

On Jan 17, 2:24 pm, Shay Banon kim...@gmail.com wrote:

You don't define it on the query, you define it in the index
configuration
with a custom analyzer, and then use that analyzer either by making it
the
default one, or specifically applying it in the mapping to specific
fields.

On Mon, Jan 16, 2012 at 8:18 AM, sam mishra.sam...@gmail.com wrote:

Hi

I have used following mapping to define custom analyzer at index level
used mapping:

Custom analyzer Mapping · GitHub

I am using following java code to search data:
String indexName="test1";
String indexType="type1";
String data="is";
       QueryStringQueryBuilder querystring = new
QueryStringQueryBuilder(data);
SearchResponse searchResponse =
client.prepareSearch(indexName).setTypes(indexType).setSearchType(SearchTyp
e.DEFAULT)

.setQuery(querystring.analyzer("whitespace")).setSize(100).execute().action
Get();

I have question to you how to define stop word filter and
stopwords_path setting on this search query.please help me.

Thanks