Tokenizer analysis is not working properly in jdbc river

samaswin · October 15, 2013, 5:41am

I used the following query,

{
"type": "jdbc",
"jdbc": {
"strategy": "simple",
"driver": "com.mysql.jdbc.Driver",
"url": "jdbc:mysql://192.168.43.204:3306/BacheCompany",
"user": "turbodbuser",
"password": "turbo@2012",
"sql": "SELECT DISTINCT b.joMasterID as
_id,a.jobStatus,a.joStatusID,b.*,c.Name FROM joMaster AS b LEFT JOIN
joStatus AS a ON b.jobstatus=a.joStatusID LEFT JOIN rxMaster AS c ON
c.rxMasterID=b.rxCustomerID",
"poll": "10s",
"autocommit": true
},
"index": {
"index": "turbo1",
"type": "jobs1",
"bulk_timeout": "60s",
"index_settings": {
"analysis": {
"analyzer": {
"synonym": {
"tokenizer": "whitespace",
"filter": [
"synonym"
]
}
},
"filter": {
"synonym": {
"type": "synonym",
"format": "wordnet",
"synonyms_path": "analysis/prolog/wn_s.pl"
}
}
}
}
}
}

But the tokenizer is not working me. Can anyone suggest me the solution

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · October 15, 2013, 6:07am

Create your index and the mapping first.
Only then, create the river.

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 15 oct. 2013 à 07:41, sam aswin samaswin@gmail.com a écrit :

I used the following query,

{
"type": "jdbc",
"jdbc": {
"strategy": "simple",
"driver": "com.mysql.jdbc.Driver",
"url": "jdbc:mysql://192.168.43.204:3306/BacheCompany",
"user": "turbodbuser",
"password": "turbo@2012",
"sql": "SELECT DISTINCT b.joMasterID as _id,a.jobStatus,a.joStatusID,b.*,c.Name FROM joMaster AS b LEFT JOIN joStatus AS a ON b.jobstatus=a.joStatusID LEFT JOIN rxMaster AS c ON c.rxMasterID=b.rxCustomerID",
"poll": "10s",
"autocommit": true
},
"index": {
"index": "turbo1",
"type": "jobs1",
"bulk_timeout": "60s",
"index_settings": {
"analysis": {
"analyzer": {
"synonym": {
"tokenizer": "whitespace",
"filter": [
"synonym"
]
}
},
"filter": {
"synonym": {
"type": "synonym",
"format": "wordnet",
"synonyms_path": "analysis/prolog/wn_s.pl"
}
}
}
}
}
}

But the tokenizer is not working me. Can anyone suggest me the solution

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

samaswin · October 15, 2013, 7:51am

Thanks David, I understood the concept.
I need to configure the wordnet prolog in elasticsearch.
how can i done that?. The query is,

{
"index" : {
"analysis" : {
"analyzer" : {
"synonym" : {
"tokenizer" : "whitespace",
"filter" : ["synonym"]
}
},
"filter" : {
"synonym" : {
"type" : "synonym",
"format" : "wordnet",
"synonyms_path" : "analysis/wn_s.pl"
}
}
}
}
}

But the query still not working. Did I made any mistake

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · October 15, 2013, 7:54am

You should first try to get the analyzer running as expected without launching any river.

So, basically, create a full curl recreation:

delete index
create index with analyzer
use analyze API to check if your analyzer is correct

If not, GIST your curl recreation and we can have a look at it.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 15 oct. 2013 à 09:51, sam aswin samaswin@gmail.com a écrit :

Thanks David, I understood the concept.
I need to configure the wordnet prolog in elasticsearch.
how can i done that?. The query is,

{
"index" : {
"analysis" : {
"analyzer" : {
"synonym" : {
"tokenizer" : "whitespace",
"filter" : ["synonym"]
}
},
"filter" : {
"synonym" : {
"type" : "synonym",
"format" : "wordnet",
"synonyms_path" : "analysis/wn_s.pl"
}
}
}
}
}

But the query still not working. Did I made any mistake

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

samaswin · October 15, 2013, 7:54am

I created analysis folder in elasticsearch directory. Is this right?

I used the following query,

{
"type": "jdbc",
"jdbc": {
"strategy": "simple",
"driver": "com.mysql.jdbc.Driver",
"url": "jdbc:mysql://192.168.43.204:3306/BacheCompany",
"user": "turbodbuser",
"password": "turbo@2012",
"sql": "SELECT DISTINCT b.joMasterID as
_id,a.jobStatus,a.joStatusID,b.*,c.Name FROM joMaster AS b LEFT JOIN
joStatus AS a ON b.jobstatus=a.joStatusID LEFT JOIN rxMaster AS c ON
c.rxMasterID=b.rxCustomerID",
"poll": "10s",
"autocommit": true
},
"index": {
"index": "turbo1",
"type": "jobs1",
"bulk_timeout": "60s",
"index_settings": {
"analysis": {
"analyzer": {
"synonym": {
"tokenizer": "whitespace",
"filter": [
"synonym"
]
}
},
"filter": {
"synonym": {
"type": "synonym",
"format": "wordnet",
"synonyms_path": "analysis/prolog/wn_s.pl"
}
}
}
}
}
}

But the tokenizer is not working me. Can anyone suggest me the solution

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · October 15, 2013, 7:58am

As far as I can recall it, I think it should be under config directory. (unsure though)

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 15 oct. 2013 à 09:54, sam aswin samaswin@gmail.com a écrit :

I created analysis folder in elasticsearch directory. Is this right?

I used the following query,

{
"type": "jdbc",
"jdbc": {
"strategy": "simple",
"driver": "com.mysql.jdbc.Driver",
"url": "jdbc:mysql://192.168.43.204:3306/BacheCompany",
"user": "turbodbuser",
"password": "turbo@2012",
"sql": "SELECT DISTINCT b.joMasterID as _id,a.jobStatus,a.joStatusID,b.*,c.Name FROM joMaster AS b LEFT JOIN joStatus AS a ON b.jobstatus=a.joStatusID LEFT JOIN rxMaster AS c ON c.rxMasterID=b.rxCustomerID",
"poll": "10s",
"autocommit": true
},
"index": {
"index": "turbo1",
"type": "jobs1",
"bulk_timeout": "60s",
"index_settings": {
"analysis": {
"analyzer": {
"synonym": {
"tokenizer": "whitespace",
"filter": [
"synonym"
]
}
},
"filter": {
"synonym": {
"type": "synonym",
"format": "wordnet",
"synonyms_path": "analysis/prolog/wn_s.pl"
}
}
}
}
}
}

But the tokenizer is not working me. Can anyone suggest me the solution

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

samaswin · October 15, 2013, 8:08am

Thanks David.
But still the problem is exist. and there is no error in log also.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · October 15, 2013, 8:14am

Then Gist the full curl recreation.

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 15 oct. 2013 à 10:08, sam aswin samaswin@gmail.com a écrit :

Thanks David.
But still the problem is exist. and there is no error in log also.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

samaswin · October 15, 2013, 8:18am

I use RESTClient in firfox,

Index mapping
Put:
http://localhost:9200/projects/
{
"index" : {
"analysis" : {
"analyzer" : {
"synonym" : {
"tokenizer" : "whitespace",
"filter" : ["synonym"]
}
},
"filter" : {
"synonym" : {
"type" : "synonym",
"format" : "wordnet",
"synonyms_path" : "analysis/wn_s.pl"
}
}
}
}
}

Put;
http://localhost:9200/projects/project/2
{
"name" : "child"
}

http://localhost:9200/projects/project/2
{
"name" : "baby"
}

Post:
http://localhost:9200/projects/_search?pretty=true
{
"query": {
"query_string": {
"query": "child"
}
}

}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · October 15, 2013, 8:46am

What output do you get?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 15 oct. 2013 à 10:18, sam aswin samaswin@gmail.com a écrit :

I use RESTClient in firfox,

Index mapping
Put:
http://localhost:9200/projects/
{
"index" : {
"analysis" : {
"analyzer" : {
"synonym" : {
"tokenizer" : "whitespace",
"filter" : ["synonym"]
}
},
"filter" : {
"synonym" : {
"type" : "synonym",
"format" : "wordnet",
"synonyms_path" : "analysis/wn_s.pl"
}
}
}
}
}

Put;
http://localhost:9200/projects/project/2
{
"name" : "child"
}

http://localhost:9200/projects/project/2
{
"name" : "baby"
}

Post:
http://localhost:9200/projects/_search?pretty=true
{
"query": {
"query_string": {
"query": "child"
}
}

}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

samaswin · October 15, 2013, 8:47am

{
```
"took": 2,
```
```
"timed_out": false,
```
```
"_shards":
```
```
{
```
```
    "total": 1,
```
```
    "successful": 1,
```
```
    "failed": 0
```
```
},
```
```
"hits":
```
```
{
```
```
    "total": 1,
```
```
    "max_score": 1,
```
```
    "hits":
```
```
    [
```
```
        {
```
```
            "_index": "projects",
```
```
            "_type": "project",
```
```
            "_id": "3",
```
```
            "_score": 1,
```
```
            "_source":
```
```
            {
```
```
                "name": "kid"
```
```
            }
```
```
        }
```
```
    ]
```
```
}
```
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

samaswin · October 15, 2013, 8:48am

{
```
"took": 2,
```
```
"timed_out": false,
```
```
"_shards":
```
```
{
```
```
    "total": 1,
```
```
    "successful": 1,
```
```
    "failed": 0
```
```
},
```
```
"hits":
```
```
{
```
```
    "total": 1,
```
```
    "max_score": 1,
```
```
    "hits":
```
```
    [
```
```
        {
```
```
            "_index": "projects",
```
```
            "_type": "project",
```
```
            "_id": "3",
```
```
            "_score": 1,
```
```
            "_source":
```
```
            {
```
```
                "name": "child"
```
```
            }
```
```
        }
```
```
    ]
```
```
}
```
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · October 15, 2013, 8:54am

So baby and child are synonyms in your file, right?
And you expect 2 results right?

In your example you set the same id for both docs. So you replaced the old one.

I think it's a typo but unsure.

It will help a lot if you Gist a full curl recreation (with synonym file as well).

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 15 oct. 2013 à 10:48, sam aswin samaswin@gmail.com a écrit :

{
"took": 2,
"timed_out": false,
"_shards":
{
"total": 1,
"successful": 1,
"failed": 0
},
"hits":
{
"total": 1,
"max_score": 1,
"hits":
[
{
"_index": "projects",
"_type": "project",
"_id": "3",
"_score": 1,
"_source":
{
"name": "child"
}
}
]
}
}

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

samaswin · October 15, 2013, 8:58am

Yes I need the two results. But above is the type mistake. If I search
GET:
http://localhost:9200/projects/_search?pretty=true

The results:

{
```
"took": 1,
```
```
"timed_out": false,
```
```
"_shards":
```
```
{
```
```
    "total": 1,
```
```
    "successful": 1,
```
```
    "failed": 0
```
```
},
```
```
"hits":
```
```
{
```
```
    "total": 2,
```
```
    "max_score": 1,
```
```
    "hits":
```
```
    [
```
```
        {
```
```
            "_index": "projects",
```
```
            "_type": "project",
```
```
            "_id": "1",
```
```
            "_score": 1,
```
```
            "_source":
```
```
            {
```
```
                "name": "child"
```
```
            }
```
```
        },
```
```
        {
```
```
            "_index": "projects",
```
```
            "_type": "project",
```
```
            "_id": "3",
```
```
            "_score": 1,
```
```
            "_source":
```
```
            {
```
```
                "name": "kid"
```
```
            }
```
```
        }
```
```
    ]
```
```
}
```
}

Above are the typo mistakes. I add it for reference.

On Tuesday, 15 October 2013 14:24:01 UTC+5:30, David Pilato wrote:

So baby and child are synonyms in your file, right?
And you expect 2 results right?

In your example you set the same id for both docs. So you replaced the old
one.

I think it's a typo but unsure.

It will help a lot if you Gist a full curl recreation (with synonym file
as well).

--

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

samaswin · October 15, 2013, 9:04am

I removed the entire index and recreated. Still i got the same problem.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

samaswin · October 15, 2013, 9:22am

PUT:

http://localhost:9200/projects/

{
"index": {
"analysis": {
"analyzer": {
"synonym": {
"tokenizer": "whitespace",
"filter": [
"synonym"
]
}
},
"filter": {
"synonym": {
"type": "synonym",
"format": "wordnet",
"synonyms_path": "analysis/wn_s.pl"
}
}
}
}
}

PUT:
http://localhost:9200/projects/project/1

{
"name" : "kid"
}

http://localhost:9200/projects/project/2

{
"name" : "child"
}

POST:
http://localhost:9200/projects/_search?pretty=true
{
"query": {
"query_string": {
"query": "child"
}
}
}

analysis folder:
/usr/share/elasticsearch/config/analysis/

Wordnet prolog file is:
/usr/share/elasticsearch/config/analysis/wn_s.pl

Wordnet version is 3.0

RESULT:

{
```
"took": 1,
```
```
"timed_out": false,
```
```
"_shards":
```
```
{
```
```
    "total": 1,
```
```
    "successful": 1,
```
```
    "failed": 0
```
```
},
```
```
"hits":
```
```
{
```
```
    "total": 1,
```
```
    "max_score": 1,
```
```
    "hits":
```
```
    [
```
```
        {
```
```
            "_index": "projects",
```
```
            "_type": "project",
```
```
            "_id": "2",
```
```
            "_score": 1,
```
```
            "_source":
```
```
            {
```
```
                "name": "child"
```
```
            }
```
```
        }
```
```
    ]
```
```
}
```
}

Expected Result:

kid and child should be display.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · October 15, 2013, 9:28am

You forgot the mapping for the field "name", compare your commands to

gist.github.com

https://gist.github.com/jprante/6968540

synonym.sh


curl -XDELETE 'localhost:9200/test'

curl -XPUT 'localhost:9200/test/' -d '
{
  "settings" : {
    "index" : {
        "analysis" : {
            "analyzer" : {
                "synonym" : {

This file has been truncated. show original

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

samaswin · October 15, 2013, 9:43am

Thank Jorg,
I solved my problem. Thanks David for your help

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Tokenizer analysis is not working properly in jdbc river

But the tokenizer is not working me. Can anyone suggest me the solution

Thanks David. But still the problem is exist. and there is no error in log also.

{ "took": 2, "timed_out": false, "_shards": { "total": 1, "successful": 1, "failed": 0 }, "hits": { "total": 1, "max_score": 1, "hits": [ { "_index": "projects", "_type": "project", "_id": "3", "_score": 1, "_source": { "name": "child" } } ] } }

Thanks David.
But still the problem is exist. and there is no error in log also.

{
"took": 2,
"timed_out": false,
"_shards":
{
"total": 1,
"successful": 1,
"failed": 0
},
"hits":
{
"total": 1,
"max_score": 1,
"hits":
[
{
"_index": "projects",
"_type": "project",
"_id": "3",
"_score": 1,
"_source":
{
"name": "child"
}
}
]
}
}