Query DSL as json

Jimi_Parekh · January 21, 2013, 7:22pm

Hi,

I have included the query dsl in a json file. Below is the exact how my
"input.json" looks like (Please note the use of '*' before and after the
text:

{
"query" : {
"term" : { "@message" : "Von" }
}
}

Belwo is my curl command:

curl -XGET -d @input.json "http://localhost:9200/mass-idx/_search"

But it don't find any result. There are many documents having text 'Von' in
the '@message' field, but still it don't find anything.

Is there anything I am doing wrong?

Thanks.

--

Jimi_Parekh · January 21, 2013, 7:52pm

Additional Info: When I use query string and pass the same condition it
gives me the desire result. Below is the curl request with query string.

curl -XGET "http://localhost:9200/_search?q=@message:Von&pretty=true"

Please let me know if both the request are translated to same search or
different?

Thanks.

On Monday, January 21, 2013 2:22:24 PM UTC-5, Jimi Parekh wrote:

Hi,

I have included the query dsl in a json file. Below is the exact how my
"input.json" looks like (Please note the use of '*' before and after the
text:

{
"query" : {
"term" : { "@message" : "Von" }
}
}

Belwo is my curl command:

curl -XGET -d @input.json "http://localhost:9200/mass-idx/_search"

But it don't find any result. There are many documents having text 'Von'
in the '@message' field, but still it don't find anything.

Is there anything I am doing wrong?

Thanks.

--

dadoonet · January 21, 2013, 8:01pm

TermQuery is not analyzed. So ES try to compare "Von" to "von" which does not
match.
Use a QueryString if you want to do the same search as q=...

HTH

Le 21 janvier 2013 à 20:52, Jimi Parekh jimiparekh@gmail.com a écrit :

Additional Info: When I use query string and pass the same condition it gives
me the desire result. Below is the curl request with query string.

curl -XGET "http://localhost:9200/_search?q=@message:Von&pretty=true"

Please let me know if both the request are translated to same search or
different?

Thanks.

On Monday, January 21, 2013 2:22:24 PM UTC-5, Jimi Parekh wrote:
Hi,

I have included the query dsl in a json file. Below is the exact how my
"input.json" looks like (Please note the use of '*' before and after the
text:

{
"query" : {
"term" : { "@message" : "Von" }
}
}

Belwo is my curl command:

curl -XGET -d @input.json " http://localhost:9200/mass-idx/_search"

But it don't find any result. There are many documents having text 'Von'
in the '@message' field, but still it don't find anything.

Is there anything I am doing wrong?

Thanks.
<http://localhost:9200/mass-idx/_search>
--

http://localhost:9200/mass-idx/_search

http://localhost:9200/mass-idx/_search

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

Jimi_Parekh · January 21, 2013, 8:15pm

David,

I am new to ES. what do you mean by 'TermQuery is not analyzed"?

I tried to use below two different json format but both gives parsing error.

{
"query_string" : {

    "default_field":"@message",
    "query" : "Von"
}

}

{
"query_string" : {

    "query" : "@message:Von"
}

}

What will be the exact json for my query?

Thanks.

On Monday, January 21, 2013 3:01:10 PM UTC-5, David Pilato wrote:

TermQuery is not analyzed. So ES try to compare "Von" to "von" which
does not match.
Use a QueryString if you want to do the same search as q=...

HTH

Le 21 janvier 2013 à 20:52, Jimi Parekh <jimip...@gmail.com <javascript:>>
a écrit :

Additional Info: When I use query string and pass the same condition it
gives me the desire result. Below is the curl request with query string.

curl -XGET "http://localhost:9200/_search?q=@message:Von&pretty=true"

Please let me know if both the request are translated to same search or
different?

Thanks.

On Monday, January 21, 2013 2:22:24 PM UTC-5, Jimi Parekh wrote:

Hi,

I have included the query dsl in a json file. Below is the exact how my
"input.json" looks like (Please note the use of '*' before and after the
text:

{
"query" : {
"term" : { "@message" : "Von" }
}
}

Belwo is my curl command:

curl -XGET -d @input.json " http://localhost:9200/mass-idx/_search"

But it don't find any result. There are many documents having text 'Von'
in the '@message' field, but still it don't find anything.

Is there anything I am doing wrong?

Thanks.

http://localhost:9200/mass-idx/_search

--

http://localhost:9200/mass-idx/_search

http://localhost:9200/mass-idx/_search

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

dadoonet · January 21, 2013, 8:36pm

Try this:

$ curl -XGET 'http://localhost:9200/_search' -d '{

"query" : {
    "query_string" : {
        "query" : "@message:Von"
    }
}

}'

I mean that your query doesn't go through the analysis process.
When you index a document, content is analyzed (break into tokens, lowercased,
filtered...) before being indexed.

Von is indexed as von in the inverted index.
If you use a matchQuery or a QueryString it goes through the same process. So,
if you search for Von, in fact you will search for von in the inverted index and
you will find your document.

Make sense?

Le 21 janvier 2013 à 21:15, Jimi Parekh jimiparekh@gmail.com a écrit :

David,

I am new to ES. what do you mean by 'TermQuery is not analyzed"?

I tried to use below two different json format but both gives parsing error.

{
"query_string" : {
     "default_field":"@message",
     "query" : "Von"
 }
}

{
"query_string" : {
     "query" : "@message:Von"
 }
}

What will be the exact json for my query?

Thanks.

On Monday, January 21, 2013 3:01:10 PM UTC-5, David Pilato wrote:
TermQuery is not analyzed. So ES try to compare "Von" to "von"
which does not match.
Use a QueryString if you want to do the same search as q=...

HTH

Le 21 janvier 2013 à 20:52, Jimi Parekh < jimip...@gmail.com> a écrit :
> > > Additional Info: When I use query string and pass the same
> > > condition it gives me the desire result. Below is the curl request
> > > with query string.
curl -XGET " http://localhost:9200/_search?q=@message:Von&pretty=true"

Please let me know if both the request are translated to same search
or different?
Thanks.

On Monday, January 21, 2013 2:22:24 PM UTC-5, Jimi Parekh wrote:
http://localhost:9200/_search?q=@message:Von&pretty=true
> > > > Hi,
  I have included the query dsl in a json file. Below is the exact
how my "input.json" looks like (Please note the use of '*' before and
after the text:
  {
    "query" : {
            "term" : { "@message" : "*Von*" }
     }
  }

  Belwo is my curl command:

  curl -XGET -d @input.json "
http://localhost:9200/_search?q=@message:Von&pretty=true
http://localhost:9200/mass-idx/_search"
  But it don't find any result. There are many documents having text
'Von' in the '@message' field, but still it don't find anything.
  Is there anything I am doing wrong?

  Thanks.



   <http://localhost:9200/mass-idx/_search>
> > > 
--


 <http://localhost:9200/mass-idx/_search>
 <http://localhost:9200/mass-idx/_search>
--
David Pilato
http://www.scrutmydocs.org/ http://www.scrutmydocs.org/
http://dev.david.pilato.fr/ http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

Jimi_Parekh · January 21, 2013, 10:11pm

David,

I tried your suggestion and it seems working but not giving the expected
result.
I want to search the "@message" field and wherever that field contain the
string 'Von' anywhere in that field I want all those documents.

We can not use wildcard with query_string? I want to use term query then
what will be the exact query syntax.?

Thanks.

On Monday, January 21, 2013 3:36:54 PM UTC-5, David Pilato wrote:

Try this:

$ curl -XGET 'http://localhost:9200/_search' -d '{
"query" : {
"query_string" : {
"query" : "@message:Von"
}
}
}'

I mean that your query doesn't go through the analysis process.
When you index a document, content is analyzed (break into tokens,
lowercased, filtered...) before being indexed.

Von is indexed as von in the inverted index.
If you use a matchQuery or a QueryString it goes through the same
process. So, if you search for Von, in fact you will search for von in the
inverted index and you will find your document.

Make sense?

Le 21 janvier 2013 à 21:15, Jimi Parekh <jimip...@gmail.com <javascript:>>
a écrit :

David,

I am new to ES. what do you mean by 'TermQuery is not analyzed"?

I tried to use below two different json format but both gives parsing
error.

{
"query_string" : {
    "default_field":"@message", 
    "query" : "Von" 
} 
}

{
"query_string" : {
    "query" : "@message:Von" 
} 
}

What will be the exact json for my query?

Thanks.

On Monday, January 21, 2013 3:01:10 PM UTC-5, David Pilato wrote:

TermQuery is not analyzed. So ES try to compare "Von" to "von" which
does not match.
Use a QueryString if you want to do the same search as q=...

HTH

Le 21 janvier 2013 à 20:52, Jimi Parekh < jimip...@gmail.com> a écrit :

Additional Info: When I use query string and pass the same condition it
gives me the desire result. Below is the curl request with query string.

curl -XGET " http://localhost:9200/_search?q=@message:Von&pretty=true"

Please let me know if both the request are translated to same search or
different?

Thanks.

On Monday, January 21, 2013 2:22:24 PM UTC-5, Jimi Parekh wrote:
http://localhost:9200/_search?q=@message:Von&pretty=true

Hi,

I have included the query dsl in a json file. Below is the exact how my
"input.json" looks like (Please note the use of '*' before and after the
text:

{
"query" : {
"term" : { "@message" : "Von" }
}
}

Belwo is my curl command:

curl -XGET -d @input.json "
http://localhost:9200/_search?q=@message:Von&pretty=true http://localhost:9200/mass-idx/_search"

But it don't find any result. There are many documents having text 'Von'
in the '@message' field, but still it don't find anything.

Is there anything I am doing wrong?

Thanks.

http://localhost:9200/mass-idx/_search

--

http://localhost:9200/mass-idx/_search

http://localhost:9200/mass-idx/_search

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

Jimi_Parekh · January 22, 2013, 4:30pm

David,

I am trying out few things on this and I getting more confuse. I also
looked into the elasticsearch document but I don't find the proper answer.

Let me explain my scenario and the issue i am facing.

In my indexed document there is a field "@message", with the following kind
of value.

"@message:Track-Id=1234|Token-Id=ABC123XYZ|Receive-Time-MS=1358423498000".
There are other field also like "@source", "@host" etc.
I have about 300 documents indexed.

Now I want to search for the documents where "@message" field has a string
"Track-Id=1234".
Now as per your previous suggestion I create a following search:

curl -XGET 'http://localhost:9200/_search' -d '{
"query" : {
"query_string" : {
"query" : "@message:Track-Id=1234"
}
}
}'

But this returned me all the documents. What wrong I am doing? What is the
better approach for such queries, Query-DSL or Query string?

Thanks.
On Monday, January 21, 2013 5:11:31 PM UTC-5, Jimi Parekh wrote:

David,

I tried your suggestion and it seems working but not giving the expected
result.
I want to search the "@message" field and wherever that field contain the
string 'Von' anywhere in that field I want all those documents.

We can not use wildcard with query_string? I want to use term query then
what will be the exact query syntax.?

Thanks.

On Monday, January 21, 2013 3:36:54 PM UTC-5, David Pilato wrote:
Try this:

$ curl -XGET 'http://localhost:9200/_search' -d '{
"query" : {
"query_string" : {
"query" : "@message:Von"
}
}
}'

I mean that your query doesn't go through the analysis process.
When you index a document, content is analyzed (break into tokens,
lowercased, filtered...) before being indexed.

Von is indexed as von in the inverted index.
If you use a matchQuery or a QueryString it goes through the same
process. So, if you search for Von, in fact you will search for von in the
inverted index and you will find your document.

Make sense?

Le 21 janvier 2013 à 21:15, Jimi Parekh jimip...@gmail.com a écrit :

David,

I am new to ES. what do you mean by 'TermQuery is not analyzed"?

I tried to use below two different json format but both gives parsing
error.

{
"query_string" : {
    "default_field":"@message", 
    "query" : "Von" 
} 
}

{
"query_string" : {
    "query" : "@message:Von" 
} 
}

What will be the exact json for my query?

Thanks.

On Monday, January 21, 2013 3:01:10 PM UTC-5, David Pilato wrote:

TermQuery is not analyzed. So ES try to compare "Von" to "von" which
does not match.
Use a QueryString if you want to do the same search as q=...

HTH

Le 21 janvier 2013 à 20:52, Jimi Parekh < jimip...@gmail.com> a écrit :

Additional Info: When I use query string and pass the same condition it
gives me the desire result. Below is the curl request with query string.

curl -XGET " http://localhost:9200/_search?q=@message:Von&pretty=true"

Please let me know if both the request are translated to same search or
different?

Thanks.

On Monday, January 21, 2013 2:22:24 PM UTC-5, Jimi Parekh wrote:
http://localhost:9200/_search?q=@message:Von&pretty=true

Hi,

I have included the query dsl in a json file. Below is the exact how my
"input.json" looks like (Please note the use of '*' before and after the
text:

{
"query" : {
"term" : { "@message" : "Von" }
}
}

Belwo is my curl command:

curl -XGET -d @input.json "
http://localhost:9200/_search?q=@message:Von&pretty=true http://localhost:9200/mass-idx/_search"

But it don't find any result. There are many documents having text 'Von'
in the '@message' field, but still it don't find anything.

Is there anything I am doing wrong?

Thanks.

http://localhost:9200/mass-idx/_search

--

http://localhost:9200/mass-idx/_search

http://localhost:9200/mass-idx/_search

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

dadoonet · January 22, 2013, 5:01pm

Can you create and gist a full curl recreation as we can reproduce your concern?
I don't think I understand field as you do. How does your JSon document look
like?

Le 22 janvier 2013 à 17:30, Jimi Parekh jimiparekh@gmail.com a écrit :

David,

I am trying out few things on this and I getting more confuse. I also looked
into the elasticsearch document but I don't find the proper answer.

Let me explain my scenario and the issue i am facing.

In my indexed document there is a field "@message", with the following kind
of value.

"@message:Track-Id=1234|Token-Id=ABC123XYZ|Receive-Time-MS=1358423498000".
There are other field also like "@source", "@host" etc.
I have about 300 documents indexed.

Now I want to search for the documents where "@message" field has a string
"Track-Id=1234".
Now as per your previous suggestion I create a following search:

curl -XGET 'http://localhost:9200/_search' -d '{
"query" : {
"query_string" : {
"query" : "@message:Track-Id=1234"
}
}
}'

But this returned me all the documents. What wrong I am doing? What is the
better approach for such queries, Query-DSL or Query string?

Thanks.
On Monday, January 21, 2013 5:11:31 PM UTC-5, Jimi Parekh wrote:
David,

I tried your suggestion and it seems working but not giving the expected
result.
I want to search the "@message" field and wherever that field contain the
string 'Von' anywhere in that field I want all those documents.

We can not use wildcard with query_string? I want to use term query then
what will be the exact query syntax.?

Thanks.

On Monday, January 21, 2013 3:36:54 PM UTC-5, David Pilato wrote:
> > >
 Try this:



 $ curl -XGET 'http://localhost:9200/_search
http://localhost:9200/_search ' -d '{ "query" : {
"query_string" : {
"query" : "@message:Von"
}
}
}'
 I mean that your query doesn't go through the analysis process.
 When you index a document, content is analyzed (break into tokens,
lowercased, filtered...) before being indexed.
 Von is indexed as von in the inverted index.
 If you use a matchQuery or a QueryString it goes through the same
process. So, if you search for Von, in fact you will search for von in the
inverted index and you will find your document.
 Make sense?

 Le 21 janvier 2013 à 21:15, Jimi Parekh < jimip...@gmail.com> a écrit
:
  > > > > David,
  I am new to ES. what do you mean by 'TermQuery is not analyzed"?

  I tried to use below two different json format but both gives
parsing error.
  1)
  {
      "query_string" : {

          "default_field":"@message",
          "query" : "Von"
      }
  }

  2)

  {
      "query_string" : {

          "query" : "@message:Von"
      }
  }

  What will be the exact json for my query?

  Thanks.

  On Monday, January 21, 2013 3:01:10 PM UTC-5, David Pilato wrote:
    > > > > >         TermQuery is not analyzed. So ES try to
    > > > > > compare "*Von*" to "von" which does not match.
    Use a QueryString if you want to do the same search as q=...

    HTH

    Le 21 janvier 2013 à 20:52, Jimi Parekh < jimip...@gmail.com>
a écrit :
     > > > > > > Additional Info: When I use query string and pass
     > > > > > > the same condition it gives me the desire result.
     > > > > > > Below is the curl request with query string.
     curl -XGET "
http://localhost:9200/_search?q=@message:Von&pretty=true"
     Please let me know if both the request are translated to
same search or different?
     Thanks.

     On Monday, January 21, 2013 2:22:24 PM UTC-5, Jimi Parekh
wrote: http://localhost:9200/_search?q=@message:Von&pretty=true
> > > > > > > http://localhost:9200/_search?q=@message:Von&pretty=true
> > > > > > > Hi,
       I have included the query dsl in a json file. Below is
the exact how my "input.json" looks like (Please note the use of
'*' before and after the text:
       {
         "query" : {
                 "term" : { "@message" : "*Von*" }
          }
       }

       Belwo is my curl command:

       curl -XGET -d @input.json "
http://localhost:9200/_search?q=@message:Von&pretty=true
http://localhost:9200/mass-idx/_search"
       But it don't find any result. There are many documents
having text 'Von' in the '@message' field, but still it don't find
anything.
       Is there anything I am doing wrong?

       Thanks.



        <http://localhost:9200/mass-idx/_search>
     > > > > > > 
     --


      <http://localhost:9200/mass-idx/_search>

    > > > > >           <http://localhost:9200/mass-idx/_search>
    --
    David Pilato
    http://www.scrutmydocs.org/ <http://www.scrutmydocs.org/>
    http://dev.david.pilato.fr/ <http://dev.david.pilato.fr/>
    Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
  > > > > 
  --



 > > > 
 --
 David Pilato
 http://www.scrutmydocs.org/ <http://www.scrutmydocs.org/>
 http://dev.david.pilato.fr/ <http://dev.david.pilato.fr/>
 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

mattweber · January 22, 2013, 5:05pm

I imagine it is the @ symbol in your field names are being tokenized via
the query_string parser. Try a match query or remove the @symbol from your
field names. Also, if you are only doing exact matches like that, make
sure your fields are not analyzed.

{
"query": {
"match": {
"@message": {
"query": "Track-Id=1234"
}
}
}
}

On Tue, Jan 22, 2013 at 8:30 AM, Jimi Parekh jimiparekh@gmail.com wrote:

David,

I am trying out few things on this and I getting more confuse. I also
looked into the elasticsearch document but I don't find the proper answer.

Let me explain my scenario and the issue i am facing.

In my indexed document there is a field "@message", with the following
kind of value.

"@message:Track-Id=1234|Token-Id=ABC123XYZ|Receive-Time-MS=1358423498000".
There are other field also like "@source", "@host" etc.
I have about 300 documents indexed.

Now I want to search for the documents where "@message" field has a string
"Track-Id=1234".
Now as per your previous suggestion I create a following search:

curl -XGET 'http://localhost:9200/_search' -d '{
"query" : {
"query_string" : {
"query" : "@message:Track-Id=1234"
}
}
}'

But this returned me all the documents. What wrong I am doing? What is the
better approach for such queries, Query-DSL or Query string?

Thanks.
On Monday, January 21, 2013 5:11:31 PM UTC-5, Jimi Parekh wrote:
David,

I tried your suggestion and it seems working but not giving the expected
result.
I want to search the "@message" field and wherever that field contain the
string 'Von' anywhere in that field I want all those documents.

We can not use wildcard with query_string? I want to use term query then
what will be the exact query syntax.?

Thanks.

On Monday, January 21, 2013 3:36:54 PM UTC-5, David Pilato wrote:
Try this:

$ curl -XGET 'http://localhost:9200/_search**' -d '{
"query" : {
"query_string" : {
"query" : "@message:Von"
}
}
}'

I mean that your query doesn't go through the analysis process.
When you index a document, content is analyzed (break into tokens,
lowercased, filtered...) before being indexed.

Von is indexed as von in the inverted index.
If you use a matchQuery or a QueryString it goes through the same
process. So, if you search for Von, in fact you will search for von in the
inverted index and you will find your document.

Make sense?

Le 21 janvier 2013 à 21:15, Jimi Parekh jimip...@gmail.com a écrit :

David,

I am new to ES. what do you mean by 'TermQuery is not analyzed"?

I tried to use below two different json format but both gives parsing
error.

{
"query_string" : {
    "default_field":"@message",
    "query" : "Von"
}
}

{
"query_string" : {
    "query" : "@message:Von"
}
}

What will be the exact json for my query?

Thanks.

On Monday, January 21, 2013 3:01:10 PM UTC-5, David Pilato wrote:

TermQuery is not analyzed. So ES try to compare "Von" to "von" which
does not match.
Use a QueryString if you want to do the same search as q=...

HTH

Le 21 janvier 2013 à 20:52, Jimi Parekh < jimip...@gmail.com> a écrit :

Additional Info: When I use query string and pass the same condition it
gives me the desire result. Below is the curl request with query string.

curl -XGET " http://localhost:9200/_search?**q=@message:Von&pretty=true"

Please let me know if both the request are translated to same search or
different?

Thanks.

On Monday, January 21, 2013 2:22:24 PM UTC-5, Jimi Parekh wrote:
http://localhost:9200/_search?q=@message:Von&pretty=true

Hi,

I have included the query dsl in a json file. Below is the exact how my
"input.json" looks like (Please note the use of '*' before and after the
text:

{
"query" : {
"term" : { "@message" : "Von" }
}
}

Belwo is my curl command:

curl -XGET -d @input.json "
http://localhost:9200/_search?q=@message:Von&pretty=true
http://localhost:9200/mass-**idx/_search"

But it don't find any result. There are many documents having text 'Von'
in the '@message' field, but still it don't find anything.

Is there anything I am doing wrong?

Thanks.

http://localhost:9200/mass-idx/_search

--

http://localhost:9200/mass-idx/_search

http://localhost:9200/mass-idx/_search

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
--

--

Jimi_Parekh · January 22, 2013, 7:11pm

Matt,
I tried your suggestion but the result is same as previous.

On Tuesday, January 22, 2013 12:05:51 PM UTC-5, Matt Weber wrote:

I imagine it is the @ symbol in your field names are being tokenized via
the query_string parser. Try a match query or remove the @symbol from your
field names. Also, if you are only doing exact matches like that, make
sure your fields are not analyzed.

{
"query": {
"match": {
"@message": {
"query": "Track-Id=1234"
}
}
}
}

On Tue, Jan 22, 2013 at 8:30 AM, Jimi Parekh <jimip...@gmail.com<javascript:>

wrote:
David,

I am trying out few things on this and I getting more confuse. I also
looked into the elasticsearch document but I don't find the proper answer.

Let me explain my scenario and the issue i am facing.

In my indexed document there is a field "@message", with the following
kind of value.

"@message:Track-Id=1234|Token-Id=ABC123XYZ|Receive-Time-MS=1358423498000".
There are other field also like "@source", "@host" etc.
I have about 300 documents indexed.

Now I want to search for the documents where "@message" field has a
string "Track-Id=1234".
Now as per your previous suggestion I create a following search:

curl -XGET 'http://localhost:9200/_search' -d '{
"query" : {
"query_string" : {
"query" : "@message:Track-Id=1234"
}
}
}'

But this returned me all the documents. What wrong I am doing? What is
the better approach for such queries, Query-DSL or Query string?

Thanks.
On Monday, January 21, 2013 5:11:31 PM UTC-5, Jimi Parekh wrote:
David,

I tried your suggestion and it seems working but not giving the expected
result.
I want to search the "@message" field and wherever that field contain
the string 'Von' anywhere in that field I want all those documents.

We can not use wildcard with query_string? I want to use term query then
what will be the exact query syntax.?

Thanks.

On Monday, January 21, 2013 3:36:54 PM UTC-5, David Pilato wrote:
Try this:

$ curl -XGET 'http://localhost:9200/_search**' -d '{
"query" : {
"query_string" : {
"query" : "@message:Von"
}
}
}'

I mean that your query doesn't go through the analysis process.
When you index a document, content is analyzed (break into tokens,
lowercased, filtered...) before being indexed.

Von is indexed as von in the inverted index.
If you use a matchQuery or a QueryString it goes through the same
process. So, if you search for Von, in fact you will search for von in the
inverted index and you will find your document.

Make sense?

Le 21 janvier 2013 à 21:15, Jimi Parekh jimip...@gmail.com a écrit :

David,

I am new to ES. what do you mean by 'TermQuery is not analyzed"?

I tried to use below two different json format but both gives parsing
error.

{
"query_string" : {
    "default_field":"@message", 
    "query" : "Von" 
} 
}

{
"query_string" : {
    "query" : "@message:Von" 
} 
}

What will be the exact json for my query?

Thanks.

On Monday, January 21, 2013 3:01:10 PM UTC-5, David Pilato wrote:

TermQuery is not analyzed. So ES try to compare "Von" to "von"
which does not match.
Use a QueryString if you want to do the same search as q=...

HTH

Le 21 janvier 2013 à 20:52, Jimi Parekh < jimip...@gmail.com> a
écrit :

Additional Info: When I use query string and pass the same condition it
gives me the desire result. Below is the curl request with query string.

curl -XGET " http://localhost:9200/_search?**q=@message:Von&pretty=true"

Please let me know if both the request are translated to same search or
different?

Thanks.

On Monday, January 21, 2013 2:22:24 PM UTC-5, Jimi Parekh wrote:
http://localhost:9200/_search?q=@message:Von&pretty=true

Hi,

I have included the query dsl in a json file. Below is the exact how my
"input.json" looks like (Please note the use of '*' before and after the
text:

{
"query" : {
"term" : { "@message" : "Von" }
}
}

Belwo is my curl command:

curl -XGET -d @input.json "
http://localhost:9200/_search?q=@message:Von&pretty=true
http://localhost:9200/mass-**idx/_search"

But it don't find any result. There are many documents having text
'Von' in the '@message' field, but still it don't find anything.

Is there anything I am doing wrong?

Thanks.

http://localhost:9200/mass-idx/_search

--

http://localhost:9200/mass-idx/_search

http://localhost:9200/mass-idx/_search

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
--

--

Jimi_Parekh · January 22, 2013, 7:14pm

David,
Below is the Json documents I am indexing.
{
"@source":"file://../../log/xyz.log",
"@tags":,
"@fields":{},
"@timestamp":"2013-01-21T21:42:11.678Z",
"@source_host":"localhost",
"@source_path":"/../log/xyz.log",
"@message":"Track-ID=3269795|Receive-Time=2013-01-16
21:02:23|Receive-Time-MS=1358370143000|Token-ID=APA91bGYkO29UHYzFxaUlEBCo4FDjZpzsTGdlhCSTGXbxGri8YSKY0fk9jH2N60Qxfp8RFbiJEnRXcCrv5zteK0UQvX2SwgHuABMluupGyUYEaxxB_Mxwjifm2vYnrQMh_QT_ZEBMpKY6OJnTtglawipHIE9ftyy9Q|User-ID=0v_odd99nf_u3facz9d|Client-Type=type|Client-Version=2.00.0|P-Time=2013-01-17
11:57:01|P-Time-MS=1358423821000|Current-Time-MS=1358416621248","@type":"applog"}
}

Actually I am using 'logstash' to get the even logs generated by my
application. And this is further indexed by logstash and stored in
elasticsearch. Then I am querying the Elasticsearch.

On Tuesday, January 22, 2013 12:01:03 PM UTC-5, David Pilato wrote:

Can you create and gist a full curl recreation as we can reproduce your
concern?
I don't think I understand field as you do. How does your JSon document
look like?

Le 22 janvier 2013 à 17:30, Jimi Parekh <jimip...@gmail.com <javascript:>>
a écrit :

David,

I am trying out few things on this and I getting more confuse. I also
looked into the elasticsearch document but I don't find the proper answer.

Let me explain my scenario and the issue i am facing.

In my indexed document there is a field "@message", with the following
kind of value.

"@message:Track-Id=1234|Token-Id=ABC123XYZ|Receive-Time-MS=1358423498000".
There are other field also like "@source", "@host" etc.
I have about 300 documents indexed.

Now I want to search for the documents where "@message" field has a string
"Track-Id=1234".
Now as per your previous suggestion I create a following search:

curl -XGET 'http://localhost:9200/_search' -d '{
"query" : {
"query_string" : {
"query" : "@message:Track-Id=1234"
}
}
}'

But this returned me all the documents. What wrong I am doing? What is the
better approach for such queries, Query-DSL or Query string?

Thanks.
On Monday, January 21, 2013 5:11:31 PM UTC-5, Jimi Parekh wrote:

David,

I tried your suggestion and it seems working but not giving the expected
result.
I want to search the "@message" field and wherever that field contain the
string 'Von' anywhere in that field I want all those documents.

We can not use wildcard with query_string? I want to use term query then
what will be the exact query syntax.?

Thanks.

On Monday, January 21, 2013 3:36:54 PM UTC-5, David Pilato wrote:

Try this:

$ curl -XGET 'http://localhost:9200/_search' -d '{ "query" : {
"query_string" : {
"query" : "@message:Von"
}
}
}'

I mean that your query doesn't go through the analysis process.
When you index a document, content is analyzed (break into tokens,
lowercased, filtered...) before being indexed.

Von is indexed as von in the inverted index.
If you use a matchQuery or a QueryString it goes through the same
process. So, if you search for Von, in fact you will search for von in the
inverted index and you will find your document.

Make sense?

Le 21 janvier 2013 à 21:15, Jimi Parekh < jimip...@gmail.com> a écrit :

David,

I am new to ES. what do you mean by 'TermQuery is not analyzed"?

I tried to use below two different json format but both gives parsing
error.

{
"query_string" : {
    "default_field":"@message", 
    "query" : "Von" 
} 
}

{
"query_string" : {
    "query" : "@message:Von" 
} 
}

What will be the exact json for my query?

Thanks.

On Monday, January 21, 2013 3:01:10 PM UTC-5, David Pilato wrote:

TermQuery is not analyzed. So ES try to compare "Von" to "von" which
does not match.
Use a QueryString if you want to do the same search as q=...

HTH

Le 21 janvier 2013 à 20:52, Jimi Parekh < jimip...@gmail.com> a écrit :

Additional Info: When I use query string and pass the same condition it
gives me the desire result. Below is the curl request with query string.

curl -XGET " http://localhost:9200/_search?q=@message:Von&pretty=true"

Please let me know if both the request are translated to same search or
different?

Thanks.

On Monday, January 21, 2013 2:22:24 PM UTC-5, Jimi Parekh wrote:
http://localhost:9200/_search?q=@message:Von&pretty=true

http://localhost:9200/_search?q=@message:Von&pretty=true Hi,

I have included the query dsl in a json file. Below is the exact how my
"input.json" looks like (Please note the use of '*' before and after the
text:

{
"query" : {
"term" : { "@message" : "Von" }
}
}

Belwo is my curl command:

curl -XGET -d @input.json "
http://localhost:9200/_search?q=@message:Von&pretty=true http://localhost:9200/mass-idx/_search"

But it don't find any result. There are many documents having text 'Von'
in the '@message' field, but still it don't find anything.

Is there anything I am doing wrong?

Thanks.

http://localhost:9200/mass-idx/_search

--

http://localhost:9200/mass-idx/_search

http://localhost:9200/mass-idx/_search

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

mattweber · January 22, 2013, 7:41pm

Did you set your @message field to not analyzed?

On Tue, Jan 22, 2013 at 11:11 AM, Jimi Parekh jimiparekh@gmail.com wrote:

Matt,
I tried your suggestion but the result is same as previous.

On Tuesday, January 22, 2013 12:05:51 PM UTC-5, Matt Weber wrote:
I imagine it is the @ symbol in your field names are being tokenized via
the query_string parser. Try a match query or remove the @symbol from your
field names. Also, if you are only doing exact matches like that, make
sure your fields are not analyzed.

{
"query": {
"match": {
"@message": {
"query": "Track-Id=1234"
}
}
}
}

On Tue, Jan 22, 2013 at 8:30 AM, Jimi Parekh jimip...@gmail.com wrote:
David,

I am trying out few things on this and I getting more confuse. I also
looked into the elasticsearch document but I don't find the proper answer.

Let me explain my scenario and the issue i am facing.

In my indexed document there is a field "@message", with the following
kind of value.

"@message:Track-Id=1234|Token-Id=ABC123XYZ|Receive-Time-MS=
1358423498000".
There are other field also like "@source", "@host" etc.
I have about 300 documents indexed.

Now I want to search for the documents where "@message" field has a
string "Track-Id=1234".
Now as per your previous suggestion I create a following search:

curl -XGET 'http://localhost:9200/_search**' -d '{
"query" : {
"query_string" : {
"query" : "@message:Track-Id=1234"
}
}
}'

But this returned me all the documents. What wrong I am doing? What is
the better approach for such queries, Query-DSL or Query string?

Thanks.
On Monday, January 21, 2013 5:11:31 PM UTC-5, Jimi Parekh wrote:
David,

I tried your suggestion and it seems working but not giving the
expected result.
I want to search the "@message" field and wherever that field contain
the string 'Von' anywhere in that field I want all those documents.

We can not use wildcard with query_string? I want to use term query
then what will be the exact query syntax.?

Thanks.

On Monday, January 21, 2013 3:36:54 PM UTC-5, David Pilato wrote:
Try this:

$ curl -XGET 'http://localhost:9200/_search****' -d '{
"query" : {
"query_string" : {
"query" : "@message:Von"
}
}
}'

I mean that your query doesn't go through the analysis process.
When you index a document, content is analyzed (break into tokens,
lowercased, filtered...) before being indexed.

Von is indexed as von in the inverted index.
If you use a matchQuery or a QueryString it goes through the same
process. So, if you search for Von, in fact you will search for von in the
inverted index and you will find your document.

Make sense?

Le 21 janvier 2013 à 21:15, Jimi Parekh jimip...@gmail.com a
écrit :

David,

I am new to ES. what do you mean by 'TermQuery is not analyzed"?

I tried to use below two different json format but both gives parsing
error.

{
"query_string" : {
    "default_field":"@message",
    "query" : "Von"
}
}

{
"query_string" : {
    "query" : "@message:Von"
}
}

What will be the exact json for my query?

Thanks.

On Monday, January 21, 2013 3:01:10 PM UTC-5, David Pilato wrote:

TermQuery is not analyzed. So ES try to compare "Von" to "von"
which does not match.
Use a QueryString if you want to do the same search as q=...

HTH

Le 21 janvier 2013 à 20:52, Jimi Parekh < jimip...@gmail.com> a
écrit :

Additional Info: When I use query string and pass the same condition
it gives me the desire result. Below is the curl request with query string.

curl -XGET " http://localhost:9200/_search?****q=@message:Von&pretty=true"

Please let me know if both the request are translated to same search
or different?

Thanks.

On Monday, January 21, 2013 2:22:24 PM UTC-5, Jimi Parekh wrote:
http://localhost:9200/_search?q=@message:Von&pretty=true

Hi,

I have included the query dsl in a json file. Below is the exact how
my "input.json" looks like (Please note the use of '*' before and after the
text:

{
"query" : {
"term" : { "@message" : "Von" }
}
}

Belwo is my curl command:

curl -XGET -d @input.json "
http://localhost:9200/_search?q=@message:Von&pretty=true
http://localhost:9200/mass-**idx**/_search"

But it don't find any result. There are many documents having text
'Von' in the '@message' field, but still it don't find anything.

Is there anything I am doing wrong?

Thanks.

http://localhost:9200/mass-idx/_search

--

http://localhost:9200/mass-idx/_search

http://localhost:9200/mass-idx/_search

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
--
--

--

dadoonet · January 23, 2013, 8:29am

Please follow Matt's advices:

avoid special chars in field names, even if it could work, I see it as a bad practice
create a mapping before indexing the first document and set the right analyzer for your use case or set it as not analyzed but not sure it will help for your use case

If the content of @message is always like: TrackID=value|Receive-Time=value|…, why not modify it before you index it in valuable fields:
trackid=XXXXX
receivetime=xxxxxx

You will be able then to do searches on theses fields or compute facets.

David

Le 22 janv. 2013 à 20:41, Matt Weber matt.weber@gmail.com a écrit :

Did you set your @message field to not analyzed?

On Tue, Jan 22, 2013 at 11:11 AM, Jimi Parekh jimiparekh@gmail.com wrote:
Matt,
I tried your suggestion but the result is same as previous.

On Tuesday, January 22, 2013 12:05:51 PM UTC-5, Matt Weber wrote:
I imagine it is the @ symbol in your field names are being tokenized via the query_string parser. Try a match query or remove the @symbol from your field names. Also, if you are only doing exact matches like that, make sure your fields are not analyzed.

{
"query": {
"match": {
"@message": {
"query": "Track-Id=1234"
}
}
}
}

On Tue, Jan 22, 2013 at 8:30 AM, Jimi Parekh jimip...@gmail.com wrote:
David,

I am trying out few things on this and I getting more confuse. I also looked into the elasticsearch document but I don't find the proper answer.

Let me explain my scenario and the issue i am facing.

In my indexed document there is a field "@message", with the following kind of value.

"@message:Track-Id=1234|Token-Id=ABC123XYZ|Receive-Time-MS=1358423498000".
There are other field also like "@source", "@host" etc.
I have about 300 documents indexed.

Now I want to search for the documents where "@message" field has a string "Track-Id=1234".
Now as per your previous suggestion I create a following search:

curl -XGET 'http://localhost:9200/_search' -d '{
"query" : {
"query_string" : {
"query" : "@message:Track-Id=1234"
}
}
}'

But this returned me all the documents. What wrong I am doing? What is the better approach for such queries, Query-DSL or Query string?

Thanks.
On Monday, January 21, 2013 5:11:31 PM UTC-5, Jimi Parekh wrote:
David,

I tried your suggestion and it seems working but not giving the expected result.
I want to search the "@message" field and wherever that field contain the string 'Von' anywhere in that field I want all those documents.

We can not use wildcard with query_string? I want to use term query then what will be the exact query syntax.?

Thanks.

On Monday, January 21, 2013 3:36:54 PM UTC-5, David Pilato wrote:
Try this:

$ curl -XGET 'http://localhost:9200/_search' -d '{
"query" : { 
    "query_string" : { 
        "query" : "@message:Von" 
    } 
} 
}'

I mean that your query doesn't go through the analysis process.
When you index a document, content is analyzed (break into tokens, lowercased, filtered...) before being indexed.

Von is indexed as von in the inverted index.
If you use a matchQuery or a QueryString it goes through the same process. So, if you search for Von, in fact you will search for von in the inverted index and you will find your document.

Make sense?

Le 21 janvier 2013 à 21:15, Jimi Parekh jimip...@gmail.com a écrit :
David,

I am new to ES. what do you mean by 'TermQuery is not analyzed"?

I tried to use below two different json format but both gives parsing error.

{
"query_string" : {
    "default_field":"@message", 
    "query" : "Von" 
} 
}

{
"query_string" : {
    "query" : "@message:Von" 
} 
}

What will be the exact json for my query?

Thanks.

On Monday, January 21, 2013 3:01:10 PM UTC-5, David Pilato wrote:
TermQuery is not analyzed. So ES try to compare "Von" to "von" which does not match.
Use a QueryString if you want to do the same search as q=...

HTH

Le 21 janvier 2013 à 20:52, Jimi Parekh < jimip...@gmail.com> a écrit :

Additional Info: When I use query string and pass the same condition it gives me the desire result. Below is the curl request with query string.

curl -XGET " http://localhost:9200/_search?q=@message:Von&pretty=true"

Please let me know if both the request are translated to same search or different?

Thanks.

On Monday, January 21, 2013 2:22:24 PM UTC-5, Jimi Parekh wrote:
Hi,

I have included the query dsl in a json file. Below is the exact how my "input.json" looks like (Please note the use of '*' before and after the text:

{
"query" : {
"term" : { "@message" : "Von" }
}
}

Belwo is my curl command:

curl -XGET -d @input.json " http://localhost:9200/mass-idx/_search"

But it don't find any result. There are many documents having text 'Von' in the '@message' field, but still it don't find anything.

Is there anything I am doing wrong?

Thanks.

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--
--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

--

--

--

Jimi_Parekh · January 23, 2013, 3:34pm

Thanks Matt and David.

I am very new to ES so I am not aware with the lot of things. I am going
through the docs on elasticsearch site, but not getting clearity in many
things.

As I mention in my earlier post, the documents I am indexing are passed by
the 'logstash' to elasticsearch. logstash reads the application eventlog
and pass them to elasticsearch for indexing and storage. Thats where it
adds the metadata like @source, @host etc. The acutal log content is the
value of message field only.

Is there a way I can remove @ before indexing? How can I seperate
different fileds of 'message'? Does elasticsearch provide some setting for
those?

On Wednesday, January 23, 2013 3:29:34 AM UTC-5, David Pilato wrote:

Please follow Matt's advices:

avoid special chars in field names, even if it could work, I see it as a
bad practice

create a mapping before indexing the first document and set the right
analyzer for your use case or set it as not analyzed but not sure it will
help for your use case

If the content of @message is always like:
TrackID=value|Receive-Time=value|…, why not modify it before you index it
in valuable fields:
trackid=XXXXX
receivetime=xxxxxx

You will be able then to do searches on theses fields or compute facets.

David

Le 22 janv. 2013 à 20:41, Matt Weber <matt....@gmail.com <javascript:>> a
écrit :

Did you set your @message field to not analyzed?

On Tue, Jan 22, 2013 at 11:11 AM, Jimi Parekh <jimip...@gmail.com<javascript:>

wrote:
Matt,
I tried your suggestion but the result is same as previous.

On Tuesday, January 22, 2013 12:05:51 PM UTC-5, Matt Weber wrote:
I imagine it is the @ symbol in your field names are being tokenized via
the query_string parser. Try a match query or remove the @symbol from your
field names. Also, if you are only doing exact matches like that, make
sure your fields are not analyzed.

{
"query": {
"match": {
"@message": {
"query": "Track-Id=1234"
}
}
}
}

On Tue, Jan 22, 2013 at 8:30 AM, Jimi Parekh jimip...@gmail.com wrote:
David,

I am trying out few things on this and I getting more confuse. I also
looked into the elasticsearch document but I don't find the proper answer.

Let me explain my scenario and the issue i am facing.

In my indexed document there is a field "@message", with the following
kind of value.

"@message:Track-Id=1234|Token-Id=ABC123XYZ|Receive-Time-MS=
1358423498000".
There are other field also like "@source", "@host" etc.
I have about 300 documents indexed.

Now I want to search for the documents where "@message" field has a
string "Track-Id=1234".
Now as per your previous suggestion I create a following search:

curl -XGET 'http://localhost:9200/_search**' -d '{
"query" : {
"query_string" : {
"query" : "@message:Track-Id=1234"
}
}
}'

But this returned me all the documents. What wrong I am doing? What is
the better approach for such queries, Query-DSL or Query string?

Thanks.
On Monday, January 21, 2013 5:11:31 PM UTC-5, Jimi Parekh wrote:
David,

I tried your suggestion and it seems working but not giving the
expected result.
I want to search the "@message" field and wherever that field contain
the string 'Von' anywhere in that field I want all those documents.

We can not use wildcard with query_string? I want to use term query
then what will be the exact query syntax.?

Thanks.

On Monday, January 21, 2013 3:36:54 PM UTC-5, David Pilato wrote:
Try this:

$ curl -XGET 'http://localhost:9200/_search****' -d '{
"query" : {
"query_string" : {
"query" : "@message:Von"
}
}
}'

I mean that your query doesn't go through the analysis process.
When you index a document, content is analyzed (break into tokens,
lowercased, filtered...) before being indexed.

Von is indexed as von in the inverted index.
If you use a matchQuery or a QueryString it goes through the same
process. So, if you search for Von, in fact you will search for von in the
inverted index and you will find your document.

Make sense?

Le 21 janvier 2013 à 21:15, Jimi Parekh jimip...@gmail.com a
écrit :

David,

I am new to ES. what do you mean by 'TermQuery is not analyzed"?

I tried to use below two different json format but both gives parsing
error.

{
"query_string" : {
    "default_field":"@message", 
    "query" : "Von" 
} 
}

{
"query_string" : {
    "query" : "@message:Von" 
} 
}

What will be the exact json for my query?

Thanks.

On Monday, January 21, 2013 3:01:10 PM UTC-5, David Pilato wrote:

TermQuery is not analyzed. So ES try to compare "Von" to "von"
which does not match.
Use a QueryString if you want to do the same search as q=...

HTH

Le 21 janvier 2013 à 20:52, Jimi Parekh < jimip...@gmail.com> a
écrit :

Additional Info: When I use query string and pass the same condition
it gives me the desire result. Below is the curl request with query string.

curl -XGET " http://localhost:9200/_search?****q=@message:Von&pretty=true"

Please let me know if both the request are translated to same search
or different?

Thanks.

On Monday, January 21, 2013 2:22:24 PM UTC-5, Jimi Parekh wrote:
http://localhost:9200/_search?q=@message:Von&pretty=true

Hi,

I have included the query dsl in a json file. Below is the exact how
my "input.json" looks like (Please note the use of '*' before and after the
text:

{
"query" : {
"term" : { "@message" : "Von" }
}
}

Belwo is my curl command:

curl -XGET -d @input.json "
http://localhost:9200/_search?q=@message:Von&pretty=true
http://localhost:9200/mass-**idx**/_search"

But it don't find any result. There are many documents having text
'Von' in the '@message' field, but still it don't find anything.

Is there anything I am doing wrong?

Thanks.

http://localhost:9200/mass-idx/_search

--

http://localhost:9200/mass-idx/_search

http://localhost:9200/mass-idx/_search

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
--
--
--

--

egaumer · January 23, 2013, 4:42pm

You shouldn't have to remove the '@' sign from your field names. This is
common in tools that translate XML to JSON where the XML attributes are
prepended with '@'.

http://json-lib.sourceforge.net/usage.html#xml

I've used these tools in the past and indexed the resulting content without
issue. You problem is likely due to improper analysis of the field.

On Wednesday, January 23, 2013 10:34:57 AM UTC-5, Jimi Parekh wrote:

Thanks Matt and David.

I am very new to ES so I am not aware with the lot of things. I am going
through the docs on elasticsearch site, but not getting clearity in many
things.

As I mention in my earlier post, the documents I am indexing are passed by
the 'logstash' to elasticsearch. logstash reads the application eventlog
and pass them to elasticsearch for indexing and storage. Thats where it
adds the metadata like @source, @host etc. The acutal log content is the
value of message field only.

Is there a way I can remove @ before indexing? How can I seperate
different fileds of 'message'? Does elasticsearch provide some setting for
those?

On Wednesday, January 23, 2013 3:29:34 AM UTC-5, David Pilato wrote:
Please follow Matt's advices:

avoid special chars in field names, even if it could work, I see it as
a bad practice

create a mapping before indexing the first document and set the right
analyzer for your use case or set it as not analyzed but not sure it will
help for your use case

If the content of @message is always like:
TrackID=value|Receive-Time=value|…, why not modify it before you index it
in valuable fields:
trackid=XXXXX
receivetime=xxxxxx

You will be able then to do searches on theses fields or compute facets.

David

Le 22 janv. 2013 à 20:41, Matt Weber matt....@gmail.com a écrit :

Did you set your @message field to not analyzed?

On Tue, Jan 22, 2013 at 11:11 AM, Jimi Parekh jimip...@gmail.com wrote:
Matt,
I tried your suggestion but the result is same as previous.

On Tuesday, January 22, 2013 12:05:51 PM UTC-5, Matt Weber wrote:
I imagine it is the @ symbol in your field names are being tokenized
via the query_string parser. Try a match query or remove the @symbol from
your field names. Also, if you are only doing exact matches like that,
make sure your fields are not analyzed.

{
"query": {
"match": {
"@message": {
"query": "Track-Id=1234"
}
}
}
}

On Tue, Jan 22, 2013 at 8:30 AM, Jimi Parekh jimip...@gmail.comwrote:
David,

I am trying out few things on this and I getting more confuse. I also
looked into the elasticsearch document but I don't find the proper answer.

Let me explain my scenario and the issue i am facing.

In my indexed document there is a field "@message", with the following
kind of value.

"@message:Track-Id=1234|Token-Id=ABC123XYZ|Receive-Time-MS=
1358423498000".
There are other field also like "@source", "@host" etc.
I have about 300 documents indexed.

Now I want to search for the documents where "@message" field has a
string "Track-Id=1234".
Now as per your previous suggestion I create a following search:

curl -XGET 'http://localhost:9200/_search**' -d '{
"query" : {
"query_string" : {
"query" : "@message:Track-Id=1234"
}
}
}'

But this returned me all the documents. What wrong I am doing? What is
the better approach for such queries, Query-DSL or Query string?

Thanks.
On Monday, January 21, 2013 5:11:31 PM UTC-5, Jimi Parekh wrote:
David,

I tried your suggestion and it seems working but not giving the
expected result.
I want to search the "@message" field and wherever that field contain
the string 'Von' anywhere in that field I want all those documents.

We can not use wildcard with query_string? I want to use term query
then what will be the exact query syntax.?

Thanks.

On Monday, January 21, 2013 3:36:54 PM UTC-5, David Pilato wrote:
Try this:

$ curl -XGET 'http://localhost:9200/_search****' -d '{
"query" : {
"query_string" : {
"query" : "@message:Von"
}
}
}'

I mean that your query doesn't go through the analysis process.
When you index a document, content is analyzed (break into tokens,
lowercased, filtered...) before being indexed.

Von is indexed as von in the inverted index.
If you use a matchQuery or a QueryString it goes through the same
process. So, if you search for Von, in fact you will search for von in the
inverted index and you will find your document.

Make sense?

Le 21 janvier 2013 à 21:15, Jimi Parekh jimip...@gmail.com a
écrit :

David,

I am new to ES. what do you mean by 'TermQuery is not analyzed"?

I tried to use below two different json format but both gives
parsing error.

{
"query_string" : {
    "default_field":"@message", 
    "query" : "Von" 
} 
}

{
"query_string" : {
    "query" : "@message:Von" 
} 
}

What will be the exact json for my query?

Thanks.

On Monday, January 21, 2013 3:01:10 PM UTC-5, David Pilato wrote:

TermQuery is not analyzed. So ES try to compare "Von" to "von"
which does not match.
Use a QueryString if you want to do the same search as q=...

HTH

Le 21 janvier 2013 à 20:52, Jimi Parekh < jimip...@gmail.com> a
écrit :

Additional Info: When I use query string and pass the same condition
it gives me the desire result. Below is the curl request with query string.

curl -XGET " http://localhost:9200/_search?****q=@message:Von&pretty=true"

Please let me know if both the request are translated to same search
or different?

Thanks.

On Monday, January 21, 2013 2:22:24 PM UTC-5, Jimi Parekh wrote:
http://localhost:9200/_search?q=@message:Von&pretty=true

Hi,

I have included the query dsl in a json file. Below is the exact how
my "input.json" looks like (Please note the use of '*' before and after the
text:

{
"query" : {
"term" : { "@message" : "Von" }
}
}

Belwo is my curl command:

curl -XGET -d @input.json "
http://localhost:9200/_search?q=@message:Von&pretty=true
http://localhost:9200/mass-**idx**/_search"

But it don't find any result. There are many documents having text
'Von' in the '@message' field, but still it don't find anything.

Is there anything I am doing wrong?

Thanks.

http://localhost:9200/mass-idx/_search

--

http://localhost:9200/mass-idx/_search

http://localhost:9200/mass-idx/_search

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
--
--
--

--

Jimi_Parekh · January 23, 2013, 6:23pm

egaumer,

You are right. I am not able to understand how exactly the document was
indexed and how to build the search DSL.

On Wednesday, January 23, 2013 11:42:43 AM UTC-5, egaumer wrote:

You shouldn't have to remove the '@' sign from your field names. This is
common in tools that translate XML to JSON where the XML attributes are
prepended with '@'.

Maven - Json-lib::Getting Started

I've used these tools in the past and indexed the resulting content
without issue. You problem is likely due to improper analysis of the field.

On Wednesday, January 23, 2013 10:34:57 AM UTC-5, Jimi Parekh wrote:
Thanks Matt and David.

I am very new to ES so I am not aware with the lot of things. I am going
through the docs on elasticsearch site, but not getting clearity in many
things.

As I mention in my earlier post, the documents I am indexing are passed
by the 'logstash' to elasticsearch. logstash reads the application eventlog
and pass them to elasticsearch for indexing and storage. Thats where it
adds the metadata like @source, @host etc. The acutal log content is the
value of message field only.

Is there a way I can remove @ before indexing? How can I seperate
different fileds of 'message'? Does elasticsearch provide some setting for
those?

On Wednesday, January 23, 2013 3:29:34 AM UTC-5, David Pilato wrote:
Please follow Matt's advices:

avoid special chars in field names, even if it could work, I see it as
a bad practice

create a mapping before indexing the first document and set the right
analyzer for your use case or set it as not analyzed but not sure it will
help for your use case

If the content of @message is always like:
TrackID=value|Receive-Time=value|…, why not modify it before you index it
in valuable fields:
trackid=XXXXX
receivetime=xxxxxx

You will be able then to do searches on theses fields or compute facets.

David

Le 22 janv. 2013 à 20:41, Matt Weber matt....@gmail.com a écrit :

Did you set your @message field to not analyzed?

On Tue, Jan 22, 2013 at 11:11 AM, Jimi Parekh jimip...@gmail.comwrote:
Matt,
I tried your suggestion but the result is same as previous.

On Tuesday, January 22, 2013 12:05:51 PM UTC-5, Matt Weber wrote:
I imagine it is the @ symbol in your field names are being tokenized
via the query_string parser. Try a match query or remove the @symbol from
your field names. Also, if you are only doing exact matches like that,
make sure your fields are not analyzed.

{
"query": {
"match": {
"@message": {
"query": "Track-Id=1234"
}
}
}
}

On Tue, Jan 22, 2013 at 8:30 AM, Jimi Parekh jimip...@gmail.comwrote:
David,

I am trying out few things on this and I getting more confuse. I also
looked into the elasticsearch document but I don't find the proper answer.

Let me explain my scenario and the issue i am facing.

In my indexed document there is a field "@message", with the
following kind of value.

"@message:Track-Id=1234|Token-Id=ABC123XYZ|Receive-Time-MS=
1358423498000".
There are other field also like "@source", "@host" etc.
I have about 300 documents indexed.

Now I want to search for the documents where "@message" field has a
string "Track-Id=1234".
Now as per your previous suggestion I create a following search:

curl -XGET 'http://localhost:9200/_search**' -d '{
"query" : {
"query_string" : {
"query" : "@message:Track-Id=1234"
}
}
}'

But this returned me all the documents. What wrong I am doing? What
is the better approach for such queries, Query-DSL or Query string?

Thanks.
On Monday, January 21, 2013 5:11:31 PM UTC-5, Jimi Parekh wrote:
David,

I tried your suggestion and it seems working but not giving the
expected result.
I want to search the "@message" field and wherever that field
contain the string 'Von' anywhere in that field I want all those documents.

We can not use wildcard with query_string? I want to use term query
then what will be the exact query syntax.?

Thanks.

On Monday, January 21, 2013 3:36:54 PM UTC-5, David Pilato wrote:
Try this:

$ curl -XGET 'http://localhost:9200/_search****' -d '{
"query" : {
"query_string" : {
"query" : "@message:Von"
}
}
}'

I mean that your query doesn't go through the analysis process.
When you index a document, content is analyzed (break into
tokens, lowercased, filtered...) before being indexed.

Von is indexed as von in the inverted index.
If you use a matchQuery or a QueryString it goes through the same
process. So, if you search for Von, in fact you will search for von in the
inverted index and you will find your document.

Make sense?

Le 21 janvier 2013 à 21:15, Jimi Parekh jimip...@gmail.com a
écrit :

David,

I am new to ES. what do you mean by 'TermQuery is not analyzed"?

I tried to use below two different json format but both gives
parsing error.

{
"query_string" : {
    "default_field":"@message", 
    "query" : "Von" 
} 
}

{
"query_string" : {
    "query" : "@message:Von" 
} 
}

What will be the exact json for my query?

Thanks.

On Monday, January 21, 2013 3:01:10 PM UTC-5, David Pilato wrote:

TermQuery is not analyzed. So ES try to compare "Von" to "von"
which does not match.
Use a QueryString if you want to do the same search as q=...

HTH

Le 21 janvier 2013 à 20:52, Jimi Parekh < jimip...@gmail.com> a
écrit :

Additional Info: When I use query string and pass the same
condition it gives me the desire result. Below is the curl request with
query string.

curl -XGET " http://localhost:9200/_search?****q=@message:Von&pretty=true"

Please let me know if both the request are translated to same
search or different?

Thanks.

On Monday, January 21, 2013 2:22:24 PM UTC-5, Jimi Parekh wrote:
http://localhost:9200/_search?q=@message:Von&pretty=true

Hi,

I have included the query dsl in a json file. Below is the exact
how my "input.json" looks like (Please note the use of '*' before and after
the text:

{
"query" : {
"term" : { "@message" : "Von" }
}
}

Belwo is my curl command:

curl -XGET -d @input.json "
http://localhost:9200/_search?q=@message:Von&pretty=true
http://localhost:9200/mass-**idx**/_search"

But it don't find any result. There are many documents having text
'Von' in the '@message' field, but still it don't find anything.

Is there anything I am doing wrong?

Thanks.

http://localhost:9200/mass-idx/_search

--

http://localhost:9200/mass-idx/_search

http://localhost:9200/mass-idx/_search

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
--
--
--

--

dadoonet · January 24, 2013, 8:19am

Try this:
curl -XGET 'localhost:9200/_analyze?analyzer=standard' -d 'Track-ID=3269795|Receive-Time=2013-01-16 21:02:23|Receive-Time-MS=1358370143000|Token-ID=APA91bGYkO29UHYzFxaUlEBCo4FDjZpzsTGdlhCSTGXbxGri8YSKY0fk9jH2N60Qxfp8RFbiJEnRXcCrv5zteK0UQvX2SwgHuABMluupGyUYEaxxB_Mxwjifm2vYnrQMh_QT_ZEBMpKY6OJnTtglawipHIE9ftyy9Q|User-ID=0v_odd99nf_u3facz9d|Client-Type=type|Client-Version=2.00.0|P-Time=2013-01-17 11:57:01|P-Time-MS=1358423821000|Current-Time-MS=1358416621248'

It will show you how Elasticsearch analyze it with the standard analyzer (default) and index your field.

Le 23 janv. 2013 à 19:23, Jimi Parekh jimiparekh@gmail.com a écrit :

egaumer,

You are right. I am not able to understand how exactly the document was indexed and how to build the search DSL.

On Wednesday, January 23, 2013 11:42:43 AM UTC-5, egaumer wrote:
You shouldn't have to remove the '@' sign from your field names. This is common in tools that translate XML to JSON where the XML attributes are prepended with '@'.

Maven - Json-lib::Getting Started

I've used these tools in the past and indexed the resulting content without issue. You problem is likely due to improper analysis of the field.

On Wednesday, January 23, 2013 10:34:57 AM UTC-5, Jimi Parekh wrote:
Thanks Matt and David.

I am very new to ES so I am not aware with the lot of things. I am going through the docs on elasticsearch site, but not getting clearity in many things.

As I mention in my earlier post, the documents I am indexing are passed by the 'logstash' to elasticsearch. logstash reads the application eventlog and pass them to elasticsearch for indexing and storage. Thats where it adds the metadata like @source, @host etc. The acutal log content is the value of message field only.

Is there a way I can remove @ before indexing? How can I seperate different fileds of 'message'? Does elasticsearch provide some setting for those?

On Wednesday, January 23, 2013 3:29:34 AM UTC-5, David Pilato wrote:
Please follow Matt's advices:

avoid special chars in field names, even if it could work, I see it as a bad practice

create a mapping before indexing the first document and set the right analyzer for your use case or set it as not analyzed but not sure it will help for your use case

If the content of @message is always like: TrackID=value|Receive-Time=value|…, why not modify it before you index it in valuable fields:
trackid=XXXXX
receivetime=xxxxxx

You will be able then to do searches on theses fields or compute facets.

David

Le 22 janv. 2013 à 20:41, Matt Weber matt....@gmail.com a écrit :
Did you set your @message field to not analyzed?

On Tue, Jan 22, 2013 at 11:11 AM, Jimi Parekh jimip...@gmail.com wrote:
Matt,
I tried your suggestion but the result is same as previous.

On Tuesday, January 22, 2013 12:05:51 PM UTC-5, Matt Weber wrote:
I imagine it is the @ symbol in your field names are being tokenized via the query_string parser. Try a match query or remove the @symbol from your field names. Also, if you are only doing exact matches like that, make sure your fields are not analyzed.

{
"query": {
"match": {
"@message": {
"query": "Track-Id=1234"
}
}
}
}

On Tue, Jan 22, 2013 at 8:30 AM, Jimi Parekh jimip...@gmail.com wrote:
David,

I am trying out few things on this and I getting more confuse. I also looked into the elasticsearch document but I don't find the proper answer.

Let me explain my scenario and the issue i am facing.

In my indexed document there is a field "@message", with the following kind of value.

"@message:Track-Id=1234|Token-Id=ABC123XYZ|Receive-Time-MS=1358423498000".
There are other field also like "@source", "@host" etc.
I have about 300 documents indexed.

Now I want to search for the documents where "@message" field has a string "Track-Id=1234".
Now as per your previous suggestion I create a following search:

curl -XGET 'http://localhost:9200/_search' -d '{
"query" : {
"query_string" : {
"query" : "@message:Track-Id=1234"
}
}
}'

But this returned me all the documents. What wrong I am doing? What is the better approach for such queries, Query-DSL or Query string?

Thanks.
On Monday, January 21, 2013 5:11:31 PM UTC-5, Jimi Parekh wrote:
David,

I tried your suggestion and it seems working but not giving the expected result.
I want to search the "@message" field and wherever that field contain the string 'Von' anywhere in that field I want all those documents.

We can not use wildcard with query_string? I want to use term query then what will be the exact query syntax.?

Thanks.

On Monday, January 21, 2013 3:36:54 PM UTC-5, David Pilato wrote:
Try this:

$ curl -XGET 'http://localhost:9200/_search' -d '{
"query" : { 
    "query_string" : { 
        "query" : "@message:Von" 
    } 
} 
}'

I mean that your query doesn't go through the analysis process.
When you index a document, content is analyzed (break into tokens, lowercased, filtered...) before being indexed.

Von is indexed as von in the inverted index.
If you use a matchQuery or a QueryString it goes through the same process. So, if you search for Von, in fact you will search for von in the inverted index and you will find your document.

Make sense?

Le 21 janvier 2013 à 21:15, Jimi Parekh jimip...@gmail.com a écrit :
David,

I am new to ES. what do you mean by 'TermQuery is not analyzed"?

I tried to use below two different json format but both gives parsing error.

{
"query_string" : {
    "default_field":"@message", 
    "query" : "Von" 
} 
}

{
"query_string" : {
    "query" : "@message:Von" 
} 
}

What will be the exact json for my query?

Thanks.

On Monday, January 21, 2013 3:01:10 PM UTC-5, David Pilato wrote:
TermQuery is not analyzed. So ES try to compare "Von" to "von" which does not match.
Use a QueryString if you want to do the same search as q=...

HTH

Le 21 janvier 2013 à 20:52, Jimi Parekh < jimip...@gmail.com> a écrit :

Additional Info: When I use query string and pass the same condition it gives me the desire result. Below is the curl request with query string.

curl -XGET " http://localhost:9200/_search?q=@message:Von&pretty=true"

Please let me know if both the request are translated to same search or different?

Thanks.

On Monday, January 21, 2013 2:22:24 PM UTC-5, Jimi Parekh wrote:
Hi,

I have included the query dsl in a json file. Below is the exact how my "input.json" looks like (Please note the use of '*' before and after the text:

{
"query" : {
"term" : { "@message" : "Von" }
}
}

Belwo is my curl command:

curl -XGET -d @input.json " http://localhost:9200/mass-idx/_search"

But it don't find any result. There are many documents having text 'Von' in the '@message' field, but still it don't find anything.

Is there anything I am doing wrong?

Thanks.

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--
--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

--

--
--

--