Query filter is not working


(bagui) #1

Hi,

I've a index in my elasticsearch like below

{

  • "_index":"logstash-2014.03.03",
  • "_type":"apache-access",
  • "_id":"snCPRnSHSvm_aaeuHxB84w",
  • "_version":1,
  • "found":true,
  • "_source":{
    • "message":"\tat
      org.apache.http.client.protocol.RequestProxyAuthentication.process(RequestProxyAuthentication.java:89)"
      ,
    • "@version":"1",
    • "@timestamp":"2014-03-03T18:39:35.425+05:30",
    • "type":"apache-access",
    • "host":"cloudclient.aricent.com",
    • "path":"/opt/apache-tomcat-7.0.40/logs/aricloud/monitoring.log"
      }

}

I’m trying to qrery the data using filter and below is the query

{
"query": {
"constant_score": {
"filter": {
"term": { "message": "RequestProxyAuthentication" }
}
}
}
}

but the same is giving be null result from index search.
{

  • "took":275,
  • "timed_out":false,
  • "_shards":{
    • "total":5,
    • "successful":5,
    • "failed":0
      },
  • "hits":{
    • "total":0,
    • "max_score":null,
    • "hits":[
      ]
      }

}

Kindly let mw know how to pass the query string

-Subhadip

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/586cb192-0abe-443b-9d4c-3d0669b89bcd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #2

lowercase your term filter.
TermFilter is not analyzed.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 11 mars 2014 à 09:18, Subhadip Bagui i.bagui@gmail.com a écrit :

Hi,

I've a index in my elasticsearch like below

{
"_index":"logstash-2014.03.03",
"_type":"apache-access",
"_id":"snCPRnSHSvm_aaeuHxB84w",
"_version":1,
"found":true,
"_source":{
"message":"\tat org.apache.http.client.protocol.RequestProxyAuthentication.process(RequestProxyAuthentication.java:89)",
"@version":"1",
"@timestamp":"2014-03-03T18:39:35.425+05:30",
"type":"apache-access",
"host":"cloudclient.aricent.com",
"path":"/opt/apache-tomcat-7.0.40/logs/aricloud/monitoring.log"
}
}

I’m trying to qrery the data using filter and below is the query

{
"query": {
"constant_score": {
"filter": {
"term": { "message": "RequestProxyAuthentication" }
}
}
}
}

but the same is giving be null result from index search.
{
"took":275,
"timed_out":false,
"_shards":{
"total":5,
"successful":5,
"failed":0
},
"hits":{
"total":0,
"max_score":null,
"hits":[
]
}
}

Kindly let mw know how to pass the query string

-Subhadip

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/586cb192-0abe-443b-9d4c-3d0669b89bcd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4CA390E6-0367-4821-A9B1-A3CA8E61E800%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.


(bagui) #3

Hi David,

Trying to query as following, but still getting null result. Please suggest.

{
"query": {
"constant_score": {
"filter": {
"term": { "message": "requestproxyauthentication" }
}
}
}
}

-Subhadip

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a9d10f48-d5fd-4578-b885-cfe0eb0fa3e1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #4

what is your mapping?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 11 mars 2014 à 11:03:00, Subhadip Bagui (i.bagui@gmail.com) a écrit:

Hi David,

Trying to query as following, but still getting null result. Please suggest.

{
"query": {
"constant_score": {
"filter": {
"term": { "message": "requestproxyauthentication" }
}
}
}
}

-Subhadip

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a9d10f48-d5fd-4578-b885-cfe0eb0fa3e1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.531ee305.7fdcc233.b095%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.


(bagui) #5

mapping...

{

  • "movies":{
    • "mappings":{
      • "movie":{
        • "properties":{
          • "director":{
            • "type":"string",
            • "fields":{
              • "original":{
                • "type":"string",
                • "index":"not_analyzed"
                  }
                  }
                  },
          • "genres":{
            • "type":"string"
              },
          • "query":{
            • "properties":{
              • "query_string":{
                • "properties":{
                  • "query":{
                    • "type":"string"
                      }
                      }
                      }
                      }
                      },
          • "title":{
            • "type":"string"
              },
          • "year":{
            • "type":"long"
              }
              }
              }
              }
              }

}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5ea73df0-1ed6-4b4e-823f-ebc2bed6e46b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(bagui) #6

mapping...default

{

  • "logstash-2014.03.03":{
    • "mappings":{
      • "apache-access":{
        • "dynamic_templates":[
          • {
            • "string_fields":{
              • "mapping":{
                • "type":"multi_field",
                • "fields":{
                  • "raw":{
                    • "index":"not_analyzed",
                    • "ignore_above":256,
                    • "type":"string"
                      },
                  • "{name}":{
                    • "index":"analyzed",
                    • "omit_norms":true,
                    • "type":"string"
                      }
                      }
                      },
              • "match":"*",
              • "match_mapping_type":"string"
                }
                }
                ],
        • "properties":{
          • "@timestamp":{
            • "type":"date",
            • "format":"dateOptionalTime"
              },
          • "@version":{
            • "type":"string",
            • "index":"not_analyzed"
              },
          • "geoip":{
            • "dynamic":"true",
            • "properties":{
              • "location":{
                • "type":"geo_point"
                  }
                  }
                  },
          • "host":{
            • "type":"string",
            • "norms":{
              • "enabled":false
                },
            • "fields":{
              • "raw":{
                • "type":"string",
                • "index":"not_analyzed",
                • "ignore_above":256
                  }
                  }
                  },
          • "message":{
            • "type":"string",
            • "norms":{
              • "enabled":false
                },
            • "fields":{
              • "raw":{
                • "type":"string",
                • "index":"not_analyzed",
                • "ignore_above":256
                  }
                  }
                  },
          • "path":{
            • "type":"string",
            • "norms":{
              • "enabled":false
                },
            • "fields":{
              • "raw":{
                • "type":"string",
                • "index":"not_analyzed",
                • "ignore_above":256
                  }
                  }
                  },
          • "query":{
            • "properties":{
              • "constant_score":{
                • "properties":{
                  • "filter":{
                    • "properties":{
                      • "term":{
                        • "properties":{
                          • "@timestamp":{
                            • "type":"date",
                            • "format":"dateOptionalTime"
                              }
                              }
                              }
                              }
                              }
                              }
                              }
                              }
                              },
          • "type":{
            • "type":"string",
            • "norms":{
              • "enabled":false
                },
            • "fields":{
              • "raw":{
                • "type":"string",
                • "index":"not_analyzed",
                • "ignore_above":256
                  }
                  }
                  }
                  }
                  }
                  }
                  }

}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f3873f82-8000-4f35-8a3e-b2e29d7410f5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #7

So message seems to use a default analyzer…

May be you could try to reproduce your concern with a full curl recreation which:

delete test index,
create index
put template
create a doc
refresh
query

It could help to understand what's wrong here. I'm probably missing something.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 11 mars 2014 à 11:49:00, Subhadip Bagui (i.bagui@gmail.com) a écrit:

mapping...default

{

"logstash-2014.03.03":{

"mappings":{

"apache-access":{

"dynamic_templates":[

{

"string_fields":{

"mapping":{

"type":"multi_field",

"fields":{

"raw":{

"index":"not_analyzed",

"ignore_above":256,

"type":"string"

},

"{name}":{

"index":"analyzed",

"omit_norms":true,

"type":"string"

}

}

},

"match":"*",

"match_mapping_type":"string"

}

}

],

"properties":{

"@timestamp":{

"type":"date",

"format":"dateOptionalTime"

},

"@version":{

"type":"string",

"index":"not_analyzed"

},

"geoip":{

"dynamic":"true",

"properties":{

"location":{

"type":"geo_point"

}

}

},

"host":{

"type":"string",

"norms":{

"enabled":false

},

"fields":{

"raw":{

"type":"string",

"index":"not_analyzed",

"ignore_above":256

}

}

},

"message":{

"type":"string",

"norms":{

"enabled":false

},

"fields":{

"raw":{

"type":"string",

"index":"not_analyzed",

"ignore_above":256

}

}

},

"path":{

"type":"string",

"norms":{

"enabled":false

},

"fields":{

"raw":{

"type":"string",

"index":"not_analyzed",

"ignore_above":256

}

}

},

"query":{

"properties":{

"constant_score":{

"properties":{

"filter":{

"properties":{

"term":{

"properties":{

"@timestamp":{

"type":"date",

"format":"dateOptionalTime"

}

}

}

}

}

}

}

}

},

"type":{

"type":"string",

"norms":{

"enabled":false

},

"fields":{

"raw":{

"type":"string",

"index":"not_analyzed",

"ignore_above":256

}

}

}

}

}

}

}

}

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f3873f82-8000-4f35-8a3e-b2e29d7410f5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.531ef342.4e6afb66.b095%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.


(bagui) #8

Hi David,

I have done like below for a test sample.

  1. deleted index.

  2. create index by following
    curl -XPUT "http://localhost:9200/movies/" -d
    '{ "index": {"_index": "movies", "_type": "movie", "_id": "1"}}'

  3. creating doc
    curl -XPUT "http://localhost:9200/movies/movie/1" -d
    '{
    "title": "The Godfather",
    "director": "Francis Ford Coppola",
    "year": 1972,
    "genres": ["Crime", "Drama"]
    }'

  4. creating mapping
    curl -XPUT "http://localhost:9200/movies/movie/_mapping" -d
    '{
    "movie": {
    "properties": {
    "director": {
    "type": "multi_field",
    "fields": {
    "director": {"type": "string"},
    "original": {"type" : "string", "index" : "not_analyzed"}
    }
    }
    }
    }
    }'

  5. query
    curl -XPOST "http://localhost:9200/_search" -d'
    {
    "query": {
    "constant_score": {
    "filter": {
    "term": { "director.original": "Francis Ford Coppola" }
    }
    }
    }
    }'

Please let me know what am I missing. I checked for template.

-Subhadip

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7a736c32-34e7-4e7d-8a3d-c431c5c42175%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #9

This example has nothing in common with your data!

That said, you need to apply the mapping before indexing any document!

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 11 mars 2014 à 14:47:34, Subhadip Bagui (i.bagui@gmail.com) a écrit:

Hi David,

I have done like below for a test sample.

  1. deleted index.

  2. create index by following
    curl -XPUT "http://localhost:9200/movies/" -d
    '{ "index": {"_index": "movies", "_type": "movie", "_id": "1"}}'

  3. creating doc
    curl -XPUT "http://localhost:9200/movies/movie/1" -d
    '{
    "title": "The Godfather",
    "director": "Francis Ford Coppola",
    "year": 1972,
    "genres": ["Crime", "Drama"]
    }'

  4. creating mapping
    curl -XPUT "http://localhost:9200/movies/movie/_mapping" -d
    '{
    "movie": {
    "properties": {
    "director": {
    "type": "multi_field",
    "fields": {
    "director": {"type": "string"},
    "original": {"type" : "string", "index" : "not_analyzed"}
    }
    }
    }
    }
    }'

  5. query
    curl -XPOST "http://localhost:9200/_search" -d'
    {
    "query": {
    "constant_score": {
    "filter": {
    "term": { "director.original": "Francis Ford Coppola" }
    }
    }
    }
    }'

Please let me know what am I missing. I checked for template.

-Subhadip

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7a736c32-34e7-4e7d-8a3d-c431c5c42175%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.531f16b4.75a2a8d4.b095%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.


(Binh Ly-2) #10

The standard analyzer makes (RequestProxyAuthentication.java) into 1 term
and lowercases it. So this one should match it:

{
"query": {
"constant_score": {
"filter": {
"term": { "message": "requestproxyauthentication.java" }
}
}
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9d79a394-cd69-4967-a3aa-74d3195ea1cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(bagui) #11

Hi Binh,

The query you given is working. Thanks for your help.
But if I change the query and search for string "requestproxyauthentication"
instead, It's not working. Below is the mapping for message field.

I'm trying to understand how elasticsearch analyze the field data. Pls
comment.

"message":{

  • "type":"string",
  • "norms":{
    • "enabled":false
      },
  • "fields":{
    • "raw":{
      • "type":"string",
      • "index":"not_analyzed",
      • "ignore_above":256
        }
        }

}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fb30d7b5-11ab-4ee7-a3cb-647663b11623%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #12

Didn't you read my previous answer?

This example has nothing in common with your data!
That said, you need to apply the mapping before indexing any document!

We can't help you without a full curl recreation which actually reproduce your issue.
I think you are doing something wrong here but it's difficult to say without any clue.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 12 mars 2014 à 11:08:24, Subhadip Bagui (i.bagui@gmail.com) a écrit:

Hi Binh,

The query you given is working. Thanks for your help.
But if I change the query and search for string "requestproxyauthentication" instead, It's not working. Below is the mapping for message field.

I'm trying to understand how elasticsearch analyze the field data. Pls comment.

"message":{
"type":"string",
"norms":{
"enabled":false
},
"fields":{
"raw":{
"type":"string",
"index":"not_analyzed",
"ignore_above":256
}
}
}

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fb30d7b5-11ab-4ee7-a3cb-647663b11623%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.53203334.2ae8944a.bfa%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.


(bagui) #13

Hi David,

The data is coming through logstash and taking default mapping in
elasticsearch. Can I do update mapping for that index id ?
Pls let me know.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/dc119c29-264c-4e23-be8b-9a7ccd18ad47%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #14

May be this could help?
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-templates.html#indices-templates

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 12 mars 2014 à 11:25:53, Subhadip Bagui (i.bagui@gmail.com) a écrit:

Hi David,

The data is coming through logstash and taking default mapping in elasticsearch. Can I do update mapping for that index id ?
Pls let me know.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/dc119c29-264c-4e23-be8b-9a7ccd18ad47%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.53203724.79e2a9e3.bfa%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.


(bagui) #15

Hi David,

I have done following steps u suggested. The exact string search is working
now.
But when I'm trying the below query for string matching it's giving null
result.

May this is very basic and I'm doing something wrong. I'm a week old on
elasticsearch and trying to understand the query-sql and text search. Pls
help to clear the conception.

create index
create mapping
create a doc
refresh
query

query==>
{
"query": {
"constant_score": {
"filter": {
"term": { "message.original":
"org.apache.http.protocol.immutablehttpprocessor.process" }
}
}
}
}

mapping ==>
{

  • "log-2014.03.03":{
    • "mappings":{
      • "apache-log":{
        • "properties":{
          • "@timestamp":{
            • "type":"date",
            • "format":"yyyy-MM-dd HH:mm:ss"
              },
          • "@version":{
            • "type":"long"
              },
          • "host":{
            • "type":"string",
            • "index":"not_analyzed"
              },
          • "message":{
            • "type":"string",
            • "fields":{
              • "actual":{
                • "type":"string",
                • "index":"not_analyzed"
                  }
                  }
                  },
          • "path":{
            • "type":"string",
            • "index":"not_analyzed"
              },
          • "type":{
            • "type":"string",
            • "index":"not_analyzed"
              }
              }
              }
              }
              }

}

doc ==>
{

  • "_index":"log-2014.03.03",
  • "_type":"apache-log",
  • "_id":"5",
  • "_version":1,
  • "found":true,
  • "_source":{
    • "message":"org.apache.http.protocol.ImmutableHttpProcessor.process",
    • "@version":"3",
    • "@timestamp":"2014-03-03 18:45:35",
    • "type":"apache-access",
    • "host":"cloudclient.aricent.com",
    • "path":"/opt/apache-tomcat-7.0.40/logs/aricloud/monitoring.log"
      }

}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/74e0cc0e-25db-47cf-8524-cc1c151a7a76%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(bagui) #16

Hi David,

I have done following steps u suggested. The string search is working now.

But for filter I have to always pass strings in lowercase; where as for
query text search I can give the proper string sequence inserted in doc.
query shown below.

May be this is very basic and I'm doing something wrong. I'm a week old on
elasticsearch and trying to understand the query-sql and text search. Pls
help to clear the conception.

  1. curl -XDELETE http://10.203.251.142:9200/log-2014.03.03

curl -XPUT http://10.203.251.142:9200/log-2014.03.03/ -d
'{
"settings": {
"index": {
"number_of_shards": 3,
"number_of_replicas": 0,
"index.cache.field.type": "soft",
"index.refresh_interval": "30s",
"index.store.compress.stored": true
}
},
"mappings": {
"apache-log": {
"properties": {
"message": {
"type": "string",
"fields": {
"actual": {
"type": "string",
"index": "not_analyzed"
}
}
},
"@version": {
"type": "long",
"index": "not_analyzed"
},
"@timestamp": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss",
"index": "not_analyzed"
},
"type": {
"type": "string",
"index": "not_analyzed"
},
"host": {
"type": "string",
"index": "not_analyzed"
},
"path": {
"type": "string",
"norms": {
"enabled": false
},
"index": "not_analyzed"
}
}
}
}
}'

  1. curl -XPUT http://10.203.251.142:9200/_bulk -d '
    { "index": {"_index": "log-2014.03.03", "_type": "apache-log", "_id": "1"}}
    { "message": "03-03-2014 18:39:35,025 DEBUG
    [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-8]
    com.aricent.aricloud.monitoring.CloudController 121 -
    com.sun.jersey.core.spi.factory.ResponseImpl@1139f1b","@version":
    "1","@timestamp": "2014-03-03 18:39:35","type": "apache-access", "host":
    "cloudclient.aricent.com", "path":
    "/opt/apache-tomcat-7.0.40/logs/aricloud/monitoring.log" }
    { "index": {"_index": "log-2014.03.03", "_type": "apache-log", "_id": "2"}}
    { "message": "\tat org.quartz.core.JobRunShell.run(JobRunShell.java:223)",
    "@version": "1", "@timestamp": "2014-03-03 18:39:36","type":
    "apache-access", "host": "cloudclient.aricent.com", "path":
    "/opt/apache-tomcat-7.0.40/logs/aricloud/monitoring.log" }
    { "index": {"_index": "log-2014.03.03", "_type": "apache-log", "_id": "3"}}
    { "message": "03-03-2014 18:39:35,030 INFO
    [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-8]
    com.amazonaws.http.HttpClientFactory 128 - Configuring Proxy. Proxy Host:
    10.203.193.227 Proxy Port: 80", "@version": "2", "@timestamp": "2014-03-03
    18:40:35", "type": "apache-access", "host": "cloudclient.aricent.com",
    "path": "/opt/apache-tomcat-7.0.40/logs/aricloud/monitoring.log" }
    { "index": {"_index": "log-2014.03.03", "_type": "apache-log", "_id": "4"}}
    { "message": "\tat
    org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:109)",
    "@version": "3", "@timestamp": "2014-03-03 18:43:35", "type":
    "apache-access", "host": "cloudclient.aricent.com", "path":
    "/opt/apache-tomcat-7.0.40/logs/aricloud/monitoring.log" }
    { "index": {"_index": "log-2014.03.03", "_type": "apache-log", "_id": "5"}}
    { "message": "03-03-2014 18:45:30,002 DEBUG
    [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-9]
    com.aricent.aricloud.monitoring.scheduler.SchedulerJob 22 - Entering
    SchedulerJob", "@version": "3", "@timestamp": "2014-03-03 18:45:35",
    "type": "apache-access", "host": "cloudclient.aricent.com", "path":
    "/opt/apache-tomcat-7.0.40/logs/aricloud/monitoring.log" }
    \n'

  2. curl -XGET 'http://10.203.251.142:9200/log-2014.03.03/_refresh'

  3. query ==>
    curl -XPOST http://10.203.251.142:9200/log-2014.03.03/_search -d
    '{
    "query": {
    "match": {
    "message" : {
    "query": "Proxy Port",
    "type" : "phrase"
    }
    }
    }
    }'

null value -
curl -XPOST http://10.203.251.142:9200/log-2014.03.03/_search -d
'{
"query": {
"constant_score": {
"filter": {
"term": { "message": "DEBUG" }
}
}
}
}
'

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9daed515-6990-45a3-af47-b87d10bc44ae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #17

message field has been analyzed using standard analyzer. It means that you message content has been indexed using lowercase.
a Term Filter does not analyze your query.

"DEBUG" is <> than "debug".

If you want to find your term in the inverted index, you have either to analyze your query (matchQuery for example) or lowercase in that case your searched term.

curl -XPOST http://10.203.251.142:9200/log-2014.03.03/_search -d
'{
"query": {
"constant_score": {
"filter": {
"term": { "message": "debug" }
}
}
}
}
'

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 13 mars 2014 à 14:24:39, Subhadip Bagui (i.bagui@gmail.com) a écrit:

Hi David,

I have done following steps u suggested. The string search is working now.

But for filter I have to always pass strings in lowercase; where as for query text search I can give the proper string sequence inserted in doc. query shown below.

May be this is very basic and I'm doing something wrong. I'm a week old on elasticsearch and trying to understand the query-sql and text search. Pls help to clear the conception.

  1. curl -XDELETE http://10.203.251.142:9200/log-2014.03.03

curl -XPUT http://10.203.251.142:9200/log-2014.03.03/ -d
'{
"settings": {
"index": {
"number_of_shards": 3,
"number_of_replicas": 0,
"index.cache.field.type": "soft",
"index.refresh_interval": "30s",
"index.store.compress.stored": true
}
},
"mappings": {
"apache-log": {
"properties": {
"message": {
"type": "string",
"fields": {
"actual": {
"type": "string",
"index": "not_analyzed"
}
}
},
"@version": {
"type": "long",
"index": "not_analyzed"
},
"@timestamp": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss",
"index": "not_analyzed"
},
"type": {
"type": "string",
"index": "not_analyzed"
},
"host": {
"type": "string",
"index": "not_analyzed"
},
"path": {
"type": "string",
"norms": {
"enabled": false
},
"index": "not_analyzed"
}
}
}
}
}'

  1. curl -XPUT http://10.203.251.142:9200/_bulk -d '
    { "index": {"_index": "log-2014.03.03", "_type": "apache-log", "_id": "1"}}
    { "message": "03-03-2014 18:39:35,025 DEBUG [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-8] com.aricent.aricloud.monitoring.CloudController 121 - com.sun.jersey.core.spi.factory.ResponseImpl@1139f1b","@version": "1","@timestamp": "2014-03-03 18:39:35","type": "apache-access", "host": "cloudclient.aricent.com", "path": "/opt/apache-tomcat-7.0.40/logs/aricloud/monitoring.log" }
    { "index": {"_index": "log-2014.03.03", "_type": "apache-log", "_id": "2"}}
    { "message": "\tat org.quartz.core.JobRunShell.run(JobRunShell.java:223)", "@version": "1", "@timestamp": "2014-03-03 18:39:36","type": "apache-access", "host": "cloudclient.aricent.com", "path": "/opt/apache-tomcat-7.0.40/logs/aricloud/monitoring.log" }
    { "index": {"_index": "log-2014.03.03", "_type": "apache-log", "_id": "3"}}
    { "message": "03-03-2014 18:39:35,030 INFO [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-8] com.amazonaws.http.HttpClientFactory 128 - Configuring Proxy. Proxy Host: 10.203.193.227 Proxy Port: 80", "@version": "2", "@timestamp": "2014-03-03 18:40:35", "type": "apache-access", "host": "cloudclient.aricent.com", "path": "/opt/apache-tomcat-7.0.40/logs/aricloud/monitoring.log" }
    { "index": {"_index": "log-2014.03.03", "_type": "apache-log", "_id": "4"}}
    { "message": "\tat org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:109)", "@version": "3", "@timestamp": "2014-03-03 18:43:35", "type": "apache-access", "host": "cloudclient.aricent.com", "path": "/opt/apache-tomcat-7.0.40/logs/aricloud/monitoring.log" }
    { "index": {"_index": "log-2014.03.03", "_type": "apache-log", "_id": "5"}}
    { "message": "03-03-2014 18:45:30,002 DEBUG [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-9] com.aricent.aricloud.monitoring.scheduler.SchedulerJob 22 - Entering SchedulerJob", "@version": "3", "@timestamp": "2014-03-03 18:45:35", "type": "apache-access", "host": "cloudclient.aricent.com", "path": "/opt/apache-tomcat-7.0.40/logs/aricloud/monitoring.log" }
    \n'

  2. curl -XGET 'http://10.203.251.142:9200/log-2014.03.03/_refresh'

  3. query ==>
    curl -XPOST http://10.203.251.142:9200/log-2014.03.03/_search -d
    '{
    "query": {
    "match": {
    "message" : {
    "query": "Proxy Port",
    "type" : "phrase"
    }
    }
    }
    }'

null value -
curl -XPOST http://10.203.251.142:9200/log-2014.03.03/_search -d
'{
"query": {
"constant_score": {
"filter": {
"term": { "message": "DEBUG" }
}
}
}
}
'

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9daed515-6990-45a3-af47-b87d10bc44ae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.5321b914.216231b.158d%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.


(system) #18