Mapping modifications in index template breaks search queries

Elastic Stack Elasticsearch
1

I've got an Elasticsearch cluster of two machines both running version
0.20.2. Log data is fed into this cluster from Logstash.

This has been running for months without issue. I've recently realised that
when I upgraded from a single machine to this cluster I forgot to port over
my index template


. So I've implemented that index template which works fine and has
properties defined for the standard fields of syslog type data that
Logstash would ordinarily feed in. So up until this point everything is
working as expected.

The problems start when I implement my new index template


which has some additional fields specific to the nginx logs that come
through that same process, from Logstash and into the same Elasticsearch
index.

To explain my problem I'll use these two fields '@message' and the
'useragent'. Both have exactly the same mappings and both should as far as
I can see be searchable in the same way but they aren't. I can perform a
simple text search on '@message' and get results, if I run a similar search
on the 'useragent' field I get no results, even though there seems to be
data that should be a match. If I edit the index template and just remove
the 'useragent' mapping and then perform a similar search I get the results
back ok. Basically if I try and add mapping for any of these fields then
they are no longer searchable even though I'm setting the fields up to be
searchable in the same way as the '@message' field is defined.

For a second yesterday I thought sod it I will just not bother adding any
mapping for those new fields, those fields will get indexed automatically
and everything will be searchable, no problems. But I NEED to add mapping
for some of those other fields because I need things like upstrtime (which
is the time the upstream server takes to generate a PHP page) to be numeric
so that I can run statistics on that field.

For your info, whenever I make modifications to the index template I do so
on both servers by modifying the template file, then restart ES on both
servers, then once the cluster status is green I delete the index starting
fresh. I've double checked that the index template is taking effect by
running this command curl -XGET
'http://192.168.11.3:9200/logstash-2013.07.27/_mapping?pretty=true' and
everything looks fine.

Has anyone any idea what could be going on here?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Could you post a query which is supposée to work?
And what kind of values you have your field useragent?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 28 juil. 2013 à 22:08, simon_thepiman simon@wpsites.co.uk a écrit :

I've got an Elasticsearch cluster of two machines both running version 0.20.2. Log data is fed into this cluster from Logstash.

This has been running for months without issue. I've recently realised that when I upgraded from a single machine to this cluster I forgot to port over my index template https://gist.github.com/WPsites/4685598/7a750337e100d1474e7dc54c1406c8a1da5e4afc . So I've implemented that index template which works fine and has properties defined for the standard fields of syslog type data that Logstash would ordinarily feed in. So up until this point everything is working as expected.

The problems start when I implement my new index template https://gist.github.com/WPsites/4685598/cc2b240e26f476e163d97ea92ea9c88a9d5f082a which has some additional fields specific to the nginx logs that come through that same process, from Logstash and into the same Elasticsearch index.

To explain my problem I'll use these two fields '@message' and the 'useragent'. Both have exactly the same mappings and both should as far as I can see be searchable in the same way but they aren't. I can perform a simple text search on '@message' and get results, if I run a similar search on the 'useragent' field I get no results, even though there seems to be data that should be a match. If I edit the index template and just remove the 'useragent' mapping and then perform a similar search I get the results back ok. Basically if I try and add mapping for any of these fields then they are no longer searchable even though I'm setting the fields up to be searchable in the same way as the '@message' field is defined.

For a second yesterday I thought sod it I will just not bother adding any mapping for those new fields, those fields will get indexed automatically and everything will be searchable, no problems. But I NEED to add mapping for some of those other fields because I need things like upstrtime (which is the time the upstream server takes to generate a PHP page) to be numeric so that I can run statistics on that field.

For your info, whenever I make modifications to the index template I do so on both servers by modifying the template file, then restart ES on both servers, then once the cluster status is green I delete the index starting fresh. I've double checked that the index template is taking effect by running this command curl -XGET 'http://192.168.11.3:9200/logstash-2013.07.27/_mapping?pretty=true' and everything looks fine.

Has anyone any idea what could be going on here?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

The 'useragent' field contains a user agent string like this 'Mozilla/5.0
(Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72
Safari/537.36'

The same issue exists for any of the fields that I'm adding mappings for,
so something like the 'syslog_program' field which just contains a simple
string such as 'upstream.log' suffers from the same issue, so that's ruling
out an issue with special characters I think.

Thanks for your input David. I'm using Kibana to search through the logs
and you asking about my query has just made me think maybe there is an
issue with the Kibana config so I'm just going to take a quick look at
that. If I don't find an issue with that then I'll post some query examples
on here.

Thanks

On Sunday, July 28, 2013 9:45:14 PM UTC+1, David Pilato wrote:

Could you post a query which is supposée to work?
And what kind of values you have your field useragent?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 28 juil. 2013 à 22:08, simon_thepiman <si...@wpsites.co.uk<javascript:>>
a écrit :

I've got an Elasticsearch cluster of two machines both running version
0.20.2. Log data is fed into this cluster from Logstash.

This has been running for months without issue. I've recently realised
that when I upgraded from a single machine to this cluster I forgot to port
over my index template
Elasticsearch index template for logstash that contains additional NGINX fields · GitHub. So I've implemented that index template which works fine and has
properties defined for the standard fields of syslog type data that
Logstash would ordinarily feed in. So up until this point everything is
working as expected.

The problems start when I implement my new index template
https://gist.github.com/WPsites/4685598/cc2b240e26f476e163d97ea92ea9c88a9d5f082awhich has some additional fields specific to the nginx logs that come
through that same process, from Logstash and into the same Elasticsearch
index.

To explain my problem I'll use these two fields '@message' and the
'useragent'. Both have exactly the same mappings and both should as far as
I can see be searchable in the same way but they aren't. I can perform a
simple text search on '@message' and get results, if I run a similar search
on the 'useragent' field I get no results, even though there seems to be
data that should be a match. If I edit the index template and just remove
the 'useragent' mapping and then perform a similar search I get the results
back ok. Basically if I try and add mapping for any of these fields then
they are no longer searchable even though I'm setting the fields up to be
searchable in the same way as the '@message' field is defined.

For a second yesterday I thought sod it I will just not bother adding any
mapping for those new fields, those fields will get indexed automatically
and everything will be searchable, no problems. But I NEED to add mapping
for some of those other fields because I need things like upstrtime (which
is the time the upstream server takes to generate a PHP page) to be numeric
so that I can run statistics on that field.

For your info, whenever I make modifications to the index template I do so
on both servers by modifying the template file, then restart ES on both
servers, then once the cluster status is green I delete the index starting
fresh. I've double checked that the index template is taking effect by
running this command curl -XGET '
http://192.168.11.3:9200/logstash-2013.07.27/_mapping?pretty=true' and
everything looks fine.

Has anyone any idea what could be going on here?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

You did not set any analyzer. Your fields are analyzed with the standard analyzer (english analyzer).
I think that if you search for mozilla, you should get a result.

I will start to run queries using curl and only when everything work as you desire, start to play with Kibana.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 28 juil. 2013 à 23:36, simon_thepiman simon@wpsites.co.uk a écrit :

The 'useragent' field contains a user agent string like this 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36'

The same issue exists for any of the fields that I'm adding mappings for, so something like the 'syslog_program' field which just contains a simple string such as 'upstream.log' suffers from the same issue, so that's ruling out an issue with special characters I think.

Thanks for your input David. I'm using Kibana to search through the logs and you asking about my query has just made me think maybe there is an issue with the Kibana config so I'm just going to take a quick look at that. If I don't find an issue with that then I'll post some query examples on here.

Thanks

On Sunday, July 28, 2013 9:45:14 PM UTC+1, David Pilato wrote:

Could you post a query which is supposée to work?
And what kind of values you have your field useragent?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 28 juil. 2013 à 22:08, simon_thepiman si...@wpsites.co.uk a écrit :

I've got an Elasticsearch cluster of two machines both running version 0.20.2. Log data is fed into this cluster from Logstash.

This has been running for months without issue. I've recently realised that when I upgraded from a single machine to this cluster I forgot to port over my index template Elasticsearch index template for logstash that contains additional NGINX fields · GitHub . So I've implemented that index template which works fine and has properties defined for the standard fields of syslog type data that Logstash would ordinarily feed in. So up until this point everything is working as expected.

The problems start when I implement my new index template Elasticsearch index template for logstash that contains additional NGINX fields · GitHub which has some additional fields specific to the nginx logs that come through that same process, from Logstash and into the same Elasticsearch index.

To explain my problem I'll use these two fields '@message' and the 'useragent'. Both have exactly the same mappings and both should as far as I can see be searchable in the same way but they aren't. I can perform a simple text search on '@message' and get results, if I run a similar search on the 'useragent' field I get no results, even though there seems to be data that should be a match. If I edit the index template and just remove the 'useragent' mapping and then perform a similar search I get the results back ok. Basically if I try and add mapping for any of these fields then they are no longer searchable even though I'm setting the fields up to be searchable in the same way as the '@message' field is defined.

For a second yesterday I thought sod it I will just not bother adding any mapping for those new fields, those fields will get indexed automatically and everything will be searchable, no problems. But I NEED to add mapping for some of those other fields because I need things like upstrtime (which is the time the upstream server takes to generate a PHP page) to be numeric so that I can run statistics on that field.

For your info, whenever I make modifications to the index template I do so on both servers by modifying the template file, then restart ES on both servers, then once the cluster status is green I delete the index starting fresh. I've double checked that the index template is taking effect by running this command curl -XGET 'http://192.168.11.3:9200/logstash-2013.07.27/_mapping?pretty=true' and everything looks fine.

Has anyone any idea what could be going on here?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Hi David,

I've modified my index template slightly to make pretty much all of the
fields analyzed. For example my 'method' field is mapped like so:

"method": { "type": "string", "index": "analyzed" }

And then performing a pretty straight forward looking query directly on the
database:

curl --globoff
'http://192.168.11.2:9200/logstash-2013.07.28/_search?q=method:"GET"&pretty=true'

{
"took" : 6,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}

When I run a trend on the 'method' field it comes back with 491 'GET' and 5
'HEAD' so the data is there, it's just not coming back for a search, even
searching method:GET. I don't get it!?

Thanks,

Simon

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Try this:

curl -XGET 'localhost:9200/_analyze?analyzer=standard' -d 'GET
You will see how Elasticsearch index it using default analyzer.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juil. 2013 à 00:28, simon_thepiman simon@wpsites.co.uk a écrit :

Hi David,

I've modified my index template slightly to make pretty much all of the fields analyzed. For example my 'method' field is mapped like so:

"method": { "type": "string", "index": "analyzed" }

And then performing a pretty straight forward looking query directly on the database:

curl --globoff 'http://192.168.11.2:9200/logstash-2013.07.28/_search?q=method:"GET"&pretty=true'

{
"took" : 6,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}

When I run a trend on the 'method' field it comes back with 491 'GET' and 5 'HEAD' so the data is there, it's just not coming back for a search, even searching method:GET. I don't get it!?

Thanks,

Simon

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

curl -XGET '192.168.11.2:9200/_analyze?analyzer=standard&pretty=true' -d
'GET'

{
"tokens" : [ {
"token" : "get",
"start_offset" : 0,
"end_offset" : 3,
"type" : "",
"position" : 1
} ]
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

Get is a stop word in english analyzer. That's the reason it's not indexed.
You should use another analyzer for your field.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juil. 2013 à 00:43, simon_thepiman simon@wpsites.co.uk a écrit :

curl -XGET '192.168.11.2:9200/_analyze?analyzer=standard&pretty=true' -d 'GET'

{
"tokens" : [ {
"token" : "get",
"start_offset" : 0,
"end_offset" : 3,
"type" : "",
"position" : 1
} ]
}

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

9

Forget what I wrote.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juil. 2013 à 00:47, David Pilato david@pilato.fr a écrit :

Get is a stop word in english analyzer. That's the reason it's not indexed.
You should use another analyzer for your field.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juil. 2013 à 00:43, simon_thepiman simon@wpsites.co.uk a écrit :

curl -XGET '192.168.11.2:9200/_analyze?analyzer=standard&pretty=true' -d 'GET'

{
"tokens" : [ {
"token" : "get",
"start_offset" : 0,
"end_offset" : 3,
"type" : "",
"position" : 1
} ]
}

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

10

Try:

curl --globoff 'http://192.168.11.2:9200/logstash-2013.07.28/_search?q=method:get&pretty=true'

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juil. 2013 à 00:43, simon_thepiman simon@wpsites.co.uk a écrit :

curl -XGET '192.168.11.2:9200/_analyze?analyzer=standard&pretty=true' -d 'GET'

{
"tokens" : [ {
"token" : "get",
"start_offset" : 0,
"end_offset" : 3,
"type" : "",
"position" : 1
} ]
}

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

11

curl --globoff
'http://192.168.11.2:9200/logstash-2013.07.28/_search?q=method:get&pretty=true'

{
"took" : 10,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

12

Could you post a document which have GET in method field?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juil. 2013 à 00:52, simon_thepiman simon@wpsites.co.uk a écrit :

curl --globoff 'http://192.168.11.2:9200/logstash-2013.07.28/_search?q=method:get&pretty=true'

{
"took" : 10,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}
}

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

13

Thanks for your assistence here David.

Here is a document returned from a search of the '@message' field for GET

{
"_index" : "logstash-2013.07.28",
"_type" : "syslog",
"_id" : "JXX9wpWfTiSJ9yR7UZJuXA",
"_score" : 1.0, "_source" :
{"@source":"tcp://172.16.50.11:36874/","@tags":["nginx","nginx_upstream"],"@fields":{"syslog_pri":["13"],"syslog_timestamp":["Jul
28
22:21:51"],"syslog_hostname":["indesit1"],"syslog_program":["upstream.log"],"clientip":["90.244.96.35"],"time":["28/Jul/2013:22:21:51
+0000"],"host":["www.hotpointservice.co.uk"],"method":["GET"],"path":["/wp-content/themes/indesit/fonts/opensans/OpenSans-Regular-webfont.woff"],"httpversion":["1.1"],"response":["200"],"bytes":["13999"],"referrer":["http://www.hotpointservice.co.uk/manual/?id=EXFL1810G"],"useragent":["Mozilla/5.0
(Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72
Safari/537.36"],"upstr_ip":["127.0.0.1"],"upstr_port":["8000"],"upstrcode":["200"],"upstrtime":["0.002"]},"@timestamp":"2013-07-28T22:21:51.000Z","@source_host":"indesit1","@source_path":"/","@message":"<13>Jul
28 22:21:51 indesit1 upstream.log: :90.244.96.35 - - [28/Jul/2013:22:21:51
+0000] "www.hotpointservice.co.uk" "GET
/wp-content/themes/indesit/fonts/opensans/OpenSans-Regular-webfont.woff
HTTP/1.1" 200 13999
"http://www.hotpointservice.co.uk/manual/?id=EXFL1810G" "Mozilla/5.0
(Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72
Safari/537.36" "127.0.0.1:8000" 200 0.002\n","@type":"syslog"}
} ]
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

14

I've just run:
curl -XGET
'http://192.168.11.3:9200/logstash-2013.07.29/_mapping?pretty=true'
To confirm how the mapping is taking effect.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

15

What happens if you search for @fields.method:get ?
It should work.

But it sounds like you defined a mapping for method field and not for @fields.method.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 29 juil. 2013 à 01:20, simon_thepiman simon@wpsites.co.uk a écrit :

Thanks for your assistence here David.

Here is a document returned from a search of the '@message' field for GET

{
"_index" : "logstash-2013.07.28",
"_type" : "syslog",
"_id" : "JXX9wpWfTiSJ9yR7UZJuXA",
"_score" : 1.0, "_source" : {"@source":"tcp://172.16.50.11:36874/","@tags":["nginx","nginx_upstream"],"@fields":{"syslog_pri":["13"],"syslog_timestamp":["Jul 28 22:21:51"],"syslog_hostname":["indesit1"],"syslog_program":["upstream.log"],"clientip":["90.244.96.35"],"time":["28/Jul/2013:22:21:51 +0000"],"host":["www.hotpointservice.co.uk"],"method":["GET"],"path":["/wp-content/themes/indesit/fonts/opensans/OpenSans-Regular-webfont.woff"],"httpversion":["1.1"],"response":["200"],"bytes":["13999"],"referrer":["http://www.hotpointservice.co.uk/manual/?id=EXFL1810G"],"useragent":["Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36"],"upstr_ip":["127.0.0.1"],"upstr_port":["8000"],"upstrcode":["200"],"upstrtime":["0.002"]},"@timestamp":"2013-07-28T22:21:51.000Z","@source_host":"indesit1","@source_path":"/","@message":"<13>Jul 28 22:21:51 indesit1 upstream.log: :90.244.96.35 - - [28/Jul/2013:22:21:51 +0000] "www.hotpointservice.co.uk" "GET /wp-content/themes/indesit/fonts/opensans/OpenSans-Regular-webfont.woff HTTP/1.1" 200 13999 "http://www.hotpointservice.co.uk/manual/?id=EXFL1810G\" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36" "127.0.0.1:8000" 200 0.002\n","@type":"syslog"}
} ]
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

16

Amazing, that worked! I thought I was never going to get anywhere with
this. Thanks.

So do I need to add further mappings somehow to make this work on
method:GET rather than @fields.method:GET ?

I've seen nothing online about @fields.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

17

@fields comes from logstash.
I don't think you are pushing any value in method field.

My 2 cents

David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 30 juil. 2013 à 15:58, simon_thepiman simon@wpsites.co.uk a écrit :

Amazing, that worked! I thought I was never going to get anywhere with this. Thanks.

So do I need to add further mappings somehow to make this work on method:GET rather than @fields.method:GET ?

I've seen nothing online about @fields.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

18

Just wanted to say thanks again for your help David.

I've updated my gist with the working Elasticsearch index template that
takes into account the fact that additional fields created by grok in
Logstash are sent into Elasticsearch to be indexed in the '@fields' object

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.