I've got an Elasticsearch cluster of two machines both running version 0.20.2. Log data is fed into this cluster from Logstash.
This has been running for months without issue. I've recently realised that when I upgraded from a single machine to this cluster I forgot to port over my index template https://gist.github.com/WPsites/4685598/7a750337e100d1474e7dc54c1406c8a1da5e4afc . So I've implemented that index template which works fine and has properties defined for the standard fields of syslog type data that Logstash would ordinarily feed in. So up until this point everything is working as expected.
The problems start when I implement my new index template https://gist.github.com/WPsites/4685598/cc2b240e26f476e163d97ea92ea9c88a9d5f082a which has some additional fields specific to the nginx logs that come through that same process, from Logstash and into the same Elasticsearch index.
To explain my problem I'll use these two fields '@message' and the 'useragent'. Both have exactly the same mappings and both should as far as I can see be searchable in the same way but they aren't. I can perform a simple text search on '@message' and get results, if I run a similar search on the 'useragent' field I get no results, even though there seems to be data that should be a match. If I edit the index template and just remove the 'useragent' mapping and then perform a similar search I get the results back ok. Basically if I try and add mapping for any of these fields then they are no longer searchable even though I'm setting the fields up to be searchable in the same way as the '@message' field is defined.
For a second yesterday I thought sod it I will just not bother adding any mapping for those new fields, those fields will get indexed automatically and everything will be searchable, no problems. But I NEED to add mapping for some of those other fields because I need things like upstrtime (which is the time the upstream server takes to generate a PHP page) to be numeric so that I can run statistics on that field.
For your info, whenever I make modifications to the index template I do so on both servers by modifying the template file, then restart ES on both servers, then once the cluster status is green I delete the index starting fresh. I've double checked that the index template is taking effect by running this command curl -XGET 'http://192.168.11.3:9200/logstash-2013.07.27/_mapping?pretty=true' and everything looks fine.
Has anyone any idea what could be going on here?