I have distilled my problem down to something fairly small.
I'm sure I'm missing something basic.
I am using ES version 2.3.2 and LogStash version 2.3.2.
I have a LogStash config that generates a trivial message with one (top-level json) field of interest.
I have an ES mapping that disables source and turns off analysis for that field.
I delete the index and then push a single event into LS which then inserts the event into ES.
When I search in ES, that field has disappeared.
If I re-enable source in the mapping for the index, the field is there under source, but I expected the field to be at the top-level json returned from the search as well.
I am using this particular index for metrics so I don't want source.
Here is the mapping for the "test-search" index:
{
"mappings": {
"v1": {
"_source": { "enabled": false },
"properties": {
"@timestamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"udn-env": {
"type" : "string",
"index": "not_analyzed"
}
}
}
}
}
I installed this mapping, by blowing away the index, and the re-creating the index using this mapping:
% curl -X DELETE localhost:9200/test-search
% curl -X PUT -d @/tmp/mapping localhost:9200/test-search
Here is my logstash config:
Use stdin to trigger an event.
input { stdin{} }
filter {
Ignore the @message and any other dreck input plugins might add.
Add a single top-level field "udn-env" : "cdx-local"
mutate {
remove_field => ["host", "headers", "path", "@version"]
add_field => { "[udn-env]" => "cdx-local" }
}
}
output {
stdout {
codec => rubydebug {
metadata => true
}
}
Everything goes to ES.
elasticsearch {
hosts =>"localhost:9200"
max_retries => 1
ssl => false
index => "test-search"
document_type => "v1"
document_id => "42"
}
}
I start logstash like this:
bin/logstash -f search.conf
Here is the the search command and results:
http://localhost:9200/test-search/_search
{
"took": 10,
"timed_out": false,
"_shards":
{
"total": 5,
"successful": 5,
"failed": 0
},
"hits":
{
"total": 1,
"max_score": 1.0,
"hits":
[
{
"_index": "test-search",
"_type": "v1",
"_id": "42",
"_score": 1.0
}
]
}
}
Where is my udn-env field?
I can see it in the output from LS using the stdout output plugin:
I left in message as well just to show that fields are not being propagated.
Pipeline main started
asdf
{
"message" => "asdf",
"@timestamp" => "2016-05-06T14:07:59.963Z",
"udn-env" => "cdx-local"
}