Disappearing data

Elastic Stack Elasticsearch
1

I have distilled my problem down to something fairly small.
I'm sure I'm missing something basic.
I am using ES version 2.3.2 and LogStash version 2.3.2.
I have a LogStash config that generates a trivial message with one (top-level json) field of interest.
I have an ES mapping that disables source and turns off analysis for that field.
I delete the index and then push a single event into LS which then inserts the event into ES.
When I search in ES, that field has disappeared.
If I re-enable source in the mapping for the index, the field is there under source, but I expected the field to be at the top-level json returned from the search as well.
I am using this particular index for metrics so I don't want source.

Here is the mapping for the "test-search" index:
{
"mappings": {
"v1": {
"_source": { "enabled": false },
"properties": {
"@timestamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"udn-env": {
"type" : "string",
"index": "not_analyzed"
}
}
}
}
}
I installed this mapping, by blowing away the index, and the re-creating the index using this mapping:
% curl -X DELETE localhost:9200/test-search
% curl -X PUT -d @/tmp/mapping localhost:9200/test-search

Here is my logstash config:

Use stdin to trigger an event.

input { stdin{} }

filter {

Ignore the @message and any other dreck input plugins might add.

Add a single top-level field "udn-env" : "cdx-local"

mutate {
remove_field => ["host", "headers", "path", "@version"]
add_field => { "[udn-env]" => "cdx-local" }
}

}

output {
stdout {
codec => rubydebug {
metadata => true
}
}

Everything goes to ES.

elasticsearch {
hosts =>"localhost:9200"
max_retries => 1
ssl => false

index         => "test-search"
document_type => "v1"
document_id   => "42"

}

}
I start logstash like this:
bin/logstash -f search.conf

Here is the the search command and results:
http://localhost:9200/test-search/_search
{

"took": ​10,
"timed_out": false,
"_shards":

{

"total": ​5,
"successful": ​5,
"failed": ​0

},
"hits":
{

"total": ​1,
"max_score": ​1.0,
"hits":

[

        {
            "_index": "test-search",
            "_type": "v1",
            "_id": "42",
            "_score": ​1.0
        }
    ]
}

}

Where is my udn-env field?
I can see it in the output from LS using the stdout output plugin:
I left in message as well just to show that fields are not being propagated.

Pipeline main started
asdf
{
"message" => "asdf",
"@timestamp" => "2016-05-06T14:07:59.963Z",
"udn-env" => "cdx-local"
}

2

I think I start to get a glimmer of understanding.
I think that the search api with no filters will return the entry, but will not print out non-analyzed fields.
I think the non-analyzed fields are there (despite being absent in the search api output) because searching for
'{ "query" : {"term" : { "env" : "dang" } } } '
returns zero hits,
but searching for
'{ "query" : {"term" : { "env" : "local" } } } '
returns a hit.

Is there an API that will print the non-analyzed fields?

3

I got an answer in the ES IRC channel.
Thanks to "dakrone".
The following will print out the data of a non-analyzed field named "env":

curl -XGET http://localhost:9200/test-search/_search -d '{ "query": { "match_all": {} }, "fielddata_fields": ["env"] }'