# Disappearing data

**URL:** <https://discuss.elastic.co/t/disappearing-data/49414>\
**Category:** Elasticsearch\
**Created:** [May 6, 2016, 2:19pm UTC](https://discuss.elastic.co/t/disappearing-data/49414 "2016-05-06T14:19:20Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![iamthealex](https://avatars.discourse-cdn.com/v4/letter/i/e9c0ed/32.png) [@iamthealex](https://discuss.elastic.co/u/iamthealex)\
**Post date:** [May 6, 2016, 2:19pm UTC](https://discuss.elastic.co/t/disappearing-data/49414/1 "2016-05-06T14:19:20Z")

</div>

I have distilled my problem down to something fairly small.  
I'm sure I'm missing something basic.  
I am using ES version 2.3.2 and LogStash version 2.3.2.  
I have a LogStash config that generates a trivial message with one (top-level json) field of interest.  
I have an ES mapping that disables source and turns off analysis for that field.  
I delete the index and then push a single event into LS which then inserts the event into ES.  
When I search in ES, that field has disappeared.  
If I re-enable source in the mapping for the index, the field is there under source, but I expected the field to be at the top-level json returned from the search as well.  
I am using this particular index for metrics so I don't want source.

Here is the mapping for the "test-search" index:  
{  
"mappings": {  
"v1": {  
"\_source": { "enabled": false },  
"properties": {  
"@timestamp": {  
"type": "date",  
"format": "strict\_date\_optional\_time||epoch\_millis"  
},  
"udn-env": {  
"type" : "string",  
"index": "not\_analyzed"  
}  
}  
}  
}  
}  
I installed this mapping, by blowing away the index, and the re-creating the index using this mapping:  
% curl -X DELETE localhost:9200/test-search  
% curl -X PUT -d @/tmp/mapping localhost:9200/test-search

Here is my logstash config:

# Use stdin to trigger an event.

input { stdin{} }

filter {

# Ignore the @message and any other dreck input plugins might add.

# Add a single top-level field "udn-env" : "cdx-local"

mutate {  
remove\_field =\> ["host", "headers", "path", "@version"]  
add\_field =\> { "[udn-env]" =\> "cdx-local" }  
}

}

output {  
stdout {  
codec =\> rubydebug {  
metadata =\> true  
}  
}

# Everything goes to ES.

elasticsearch {  
hosts =\>"localhost:9200"  
max\_retries =\> 1  
ssl =\> false

```
index => "test-search"
document_type => "v1"
document_id => "42"

```

}

}  
I start logstash like this:  
bin/logstash -f search.conf

Here is the the search command and results:  
[http://localhost:9200/test-search/\_search](http://localhost:9200/test-search/_search)  
{

```
"took": ​10,
"timed_out": false,
"_shards": 

```

{

```
"total": ​5,
"successful": ​5,
"failed": ​0

```

},  
"hits":  
{

```
"total": ​1,
"max_score": ​1.0,
"hits": 

```

[

```
        {
            "_index": "test-search",
            "_type": "v1",
            "_id": "42",
            "_score": ​1.0
        }
    ]
}

```

}

Where is my udn-env field?  
I can see it in the output from LS using the stdout output plugin:  
I left in message as well just to show that fields are not being propagated.

Pipeline main started  
asdf  
{  
"message" =\> "asdf",  
"@timestamp" =\> "2016-05-06T14:07:59.963Z",  
"udn-env" =\> "cdx-local"  
}

---

<div class="post-metadata">

**Author:** ![iamthealex](https://avatars.discourse-cdn.com/v4/letter/i/e9c0ed/32.png) [@iamthealex](https://discuss.elastic.co/u/iamthealex)\
**Post date:** [May 6, 2016, 4:02pm UTC](https://discuss.elastic.co/t/disappearing-data/49414/2 "2016-05-06T16:02:41Z")

</div>

I think I start to get a glimmer of understanding.  
I think that the search api with no filters will return the entry, but will not print out non-analyzed fields.  
I think the non-analyzed fields are there (despite being absent in the search api output) because searching for  
'{ "query" : {"term" : { "env" : "dang" } } } '  
returns zero hits,  
but searching for  
'{ "query" : {"term" : { "env" : "local" } } } '  
returns a hit.

Is there an API that will print the non-analyzed fields?

---

<div class="post-metadata">

**Author:** ![iamthealex](https://avatars.discourse-cdn.com/v4/letter/i/e9c0ed/32.png) [@iamthealex](https://discuss.elastic.co/u/iamthealex)\
**Post date:** [May 6, 2016, 6:38pm UTC](https://discuss.elastic.co/t/disappearing-data/49414/3 "2016-05-06T18:38:56Z")

</div>

I got an answer in the ES IRC channel.  
Thanks to "dakrone".  
The following will print out the data of a non-analyzed field named "env":

curl -XGET [http://localhost:9200/test-search/\_search](http://localhost:9200/test-search/_search) -d '{ "query": { "match\_all": {} }, "fielddata\_fields": ["env"] }'

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:53pm UTC](https://discuss.elastic.co/t/disappearing-data/49414/4 "2017-07-05T22:53:16Z")

</div>