# Elasticsearch response does not have desired fields/data

**URL:** <https://discuss.elastic.co/t/elasticsearch-response-does-not-have-desired-fields-data/298237>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [February 25, 2022, 4:07am UTC](https://discuss.elastic.co/t/elasticsearch-response-does-not-have-desired-fields-data/298237 "2022-02-25T04:07:47Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![roman-tasi](https://avatars.discourse-cdn.com/v4/letter/r/7ea924/32.png) [@roman-tasi](https://discuss.elastic.co/u/roman-tasi)\
**Post date:** [February 25, 2022, 4:07am UTC](https://discuss.elastic.co/t/elasticsearch-response-does-not-have-desired-fields-data/298237/1 "2022-02-25T04:07:48Z")

</div>

I have been using ElastAlert to monitor and alert on my Elasticsearch data. However the Elasticsearch response does not have the fields that I want to pull data from in my alerts. I have been using the `DEBUG` feature to view the response this is the result of two of my alerting rules:

This is the first alerting query:

```auto
2022-02-18 23:30:25,833 DEBUG elasticsearch > {"query":{"bool":{"filter":{"bool":{"must":[{"range":{"@timestamp":{"gt":"2022-02-18T23:15:25.765064Z","lte":"2022-02-18T23:30:25.765064Z"}}},{"query_string":{"query":"\"Priority: Low\""}}]}}}},"sort":[{"@timestamp":{"order":"asc"}}]}
2022-02-18 23:30:25,833 DEBUG elasticsearch < {"_scroll_id":"FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoARZxMkhzTnROclFBT2pzX01tUFM5R3N3AAAAAAFzWlUWTTNtcjR4eHdRRXl5ekVWNEptMEgzZw==","took":23,"timed_out":false,"_shards":{"total":2,"successful":2,"skipped":1,"failed":0},"hits":{"total":{"value":2,"relation":"eq"},"max_score":null,"hits":[{"_index":"logstash-2022.02.18-000419","_type":"_doc","_id":"XFQgD38BQXFqgSjg18JP","_score":null,"_source":{"@timestamp":"2022-02-18T23:18:03.684Z"},"sort":[1645226283684]},{"_index":"logstash-2022.02.18-000419","_type":"_doc","_id":"GVQjD38BQXFqgSjgYPxU","_score":null,"_source":{"@timestamp":"2022-02-18T23:20:50.756Z"},"sort":[1645226450756]}]}}

```

This is the second alerting query:

```auto
2022-02-22 18:42:48,894 DEBUG elasticsearch > {"query":{"bool":{"filter":{"bool":{"must":[{"range":{"@timestamp":{"gt":"2022-02-22T18:27:48.820360Z","lte":"2022-02-22T18:42:48.820360Z"}}},{"query_string":{"query":"\"A user account was locked out\""}}]}}}},"sort":[{"@timestamp":{"order":"asc"}}]}
2022-02-22 18:42:48,895 DEBUG elasticsearch < {"_scroll_id":"FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoARZxMkhzTnROclFBT2pzX01tUFM5R3N3AAAAAAG29ikWTTNtcjR4eHdRRXl5ekVWNEptMEgzZw==","took":66,"timed_out":false,"_shards":{"total":2,"successful":2,"skipped":1,"failed":0},"hits":{"total":{"value":1,"relation":"eq"},"max_score":null,"hits":[{"_index":"winlogbeat-7.10.0-2022.02.22-000251","_type":"_doc","_id":"d1OrIn8BQXFqgSjgZ93Z","_score":null,"_source":{"@timestamp":"2022-02-22T18:31:28.864Z","message":"A user account was locked out.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-18\n\tAccount Name:\t\tDCON3$\n\tAccount Domain:\t\tUHTASI\n\tLogon ID:\t\t0x3E7\n\nAccount That Was Locked Out:\n\tSecurity ID:\t\tS-1-5-21-3693063922-990322624-829951171-1332\n\tAccount Name:\t\tadmin\n\nAdditional Information:\n\tCaller Computer Name:\tDCON3"},"sort":[1645554688864]}]}}

```

Overall it looks like I can utilize the fields that are in the `_source{}` field in the `DEBUG` logs. Is there any way that I can add fields to that `_source` field, or a better way to access the fields that I do not have currently? I can view the fields that I want to have in my Kibana Discover tool.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 27, 2022, 11:40pm UTC](https://discuss.elastic.co/t/elasticsearch-response-does-not-have-desired-fields-data/298237/2 "2022-02-27T23:40:02Z")

</div>

`_source` is what was in the original document that was passed to Elasticsearch, so it's up to you to add things to that if that's what you want to have in a response.

---

<div class="post-metadata">

**Author:** ![roman-tasi](https://avatars.discourse-cdn.com/v4/letter/r/7ea924/32.png) [@roman-tasi](https://discuss.elastic.co/u/roman-tasi)\
**Post date:** [February 28, 2022, 7:49am UTC](https://discuss.elastic.co/t/elasticsearch-response-does-not-have-desired-fields-data/298237/3 "2022-02-28T07:49:09Z")

</div>

Thank you. So how do I do that? I'm assuming it won't be in Kibana.

Actually would the second example in this link work?

> **[Elasticsearch \_Source - Source Filtering, Storing Selected Fields & More](https://opster.com/guides/elasticsearch/glossary/elasticsearch-source/)**
>
> Elasticsearch keeps the original JSON document in a field called \_source. The source field serves special purposes such as...

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 28, 2022, 9:12am UTC](https://discuss.elastic.co/t/elasticsearch-response-does-not-have-desired-fields-data/298237/4 "2022-02-28T09:12:59Z")

</div>

How are you indexing data into Elasticsearch?

---

<div class="post-metadata">

**Author:** ![roman-tasi](https://avatars.discourse-cdn.com/v4/letter/r/7ea924/32.png) [@roman-tasi](https://discuss.elastic.co/u/roman-tasi)\
**Post date:** [February 28, 2022, 10:06am UTC](https://discuss.elastic.co/t/elasticsearch-response-does-not-have-desired-fields-data/298237/5 "2022-02-28T10:06:29Z")

</div>

I am sending data through Logstash and Winlogbeat. My ILM policies create daily logstash/winlogbeat indices.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 28, 2022, 9:58pm UTC](https://discuss.elastic.co/t/elasticsearch-response-does-not-have-desired-fields-data/298237/6 "2022-02-28T21:58:16Z")

</div>

OK, then you want Logstash or Winlogbeat to add the data in that you are missing, so that it shows up in Elasticsearch.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 28, 2022, 9:59pm UTC](https://discuss.elastic.co/t/elasticsearch-response-does-not-have-desired-fields-data/298237/7 "2022-03-28T21:59:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.