Elasticsearch response does not have desired fields/data

Elastic Stack Elasticsearch
1

I have been using ElastAlert to monitor and alert on my Elasticsearch data. However the Elasticsearch response does not have the fields that I want to pull data from in my alerts. I have been using the DEBUG feature to view the response this is the result of two of my alerting rules:

This is the first alerting query:

2022-02-18 23:30:25,833    DEBUG        elasticsearch > {"query":{"bool":{"filter":{"bool":{"must":[{"range":{"@timestamp":{"gt":"2022-02-18T23:15:25.765064Z","lte":"2022-02-18T23:30:25.765064Z"}}},{"query_string":{"query":"\"Priority: Low\""}}]}}}},"sort":[{"@timestamp":{"order":"asc"}}]}
2022-02-18 23:30:25,833    DEBUG        elasticsearch < {"_scroll_id":"FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoARZxMkhzTnROclFBT2pzX01tUFM5R3N3AAAAAAFzWlUWTTNtcjR4eHdRRXl5ekVWNEptMEgzZw==","took":23,"timed_out":false,"_shards":{"total":2,"successful":2,"skipped":1,"failed":0},"hits":{"total":{"value":2,"relation":"eq"},"max_score":null,"hits":[{"_index":"logstash-2022.02.18-000419","_type":"_doc","_id":"XFQgD38BQXFqgSjg18JP","_score":null,"_source":{"@timestamp":"2022-02-18T23:18:03.684Z"},"sort":[1645226283684]},{"_index":"logstash-2022.02.18-000419","_type":"_doc","_id":"GVQjD38BQXFqgSjgYPxU","_score":null,"_source":{"@timestamp":"2022-02-18T23:20:50.756Z"},"sort":[1645226450756]}]}}

This is the second alerting query:

2022-02-22 18:42:48,894    DEBUG        elasticsearch > {"query":{"bool":{"filter":{"bool":{"must":[{"range":{"@timestamp":{"gt":"2022-02-22T18:27:48.820360Z","lte":"2022-02-22T18:42:48.820360Z"}}},{"query_string":{"query":"\"A user account was locked out\""}}]}}}},"sort":[{"@timestamp":{"order":"asc"}}]}
2022-02-22 18:42:48,895    DEBUG        elasticsearch < {"_scroll_id":"FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoARZxMkhzTnROclFBT2pzX01tUFM5R3N3AAAAAAG29ikWTTNtcjR4eHdRRXl5ekVWNEptMEgzZw==","took":66,"timed_out":false,"_shards":{"total":2,"successful":2,"skipped":1,"failed":0},"hits":{"total":{"value":1,"relation":"eq"},"max_score":null,"hits":[{"_index":"winlogbeat-7.10.0-2022.02.22-000251","_type":"_doc","_id":"d1OrIn8BQXFqgSjgZ93Z","_score":null,"_source":{"@timestamp":"2022-02-22T18:31:28.864Z","message":"A user account was locked out.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-18\n\tAccount Name:\t\tDCON3$\n\tAccount Domain:\t\tUHTASI\n\tLogon ID:\t\t0x3E7\n\nAccount That Was Locked Out:\n\tSecurity ID:\t\tS-1-5-21-3693063922-990322624-829951171-1332\n\tAccount Name:\t\tadmin\n\nAdditional Information:\n\tCaller Computer Name:\tDCON3"},"sort":[1645554688864]}]}}

Overall it looks like I can utilize the fields that are in the _source{} field in the DEBUG logs. Is there any way that I can add fields to that _source field, or a better way to access the fields that I do not have currently? I can view the fields that I want to have in my Kibana Discover tool.

2

_source is what was in the original document that was passed to Elasticsearch, so it's up to you to add things to that if that's what you want to have in a response.

1 Like
3

Thank you. So how do I do that? I'm assuming it won't be in Kibana.

Actually would the second example in this link work?

4

How are you indexing data into Elasticsearch?

5

I am sending data through Logstash and Winlogbeat. My ILM policies create daily logstash/winlogbeat indices.

6

OK, then you want Logstash or Winlogbeat to add the data in that you are missing, so that it shows up in Elasticsearch.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.