# Issues with searching data in array

**URL:** <https://discuss.elastic.co/t/issues-with-searching-data-in-array/204420>\
**Category:** Elasticsearch\
**Created:** [October 21, 2019, 9:03am UTC](https://discuss.elastic.co/t/issues-with-searching-data-in-array/204420 "2019-10-21T09:03:57Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![karthiknpy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karthiknpy/32/53587_2.png) [@karthiknpy](https://discuss.elastic.co/u/karthiknpy)\
**Post date:** [October 21, 2019, 9:03am UTC](https://discuss.elastic.co/t/issues-with-searching-data-in-array/204420/1 "2019-10-21T09:03:57Z")

</div>

Hi Team,

Issues in searching the data in kibana. It is not working as expected. The situation is

1. The kind of logs which are collected are firewall logs

2. There are 2 fields Action and matches.action

3. Action specifies the end action taken by firewall and matches.action specifies the preliminary action taken. The values can be either one or more of (log, allow, drop, challenge, simulate)

4. Since I dont want 2 fields for the same kind of data. I merged matches.action to action which makes the field action an array.

5. When I'm filtering for a data it is showing what else is not required is also shown.

Eg: when searching for "drop" traffic it is showing "simulate" traffic also as well.  
Refer the the image below

 ![pie%20chart](https://us1.discourse-cdn.com/elastic/original/3X/3/f/3f882e9309a1e5d6ed9059cca17cfda15817ce25.png)

Data in the filed is like

![screenshot](https://us1.discourse-cdn.com/elastic/original/3X/3/e/3e89226c04f9d41f8df32cf2480ace1a49e9d65a.png)

I will be grateful if anyone helps me with this.

Regards  
Karthik.

---

<div class="post-metadata">

**Author:** ![KrishnaShah7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krishnashah7/32/66810_2.png) [@KrishnaShah7](https://discuss.elastic.co/u/KrishnaShah7)\
**Post date:** [October 21, 2019, 11:02am UTC](https://discuss.elastic.co/t/issues-with-searching-data-in-array/204420/2 "2019-10-21T11:02:54Z")

</div>

When you say "I merged matches.action to action which makes the field action an array.", did you use nested data types for doing that ?

---

<div class="post-metadata">

**Author:** ![karthiknpy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karthiknpy/32/53587_2.png) [@karthiknpy](https://discuss.elastic.co/u/karthiknpy)\
**Post date:** [October 21, 2019, 11:43am UTC](https://discuss.elastic.co/t/issues-with-searching-data-in-array/204420/3 "2019-10-21T11:43:05Z")

</div>

Hey Krishna,

Thank you for your helping hand.

I didn't use any nested datatype its just a text datatype upon simple merge operation. It would have been a array now.

Can you please guide me what to do. Or feel free to ask any more info required from my end.

Thanks  
Karthik. K

---

<div class="post-metadata">

**Author:** ![KrishnaShah7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krishnashah7/32/66810_2.png) [@KrishnaShah7](https://discuss.elastic.co/u/KrishnaShah7)\
**Post date:** [October 22, 2019, 2:58am UTC](https://discuss.elastic.co/t/issues-with-searching-data-in-array/204420/4 "2019-10-22T02:58:45Z")

</div>

If you have an array of objects under one entity, it is recommended to map data in a nested way. Could you please show how does your mapping looks like for "action" field ?

---

<div class="post-metadata">

**Author:** ![karthiknpy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karthiknpy/32/53587_2.png) [@karthiknpy](https://discuss.elastic.co/u/karthiknpy)\
**Post date:** [October 22, 2019, 5:35am UTC](https://discuss.elastic.co/t/issues-with-searching-data-in-array/204420/5 "2019-10-22T05:35:28Z")

</div>

Hi Krishna,

My index mapping for the field action is as follows.

```
"action" : {
      "type" : "text",
      "fields" : {
        "key" : {
          "type" : "keyword"
        }
      }
    }

```

I'm using logstash http\_input plugin to ingest data to elastic search from a python script. I have a logstash if condition which does the merge operation which is as follows

```
if [action] == [matches][action] {
mutate {
  remove_field => "[matches][action]"
  }
}
else {
  mutate {
    merge => { "action" => "[matches][action]" }
    remove_field => "[matches][action]"
  }
}

```

Could you please advise what are the changes I need to make in the mapping and the logstash config as I'm new to this ?

The Goal is I need the field action to be queriable and should be able to generate visualizations based on that.

Thanks  
Karthik.K

---

<div class="post-metadata">

**Author:** ![KrishnaShah7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krishnashah7/32/66810_2.png) [@KrishnaShah7](https://discuss.elastic.co/u/KrishnaShah7)\
**Post date:** [October 22, 2019, 11:09am UTC](https://discuss.elastic.co/t/issues-with-searching-data-in-array/204420/6 "2019-10-22T11:09:56Z")

</div>

Your logstash config looks good, but once your array type of data enters elasticsearch, it will convert the data into flat structure. Meaning, your relationship of the array's objects will be lost. That is the reason why you're getting the hits for match field when you're interested in match.action field and vice versa.  
Try denormalising your data before you index it to elasticsearch, rather than keeping it as an array.  
Arrays of objects do not work as you would expect: you cannot query each object independently of the other objects in the array. If you need to be able to do this then you should use the [nested datatypes](https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html) where in you can define your mapping for your usecase in a below way:  
Eg. Create a mapping for your index where you map your action field to "action.second" and your preliminary action (matches.action) to action.first.

> ```
> PUT my_index
> {
> "mappings": {
> "properties": {
> "action": {
> "type": "nested" 
> }
> }
> }
> }
> 
> ```

When you start indexing, you could potentially tweak your logstash config file, in a way which indexes your data in the below manner:

> ```
> PUT my_index/_doc/1
> {
> "actions" : [
> {
> "first" : "firewall_sampleAction",
> "second" : "log"
> },
> {
> "first" : "firewall_sampleAction2",
> "second" : "drop"
> }
> ]
> }
> 
> ```

So now when you query, your queries could be framed as:\

> ```
> GET my_index/_search
> {
> query": {
> "nested": {
> "path": "action",
> "query": {
> "must": [
> { "match": { "action.first": "firewall_sampleAction2" }}
> ]
> }
> }
> }
> 
> ```

Note: if you're end goal is also to create visualisations/dashboards using this data, then you could avoid using these nested datatypes, as Kibana is yet to provide support for those. In that case, try denormalising your data.

---

<div class="post-metadata">

**Author:** ![karthiknpy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karthiknpy/32/53587_2.png) [@karthiknpy](https://discuss.elastic.co/u/karthiknpy)\
**Post date:** [October 22, 2019, 12:03pm UTC](https://discuss.elastic.co/t/issues-with-searching-data-in-array/204420/7 "2019-10-22T12:03:44Z")

</div>

I have a question here. Can a nested data type can hold data with same key name and different value

```auto
PUT my_index/_doc/1
{
  "actions" : [
    {
      "actiontaken" : "allow",
    },
    {
      "actiontaken" : "drop",
    }
  ]
}

```

where allow and drop refers to the 2 actions taken by firewall ?

---

<div class="post-metadata">

**Author:** ![KrishnaShah7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krishnashah7/32/66810_2.png) [@KrishnaShah7](https://discuss.elastic.co/u/KrishnaShah7)\
**Post date:** [October 22, 2019, 10:01pm UTC](https://discuss.elastic.co/t/issues-with-searching-data-in-array/204420/8 "2019-10-22T22:01:41Z")

</div>

I don't think that will work as expected. As when you try to use that field in queries by accessing it as "actions.actiontaken" , it wouldn't know which value to specifically consider.

---

<div class="post-metadata">

**Author:** ![karthiknpy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karthiknpy/32/53587_2.png) [@karthiknpy](https://discuss.elastic.co/u/karthiknpy)\
**Post date:** [October 23, 2019, 4:41am UTC](https://discuss.elastic.co/t/issues-with-searching-data-in-array/204420/9 "2019-10-23T04:41:13Z")

</div>

Ok understood, Could you please help me with few lines of logstash config how can I add data to a Nested data type

---

<div class="post-metadata">

**Author:** ![karthiknpy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karthiknpy/32/53587_2.png) [@karthiknpy](https://discuss.elastic.co/u/karthiknpy)\
**Post date:** [November 8, 2019, 12:25pm UTC](https://discuss.elastic.co/t/issues-with-searching-data-in-array/204420/12 "2019-11-08T12:25:30Z")

</div>

Hi Krishna,

Requesting you with the above.

Regards  
Karthik.K

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 6, 2019, 12:25pm UTC](https://discuss.elastic.co/t/issues-with-searching-data-in-array/204420/13 "2019-12-06T12:25:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.