# Kibana cloudtrail visualization

**URL:** <https://discuss.elastic.co/t/kibana-cloudtrail-visualization/111864>\
**Category:** Kibana\
**Created:** [December 14, 2017, 9:36pm UTC](https://discuss.elastic.co/t/kibana-cloudtrail-visualization/111864 "2017-12-14T21:36:35Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\
**Post date:** [December 14, 2017, 9:36pm UTC](https://discuss.elastic.co/t/kibana-cloudtrail-visualization/111864/1 "2017-12-14T21:36:36Z")

</div>

Hi Team,  
I am forwarding aws cloudtrail logs to ES-5.5 and it is forwarded by logstash-5 (using cloudtrail plugin). While preparing visualizations, i realized that for some events "responseElements" field is not organized, it's in array format.

e.g.: i am expecting fields:  
responseElements.ipAddressType  
responseElements.vpcId

My question is, can we visualize such data in Kibana, if yes how?  
Or  
Do i need to work on logstash to organized it more? --- i was assuming logstash-cloudtrail plugin would do this job.

e.g.: "eventName": "DescribeLoadBalancers"

## below is not a complete response:

## `"responseElements": "{\"loadBalancers\": [{\"ipAddressType\": \"ipv4\", \"vpcId\": \"vpc-1111111\", \"loadBalancerArn\": \"arn:aws:elasticloadbalancing:us-west-2:1111:loadbalancer/net/quuuuuuuulb/4ckkkkkkkk\", \"type\": \"network\", \"state\": {\"code\": \"active\"}, \"dNSName\":`

Thanks & Regards...

---

<div class="post-metadata">

**Author:** ![Bargs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bargs/32/5429_2.png) [@Bargs](https://discuss.elastic.co/u/Bargs)\
**Post date:** [December 14, 2017, 10:09pm UTC](https://discuss.elastic.co/t/kibana-cloudtrail-visualization/111864/2 "2017-12-14T22:09:59Z")

</div>

I'm not familiar with the cloud trail plugin, but by default elasticsearch will flatten documents before indexing. So when you go to build a visualization you should see fields like `responseElements.loadBalancers.ipAddressType` in the field list. The one caveat is that Kibana [does not support nested aggregations](https://github.com/elastic/kibana/issues/1084), so if the cloud trail plugin maps those fields as [nested](https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html) it won't work.

---

<div class="post-metadata">

**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\
**Post date:** [December 14, 2017, 10:33pm UTC](https://discuss.elastic.co/t/kibana-cloudtrail-visualization/111864/3 "2017-12-14T22:33:44Z")

</div>

> [@piyush](#):
>
> Hi Team,
> 
> I am forwarding aws cloudtrail logs to ES-5.5 and it is forwarded by logstash-5 (using cloudtrail plugin). While preparing visualizations, i realized that for some events "responseElements" field is not organized, it's in array format.
> 
> e.g.: i am expecting fields:
> 
> responseElements.ipAddressType
> 
> responseElements.vpcId
> 
> My question is, can we visualize such data in Kibana, if yes how?
> 
> Or
> 
> Do i need to work on logstash to organized it more? --- i was assuming logstash-cloudtrail plugin would do this job.
> 
> e.g.: "eventName": "DescribeLoadBalancers"
> 
> below is not a complete response:  
> "responseElements": "{"loadBalancers": [{"ipAddressType": "ipv4", "vpcId": "vpc-1111111", "loadBalancerArn": "arn:aws:elasticloadbalancing:us-west-2:1111:loadbalancer/net/quuuuuuuulb/4ckkkkkkkk", "type": "network", "state": {"code": "active"}, "dNSName":  
> Thanks & Regards...

Thanks....

---

<div class="post-metadata">

**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\
**Post date:** [December 14, 2017, 11:06pm UTC](https://discuss.elastic.co/t/kibana-cloudtrail-visualization/111864/4 "2017-12-14T23:06:34Z")

</div>

Thanks Bargs...But i can't see the field "responseElements.loadBalancers.ipAddressType"

You are right, Ideally logstash cloud trail plugin should do this job and this value should be available as a field. But it's not, so my primary question was why logstash cloud trail plugin is not working as expected? or is it expected and i need to put more efforts to format responseElements.

---

<div class="post-metadata">

**Author:** ![Bargs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bargs/32/5429_2.png) [@Bargs](https://discuss.elastic.co/u/Bargs)\
**Post date:** [December 15, 2017, 5:55pm UTC](https://discuss.elastic.co/t/kibana-cloudtrail-visualization/111864/5 "2017-12-15T17:55:40Z")

</div>

I would need to know what your mappings look like to see why that field isn't showing up. Could you post the response of the [get mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html) API for your index?

---

<div class="post-metadata">

**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\
**Post date:** [December 15, 2017, 7:09pm UTC](https://discuss.elastic.co/t/kibana-cloudtrail-visualization/111864/6 "2017-12-15T19:09:01Z")

</div>

Thanks, can't put all mappings due to character limitation, PFB relevant ones:

Please let me know if you need more details.

* * *

```
  "responseElements": {
    "properties": {
      "x-amz-server-side-encryption": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  },

,
      "x-amz-server-side-encryption": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  },

```

* * *

---

<div class="post-metadata">

**Author:** ![Bargs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bargs/32/5429_2.png) [@Bargs](https://discuss.elastic.co/u/Bargs)\
**Post date:** [December 15, 2017, 7:49pm UTC](https://discuss.elastic.co/t/kibana-cloudtrail-visualization/111864/7 "2017-12-15T19:49:15Z")

</div>

> [@piyush](#):
>
> can't put all mappings due to character limitation

Github gist?

Based on that small snippet, you should see a `responseElements.x-amz-server-side-encryption.keyword` field in the visualization editor's field drop down.

---

<div class="post-metadata">

**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\
**Post date:** [December 15, 2017, 11:08pm UTC](https://discuss.elastic.co/t/kibana-cloudtrail-visualization/111864/8 "2017-12-15T23:08:09Z")

</div>

Thanks, i found this field . Also got understanding for mappings.

Thanks Matt...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 12, 2018, 11:08pm UTC](https://discuss.elastic.co/t/kibana-cloudtrail-visualization/111864/9 "2018-01-12T23:08:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.