Kibana cloudtrail visualization

Elastic Stack Kibana
1

Hi Team,
I am forwarding aws cloudtrail logs to ES-5.5 and it is forwarded by logstash-5 (using cloudtrail plugin). While preparing visualizations, i realized that for some events "responseElements" field is not organized, it's in array format.

e.g.: i am expecting fields:
responseElements.ipAddressType
responseElements.vpcId

My question is, can we visualize such data in Kibana, if yes how?
Or
Do i need to work on logstash to organized it more? --- i was assuming logstash-cloudtrail plugin would do this job.

e.g.: "eventName": "DescribeLoadBalancers"

below is not a complete response:

"responseElements": "{\"loadBalancers\": [{\"ipAddressType\": \"ipv4\", \"vpcId\": \"vpc-1111111\", \"loadBalancerArn\": \"arn:aws:elasticloadbalancing:us-west-2:1111:loadbalancer/net/quuuuuuuulb/4ckkkkkkkk\", \"type\": \"network\", \"state\": {\"code\": \"active\"}, \"dNSName\":

Thanks & Regards...

2

I'm not familiar with the cloud trail plugin, but by default elasticsearch will flatten documents before indexing. So when you go to build a visualization you should see fields like responseElements.loadBalancers.ipAddressType in the field list. The one caveat is that Kibana does not support nested aggregations, so if the cloud trail plugin maps those fields as nested it won't work.

3

Thanks....

4

Thanks Bargs...But i can't see the field "responseElements.loadBalancers.ipAddressType"

You are right, Ideally logstash cloud trail plugin should do this job and this value should be available as a field. But it's not, so my primary question was why logstash cloud trail plugin is not working as expected? or is it expected and i need to put more efforts to format responseElements.

5

I would need to know what your mappings look like to see why that field isn't showing up. Could you post the response of the get mapping API for your index?

6

Thanks, can't put all mappings due to character limitation, PFB relevant ones:

Please let me know if you need more details.

  "responseElements": {
    "properties": {
      "x-amz-server-side-encryption": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  },



,
      "x-amz-server-side-encryption": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  },
7

Github gist?

Based on that small snippet, you should see a responseElements.x-amz-server-side-encryption.keyword field in the visualization editor's field drop down.

8

Thanks, i found this field . Also got understanding for mappings.

Thanks Matt...

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.