# Cloudtrail logs using logstash

**URL:** <https://discuss.elastic.co/t/cloudtrail-logs-using-logstash/143955>\
**Category:** Logstash\
**Created:** [August 11, 2018, 3:05am UTC](https://discuss.elastic.co/t/cloudtrail-logs-using-logstash/143955 "2018-08-11T03:05:52Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![jellsworth](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@jellsworth](https://discuss.elastic.co/u/jellsworth)\
**Post date:** [August 11, 2018, 3:05am UTC](https://discuss.elastic.co/t/cloudtrail-logs-using-logstash/143955/1 "2018-08-11T03:05:52Z")

</div>

I have a mostly-working config for ingesting my cloudtrail logs using logstash. ElasticSearch doesn't seem to be parsing the 'ResponseElements' field, though, as well as a few other fields that can vary from being empty to being a hash of potentially many values.

Here is my logstash config;

input {  
s3 {  
bucket =\> "cloudtrail"  
prefix =\> "cloudtrail"  
region =\> "us-somewhere"  
type =\> "cloudtrail"  
codec =\> "cloudtrail"  
access\_key\_id =\> "asdf"  
secret\_access\_key =\> "asdfasdf"  
}  
}

filter {  
mutate {  
gsub =\> [  
"apiVersion", "\_", "-"  
]  
}

```
	if [eventSource] == "s3.amazonaws.com" and [eventName] == "ListObjects" and [userIdentity.userName] == "cloudtrail-s3" {
		drop {}
	}

```

}

output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "cloudtrail-%{+YYYY.MM.dd}"  
}

}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 11, 2018, 5:32pm UTC](https://discuss.elastic.co/t/cloudtrail-logs-using-logstash/143955/2 "2018-08-11T17:32:42Z")

</div>

It's not clear what the problem is. What do the documents currently stored in ES look like? What would you like for them to look like instead? Copy/paste from Kibana's JSON tab (in the Discover panel).

---

<div class="post-metadata">

**Author:** ![jellsworth](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@jellsworth](https://discuss.elastic.co/u/jellsworth)\
**Post date:** [August 13, 2018, 12:34am UTC](https://discuss.elastic.co/t/cloudtrail-logs-using-logstash/143955/3 "2018-08-13T00:34:34Z")

</div>

Thanks Magnus.

The issue I'm having is that structures like the userIdentity are getting broken down into separate entries (userIdentity.accessKeyId, userIdentity.sessionIssuer.userName, etc) while the responseElements structure is in elasticSearch as the full json.

I think that this may be because responseElements is sometimes null, but I'm not sure how to make the stack handle both null and json array conditions.

is there a more common way of handling cloudtrail logs with an on-prem elastic stack?

Here's the json:

````auto
  "_index": "cloudtrail-2018.08.12",
  "_type": "doc",
  "_id": "oT-bMwerttjvhbyK4iBns",
  "_score": 1,
  "_source": {
    "userIdentity": {
      "accessKeyId": "asdf",
      "accountId": "78234508643",
      "principalId": "Aasdfdf:i-01asdfasdf11",
      "sessionContext": {
        "attributes": {
          "mfaAuthenticated": "false",
          "creationDate": "2018-08-12T23:00:34Z"
        },
        "sessionIssuer": {
          "userName": "limited_ec2_role",
          "accountId": "78234508643",
          "principalId": "AAasdfdf",
          "type": "Role",
          "arn": "arn:aws:iam::78234508643:role/limited_ec2_role"
        }
      },
      "type": "AssumedRole",
      "arn": "arn:aws:sts::78234508643:assumed-role/limited_ec2_role/i-013d536c400769c11"
    },
    "awsRegion": "us-west-1",
    "@version": "1",
    "responseElements": {
      "requestId": "162b3d9f-263b-4aeb-8345-f0e4234542bd",
      "reservationId": "r-05aaabe2345sf710",
      "instancesSet": {
        "items": [
          {
            "ebsOptimized": false,
            "blockDeviceMapping": {},
            "sourceDestCheck": true,
            "amiLaunchIndex": 0,
            "hypervisor": "xen",
            "monitoring": {
              "state": "disabled"
            },
            "stateReason": {
              "message": "pending",
              "code": "pending"
            },
            "vpcId": "vpc-f82348c9a",
            "keyName": "demo",
            "rootDeviceName": "/dev/xvda",
            "imageId": "ami-43e1f346",
            "instanceId": "i-04343434b4e9033c843",
            "privateDnsName": "ip-172-31-12-18.us-west-1.compute.internal",
            "launchTime": 1534118103000,
            "subnetId": "subnet-bgf598fb",
            "placement": {
              "availabilityZone": "us-west-1c",
              "tenancy": "default"
            },
            "instanceState": {
              "name": "pending",
              "code": 0
            },
            "architecture": "x86_64",
            "virtualizationType": "hvm",
            "privateIpAddress": "172.31.12.18",
            "tagSet": {
              "items": [
                {
                  "value": "23f54gt6kjhg",
                  "key": "authcode"
                },
                {
                  "value": "pl2018devdb",
                  "key": "Name"
                }
              ]
            },
            "networkInterfaceSet": {
              "items": [
                {
                  "sourceDestCheck": true,
                  "macAddress": "06:54:3d:c9:a2:e8",
                  "ownerId": "78234508643",
                  "privateIpAddressesSet": {
                    "item": [
                      {
                        "primary": true,
                        "privateDnsName": "ip-172-31-12-18.us-west-1.compute.internal",
                        "privateIpAddress": "172.31.12.18"
                      }
                    ]
                  },
                  "attachment": {
                    "status": "attaching",
                    "deleteOnTermination": true,
                    "attachmentId": "eni-attach-08299234528f3bd3",
                    "attachTime": 1534118103000,
                    "deviceIndex": 0
                  },
                  "ipv6AddressesSet": {},
                  "privateIpAddress": "172.31.12.18",
                  "vpcId": "vpc-f8928c9a",
                  "tagSet": {},
                  "networkInterfaceId": "eni-0792345455eef0c4",
                  "status": "in-use",
                  "privateDnsName": "ip-172-31-12-18.us-west-1.compute.internal",
                  "subnetId": "subnet-bdb8654b",
                  "groupSet": {
                    "items": [
                      {
                        "groupName": "default",
                        "groupId": "sg-4secfde"
                      }
                    ]
                  }
                }
              ]
            },
            "cpuOptions": {
              "coreCount": 1,
              "threadsPerCore": 1
            },
            "instanceType": "t2.micro",
            "productCodes": {},
            "rootDeviceType": "ebs",
            "groupSet": {
              "items": [
                {
                  "groupName": "default",
                  "groupId": "sg-4s2ecfde"
                }
              ]
            }
          }
        ]
      },
      "ownerId": "78234508643",
      "groupSet": {}
    },
    "type": "cloudtrail",
    "@timestamp": "2018-08-12T23:55:03.000Z",
    "eventName": "RunInstances",
    "requestID": "1asdf34-263b-4aeb-3456-f0e41163d2bd",
    "eventSource": "ec2.amazonaws.com",
    "sourceIPAddress": "1.2.3.4",
    "eventType": "AwsApiCall",
    "requestParameters": {
      "blockDeviceMapping": {},
      "instanceType": "t2.micro",
      "tagSpecificationSet": {
        "items": [
          {
            "resourceType": "instance",
            "tags": [
              {
                "value": "pl2018devdb",
                "key": "Name"
              },
              {
                "value": "23f54gt6kjhg",
                "key": "authcode"
              }
            ]
          }
        ]
      },
      "disableApiTermination": false,
      "instancesSet": {
        "items": [
          {
            "keyName": "demo",
            "minCount": 1,
            "maxCount": 1,
            "imageId": "ami-46e1f123"
          }
        ]
      },
      "monitoring": {
        "enabled": false
      }
    },
    "userAgent": "aws-cli/1.15.19 Python/2.7.13 Linux/4.14.26-46.32.amzn1.x86_64 botocore/1.10.19",
    "eventID": "be389018-08ca-4ace-b6dc-83e6543d8bf",
    "recipientAccountId": "78234508643",
    "eventVersion": "1.05"
  },
  "fields": {
    "@timestamp": [
      "2018-08-12T23:55:03.000Z"
    ],
    "userIdentity.sessionContext.attributes.creationDate": [
      "2018-08-12T23:00:34.000Z"
    ]
  }
}```
````

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 13, 2018, 6:18am UTC](https://discuss.elastic.co/t/cloudtrail-logs-using-logstash/143955/4 "2018-08-13T06:18:55Z")

</div>

> The issue I'm having is that structures like the userIdentity are getting broken down into separate entries (userIdentity.accessKeyId, userIdentity.sessionIssuer.userName, etc) while the responseElements structure is in elasticSearch as the full json.

Yes, but in what way is that a problem? The JSON you posted looks fine.

> is there a more common way of handling cloudtrail logs with an on-prem elastic stack?

I have no idea.

---

<div class="post-metadata">

**Author:** ![jellsworth](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@jellsworth](https://discuss.elastic.co/u/jellsworth)\
**Post date:** [August 13, 2018, 1:37pm UTC](https://discuss.elastic.co/t/cloudtrail-logs-using-logstash/143955/5 "2018-08-13T13:37:26Z")

</div>

I want to be able to search by the sub-entries in the requestParameters json, for example requestParameters.instanceType.

I guess this may be an elasticsearch configuration issue, though.

I appreciate knowing that the json is right. I hadn't seen that tab, thank you.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 13, 2018, 2:04pm UTC](https://discuss.elastic.co/t/cloudtrail-logs-using-logstash/143955/6 "2018-08-13T14:04:34Z")

</div>

> I want to be able to search by the sub-entries in the requestParameters json, for example requestParameters.instanceType.

`requestParameters.instanceType:t2.micro` doesn't work? That field has probably been mapped as text field but you'll most likely want it to be a keyword field instead, although even with an incorrect mapping the search should work (but I'd expect it to return more documents than expected).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 10, 2018, 2:04pm UTC](https://discuss.elastic.co/t/cloudtrail-logs-using-logstash/143955/7 "2018-09-10T14:04:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.