Cloudtrail logs using logstash

Elastic Stack Logstash
1

I have a mostly-working config for ingesting my cloudtrail logs using logstash. ElasticSearch doesn't seem to be parsing the 'ResponseElements' field, though, as well as a few other fields that can vary from being empty to being a hash of potentially many values.

Here is my logstash config;

input {
s3 {
bucket => "cloudtrail"
prefix => "cloudtrail"
region => "us-somewhere"
type => "cloudtrail"
codec => "cloudtrail"
access_key_id => "asdf"
secret_access_key => "asdfasdf"
}
}

filter {
mutate {
gsub => [
"apiVersion", "_", "-"
]
}

	if [eventSource] == "s3.amazonaws.com" and [eventName] == "ListObjects" and [userIdentity.userName] == "cloudtrail-s3" {
		drop {}
	}

}

output {
elasticsearch {
hosts => ["localhost:9200"]
index => "cloudtrail-%{+YYYY.MM.dd}"
}

}

2

It's not clear what the problem is. What do the documents currently stored in ES look like? What would you like for them to look like instead? Copy/paste from Kibana's JSON tab (in the Discover panel).

3

Thanks Magnus.

The issue I'm having is that structures like the userIdentity are getting broken down into separate entries (userIdentity.accessKeyId, userIdentity.sessionIssuer.userName, etc) while the responseElements structure is in elasticSearch as the full json.

I think that this may be because responseElements is sometimes null, but I'm not sure how to make the stack handle both null and json array conditions.

is there a more common way of handling cloudtrail logs with an on-prem elastic stack?

Here's the json:

  "_index": "cloudtrail-2018.08.12",
  "_type": "doc",
  "_id": "oT-bMwerttjvhbyK4iBns",
  "_score": 1,
  "_source": {
    "userIdentity": {
      "accessKeyId": "asdf",
      "accountId": "78234508643",
      "principalId": "Aasdfdf:i-01asdfasdf11",
      "sessionContext": {
        "attributes": {
          "mfaAuthenticated": "false",
          "creationDate": "2018-08-12T23:00:34Z"
        },
        "sessionIssuer": {
          "userName": "limited_ec2_role",
          "accountId": "78234508643",
          "principalId": "AAasdfdf",
          "type": "Role",
          "arn": "arn:aws:iam::78234508643:role/limited_ec2_role"
        }
      },
      "type": "AssumedRole",
      "arn": "arn:aws:sts::78234508643:assumed-role/limited_ec2_role/i-013d536c400769c11"
    },
    "awsRegion": "us-west-1",
    "@version": "1",
    "responseElements": {
      "requestId": "162b3d9f-263b-4aeb-8345-f0e4234542bd",
      "reservationId": "r-05aaabe2345sf710",
      "instancesSet": {
        "items": [
          {
            "ebsOptimized": false,
            "blockDeviceMapping": {},
            "sourceDestCheck": true,
            "amiLaunchIndex": 0,
            "hypervisor": "xen",
            "monitoring": {
              "state": "disabled"
            },
            "stateReason": {
              "message": "pending",
              "code": "pending"
            },
            "vpcId": "vpc-f82348c9a",
            "keyName": "demo",
            "rootDeviceName": "/dev/xvda",
            "imageId": "ami-43e1f346",
            "instanceId": "i-04343434b4e9033c843",
            "privateDnsName": "ip-172-31-12-18.us-west-1.compute.internal",
            "launchTime": 1534118103000,
            "subnetId": "subnet-bgf598fb",
            "placement": {
              "availabilityZone": "us-west-1c",
              "tenancy": "default"
            },
            "instanceState": {
              "name": "pending",
              "code": 0
            },
            "architecture": "x86_64",
            "virtualizationType": "hvm",
            "privateIpAddress": "172.31.12.18",
            "tagSet": {
              "items": [
                {
                  "value": "23f54gt6kjhg",
                  "key": "authcode"
                },
                {
                  "value": "pl2018devdb",
                  "key": "Name"
                }
              ]
            },
            "networkInterfaceSet": {
              "items": [
                {
                  "sourceDestCheck": true,
                  "macAddress": "06:54:3d:c9:a2:e8",
                  "ownerId": "78234508643",
                  "privateIpAddressesSet": {
                    "item": [
                      {
                        "primary": true,
                        "privateDnsName": "ip-172-31-12-18.us-west-1.compute.internal",
                        "privateIpAddress": "172.31.12.18"
                      }
                    ]
                  },
                  "attachment": {
                    "status": "attaching",
                    "deleteOnTermination": true,
                    "attachmentId": "eni-attach-08299234528f3bd3",
                    "attachTime": 1534118103000,
                    "deviceIndex": 0
                  },
                  "ipv6AddressesSet": {},
                  "privateIpAddress": "172.31.12.18",
                  "vpcId": "vpc-f8928c9a",
                  "tagSet": {},
                  "networkInterfaceId": "eni-0792345455eef0c4",
                  "status": "in-use",
                  "privateDnsName": "ip-172-31-12-18.us-west-1.compute.internal",
                  "subnetId": "subnet-bdb8654b",
                  "groupSet": {
                    "items": [
                      {
                        "groupName": "default",
                        "groupId": "sg-4secfde"
                      }
                    ]
                  }
                }
              ]
            },
            "cpuOptions": {
              "coreCount": 1,
              "threadsPerCore": 1
            },
            "instanceType": "t2.micro",
            "productCodes": {},
            "rootDeviceType": "ebs",
            "groupSet": {
              "items": [
                {
                  "groupName": "default",
                  "groupId": "sg-4s2ecfde"
                }
              ]
            }
          }
        ]
      },
      "ownerId": "78234508643",
      "groupSet": {}
    },
    "type": "cloudtrail",
    "@timestamp": "2018-08-12T23:55:03.000Z",
    "eventName": "RunInstances",
    "requestID": "1asdf34-263b-4aeb-3456-f0e41163d2bd",
    "eventSource": "ec2.amazonaws.com",
    "sourceIPAddress": "1.2.3.4",
    "eventType": "AwsApiCall",
    "requestParameters": {
      "blockDeviceMapping": {},
      "instanceType": "t2.micro",
      "tagSpecificationSet": {
        "items": [
          {
            "resourceType": "instance",
            "tags": [
              {
                "value": "pl2018devdb",
                "key": "Name"
              },
              {
                "value": "23f54gt6kjhg",
                "key": "authcode"
              }
            ]
          }
        ]
      },
      "disableApiTermination": false,
      "instancesSet": {
        "items": [
          {
            "keyName": "demo",
            "minCount": 1,
            "maxCount": 1,
            "imageId": "ami-46e1f123"
          }
        ]
      },
      "monitoring": {
        "enabled": false
      }
    },
    "userAgent": "aws-cli/1.15.19 Python/2.7.13 Linux/4.14.26-46.32.amzn1.x86_64 botocore/1.10.19",
    "eventID": "be389018-08ca-4ace-b6dc-83e6543d8bf",
    "recipientAccountId": "78234508643",
    "eventVersion": "1.05"
  },
  "fields": {
    "@timestamp": [
      "2018-08-12T23:55:03.000Z"
    ],
    "userIdentity.sessionContext.attributes.creationDate": [
      "2018-08-12T23:00:34.000Z"
    ]
  }
}```
4

The issue I'm having is that structures like the userIdentity are getting broken down into separate entries (userIdentity.accessKeyId, userIdentity.sessionIssuer.userName, etc) while the responseElements structure is in elasticSearch as the full json.

Yes, but in what way is that a problem? The JSON you posted looks fine.

is there a more common way of handling cloudtrail logs with an on-prem elastic stack?

I have no idea.

5

I want to be able to search by the sub-entries in the requestParameters json, for example requestParameters.instanceType.

I guess this may be an elasticsearch configuration issue, though.

I appreciate knowing that the json is right. I hadn't seen that tab, thank you.

6

I want to be able to search by the sub-entries in the requestParameters json, for example requestParameters.instanceType.

requestParameters.instanceType:t2.micro doesn't work? That field has probably been mapped as text field but you'll most likely want it to be a keyword field instead, although even with an incorrect mapping the search should work (but I'd expect it to return more documents than expected).

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.