Hello, I'm trying to index custom fields that are parsed from s3 json log, it's not seeming to work, I have these processors:
filebeat.inputs:
- type: aws-s3
queue_url: https://sqs.XXX>XXXX.com/XXXX/XXXXX
access_key_id: 'XXXX-XXX'
secret_access_key: 'XXXXXXX-XXXXXXX'
file_selectors:
- regex: '^AWSLogs/\d+/CloudTrail/'
expand_event_list_from_field: 'Records'
processors:
- decode_json_fields:
fields: ["message"]
process_array: true
max_depth: 1
target: "s3.access"
overwrite_keys: false
And so end up with ~15 fields all prepended with "s3.access.*" but when I check in Kibana discover they're not index and I can't search based on them, how do I get this working? I already tried a dynamic template:
{
"s3.access.*": {
"path_match": "s3.access.*",
"mapping": {
"type": "text"
},
"match_mapping_type": "*"
}
}
but that returned the error - "Can't get text on a START_OBJECT at 1:995"
I think I'm close but any help on this would be great, thank you.