# Add a field to json body when indexing
**URL:** https://discuss.elastic.co/t/add-a-field-to-json-body-when-indexing/180339
**Category:** Elasticsearch
**Created:** [May 9, 2019, 10:54am UTC](https://discuss.elastic.co/t/add-a-field-to-json-body-when-indexing/180339 "2019-05-09T10:54:46Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@davidbien](https://discuss.elastic.co/u/davidbien)
#### Post date: [May 9, 2019, 10:54am UTC](https://discuss.elastic.co/t/add-a-field-to-json-body-when-indexing/180339/1 "2019-05-09T10:54:46Z")
I have a lambda function which sends cloudtrail logs to elasticsearch whenever new data arrives in a bucket. This works fine, however I would like to add one extra field that is not found in standard cloudtrail logs. This is what I have atm:
```
def handler(event, context):
logger.info('Event: ' + json.dumps(event, indent=2))
s3Bucket = event['Records'][0]['s3']['bucket']['name']
key = event['Records'][0]['s3']['object']['key']
try:
response = s3.get_object(Bucket=s3Bucket, Key=key)
content = gzip.GzipFile(fileobj=BytesIO(response['Body'].read())).read()
for record in json.loads(content)['Records']:
recordJson = json.dumps(record)
indexName = 'cloudtrail-' + datetime.datetime.now().strftime("%Y.%m.%d")
res = es.index(index=indexName, doc_type='record', id=record['eventID'], body=recordJson)
logger.info(res)
return True
except Exception as e:
logger.error('Something went wrong: ' + str(e))
traceback.print_exc()
return False
```
What I want to do is to add a field
> type: cloudtrail
But I'm not sure where to add this? Do I use elasticsearch API call for this or do I simply insert the field in json?
---
### Author:  [@gabriel\_tessier](https://discuss.elastic.co/u/gabriel_tessier)
#### Post date: [May 10, 2019, 8:21am UTC](https://discuss.elastic.co/t/add-a-field-to-json-body-when-indexing/180339/2 "2019-05-10T08:21:46Z")
Hi,
I think the simplest way will be to add it in your lambda function or if you cannot check about ingest :
[https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html)
Also avoid call your field type depend on the language you use it can be reserved word and it may confuse with the \_type field.
hope it will help you.
---
### Author:  [@davidbien](https://discuss.elastic.co/u/davidbien)
#### Post date: [May 10, 2019, 8:28am UTC](https://discuss.elastic.co/t/add-a-field-to-json-body-when-indexing/180339/3 "2019-05-10T08:28:37Z")
I can add it to my lambda function, my question is how? Do I inject it to json or do I add it as I send the body for indexing?
---
### Author:  [@gabriel\_tessier](https://discuss.elastic.co/u/gabriel_tessier)
#### Post date: [May 10, 2019, 8:32am UTC](https://discuss.elastic.co/t/add-a-field-to-json-body-when-indexing/180339/4 "2019-05-10T08:32:53Z")
if record is a dict something like may work
```auto
for record in json.loads(content)['Records']:
record['my_type'] = 'cloudtrail'
recordJson = json.dumps(record)
```
---
### Author:  [@gabriel\_tessier](https://discuss.elastic.co/u/gabriel_tessier)
#### Post date: [May 10, 2019, 8:35am UTC](https://discuss.elastic.co/t/add-a-field-to-json-body-when-indexing/180339/5 "2019-05-10T08:35:52Z")
Sorry just notice one other thing:
You set doc\_type as record
```auto
res = es.index(index=indexName, doc_type='record',
```
it's better to set as "\_doc" it will help when you'll upgrade to 7 or later version as lot of people use this convention.
```auto
res = es.index(index=indexName, doc_type='_doc',
```
---
### Author:  [@davidbien](https://discuss.elastic.co/u/davidbien)
#### Post date: [May 10, 2019, 8:46am UTC](https://discuss.elastic.co/t/add-a-field-to-json-body-when-indexing/180339/6 "2019-05-10T08:46:02Z")
Hi Gabriel,
Thank you for your suggestion. I managed to insert the type field successfully.
Unfortunately when I changed doc\_type to \_doc I got an error and had to revert the change. We won't be able to upgrade to version 7 for a while since we're using aws's elasticsearch service which is at 6.5 atm.
Thank you again.
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [May 10, 2019, 8:57am UTC](https://discuss.elastic.co/t/add-a-field-to-json-body-when-indexing/180339/7 "2019-05-10T08:57:27Z")
Not an answer but did you look at [https://www.elastic.co/cloud](https://www.elastic.co/cloud) and [https://aws.amazon.com/marketplace/pp/B01N6YCISK](https://aws.amazon.com/marketplace/pp/B01N6YCISK) ?
Cloud by elastic is one way to have access to all features, all managed by us. Think about what is there yet like Security, Monitoring, Reporting, SQL, Canvas, APM, Logs UI, Infra UI and what is coming next 🙂 ...
It includes all recent releases anytime a new version is published.
---
### Author:  [@davidbien](https://discuss.elastic.co/u/davidbien)
#### Post date: [May 10, 2019, 9:10am UTC](https://discuss.elastic.co/t/add-a-field-to-json-body-when-indexing/180339/8 "2019-05-10T09:10:29Z")
Hi David,
Yes, we considered those options but finally decided to go for AWS managed service.
---
### Author:  [@martinr\_ubi](https://discuss.elastic.co/u/martinr_ubi)
#### Post date: [May 10, 2019, 10:10am UTC](https://discuss.elastic.co/t/add-a-field-to-json-body-when-indexing/180339/9 "2019-05-10T10:10:44Z")
Hi @davidbien,
I recommend looking at this for possible inspiration. I use that, heavily modified, to bulk index AWS "stuff" (e.g. cloudtrail) into my Elastic ES.
> **[DataDog/datadog-serverless-functions](https://github.com/DataDog/datadog-serverless-functions/tree/master/aws/logs_monitoring)**
>
> Repo of lambda functions that process streams and send data to datadog - DataDog/datadog-serverless-functions
You should also consider doing cloudtrail-\>S3-\>S3 Event-\>SQS-\>Lambda and researching and designing for all the possible failure modes. ES will sometimes give you errors during indexing and if you don't handle retries and make the whole design with "guarantee at least once delivery", you'll loose events because of back-pressure, downtime/maintenance, mapping errors, etc. Lambda will only retry 3 times if the execution throws an exception which easily lead to data loss.
If you have enough activity in the AWS account you will find discrepancies between the number of cloudtrail events and the number of events that made it into ES.
I have also used bulk requests in my lambda to ship many events per request toward ES, your current code does 1 event per request which is slower and more wasteful, just a thing to consider depending on your volume. Your lambda could process events at a rate lower than the incoming rate, which would lead to very high concurrency for your lambda or lost events if you limited its concurrency.
I can also tell you that cloudtrail events produces collisions in the index mapping, because their schema is not consistent enough. Some event have differing types for the same field name, such events can't be indexed by ES and are lost if you don't handle them.
---
### Author:  [@davidbien](https://discuss.elastic.co/u/davidbien)
#### Post date: [May 10, 2019, 12:38pm UTC](https://discuss.elastic.co/t/add-a-field-to-json-body-when-indexing/180339/10 "2019-05-10T12:38:33Z")
Hi Martin,
Thank you for the link. I will have a look at it now and see how can I improve my current code.
Currently I use this setting for sending cloudtrail logs to ES:
Cloudtrail -\> S3 -\> S3 trigger -\> Lambda -\> Redis -\> logstash -\> Elasticsearch.
The reason behind this is that we ahve two endpoints at the moment and we need to send them to both. With redis I am not afraid that logs will be lost.
I agree that I need to consider error handling. I occasionally get KeyError for ['Records'] but I am not sure how to handle this yet. I also noticed some events not appearing in my ES because of this.
---
### Author:  [@gabriel\_tessier](https://discuss.elastic.co/u/gabriel_tessier)
#### Post date: [May 12, 2019, 6:28am UTC](https://discuss.elastic.co/t/add-a-field-to-json-body-when-indexing/180339/11 "2019-05-12T06:28:36Z")
Maybe not useful for you as you use elastic on Amazon Service and you stuck with 6.5 version [functionbeat is on beta for this version, after I don't know how the Amazon version diverge from the official version].
But just in case if somebody else come to this message, there's a more easy way to go with functionbeat.
[https://www.elastic.co/guide/en/beats/functionbeat/current/functionbeat-overview.html](https://www.elastic.co/guide/en/beats/functionbeat/current/functionbeat-overview.html)
Or you can check about how they did and copy the functionbeat code 😅
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 9, 2019, 6:28am UTC](https://discuss.elastic.co/t/add-a-field-to-json-body-when-indexing/180339/12 "2019-06-09T06:28:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.