# How to remove some text in json file using logstash filter plugin? Which plugin to use?
**URL:** https://discuss.elastic.co/t/how-to-remove-some-text-in-json-file-using-logstash-filter-plugin-which-plugin-to-use/130437
**Category:** Logstash
**Created:** [May 3, 2018, 11:08am UTC](https://discuss.elastic.co/t/how-to-remove-some-text-in-json-file-using-logstash-filter-plugin-which-plugin-to-use/130437 "2018-05-03T11:08:14Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@omkarr](https://discuss.elastic.co/u/omkarr)
#### Post date: [May 3, 2018, 11:08am UTC](https://discuss.elastic.co/t/how-to-remove-some-text-in-json-file-using-logstash-filter-plugin-which-plugin-to-use/130437/1 "2018-05-03T11:08:15Z")
We are trying to load cloudtrail logs into elasticsearch using logstash. But the json is in following format.
{"Records":[{"eventVersion":"1.05","userIdentity":{"type":"AWSService","invokedBy":"[config.amazonaws.com](http://config.amazonaws.com)"}........... (json file continuation). But if we load the log without any filter plugin , all the fields(awsRegion,type,invokedBy,etc) are coming into message field only,which doesn't serve us.
Tried using below filter "mutate" plugin different configs, but it didn't work.
filter{
mutate{
remove\_field =\> ["Records"] ----tried this
replace =\> { "message" =\> "Records: }----tried this
update =\> { "Records" =\> ""}-----tried this
}
}
Any help will be appreciated.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 3, 2018, 11:18am UTC](https://discuss.elastic.co/t/how-to-remove-some-text-in-json-file-using-logstash-filter-plugin-which-plugin-to-use/130437/2 "2018-05-03T11:18:55Z")
Use a json\_lines codec in your input plugin to deserialize the JSON strings in each input event.
---
### Author:  [@omkarr](https://discuss.elastic.co/u/omkarr)
#### Post date: [May 3, 2018, 11:52am UTC](https://discuss.elastic.co/t/how-to-remove-some-text-in-json-file-using-logstash-filter-plugin-which-plugin-to-use/130437/3 "2018-05-03T11:52:23Z")
Tried using json\_lines codec, but didn't work.
input {
s3 {
"access\_key\_id" =\> "AKIAIELI3NYPJTWBGYKA"
"secret\_access\_key" =\> "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"
"bucket" =\> "cloudtraillogsvirtusacloud"
"region" =\> "sa-east-1"
"prefix" =\> "AWSLogs/912607726479/CloudTrail/sa-east-1/2018/05/03"
"codec" =\> "json\_lines"
}
}
any suggestions?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 3, 2018, 12:35pm UTC](https://discuss.elastic.co/t/how-to-remove-some-text-in-json-file-using-logstash-filter-plugin-which-plugin-to-use/130437/4 "2018-05-03T12:35:11Z")
Exactly what do the events processed by Logstash look like? Use a `stdout { codec => rubydebug }` output.
---
### Author:  [@omkarr](https://discuss.elastic.co/u/omkarr)
#### Post date: [May 4, 2018, 10:47am UTC](https://discuss.elastic.co/t/how-to-remove-some-text-in-json-file-using-logstash-filter-plugin-which-plugin-to-use/130437/5 "2018-05-04T10:47:21Z")
We are not getting any output even if we give stdout{codec =\> rubydebug}.
If we manually download and update the json file by removing "Records" field that is in the starting of the file ,then using following config we are able to process the json file with all the fields available.Attached the json file .
 
input{
exec{
command =\> "cat 912607726479\_CloudTrail\_sa-east-1\_20180504T0000Z\_yK8mHG7E05CkUtrF.json"
codec =\> json\_lines
interval =\> 60
}
}
output{
stdout{codec =\> rubydebug}
}
**Output:**
All the fields are getting separately (awsRegion,eventtype.eventsource,etc).
But our requirement is to pull hundreds of logs generated daily from S3 bucket. But without removing "Records" field from logs it is not processing. So kindly help to achieve this one.
Following is the config file we used:
input {
s3 {
"access\_key\_id" =\> "AKIAIELI3NYPJTWBGYKA"
"secret\_access\_key" =\> "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"
"bucket" =\> "cloudtraillogsvirtusacloud"
"region" =\> "sa-east-1"
"prefix" =\> "AWSLogs/912607726479/CloudTrail/sa-east-1/2018/05/03"
"codec" =\> "json\_lines"
}
}
filter{
mutate{
remove\_field =\> ["Records"]
}}
output{
stdout{codec =\> rubydebug}
}
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [May 4, 2018, 5:17pm UTC](https://discuss.elastic.co/t/how-to-remove-some-text-in-json-file-using-logstash-filter-plugin-which-plugin-to-use/130437/6 "2018-05-04T17:17:21Z")
> But our requirement is to pull hundreds of logs generated daily from S3 bucket. But without removing "Records" field from logs it is not processing.
What do you mean by that? There should be errors or warnings in the log file.
---
### Author:  [@omkarr](https://discuss.elastic.co/u/omkarr)
#### Post date: [May 5, 2018, 6:13pm UTC](https://discuss.elastic.co/t/how-to-remove-some-text-in-json-file-using-logstash-filter-plugin-which-plugin-to-use/130437/7 "2018-05-05T18:13:02Z")
Same thing we have achieved using [Logz.io](http://Logz.io) setup where they have a interface with different log shippers and using AWS cloud trail log shipper,configured our S3 bucket and saved it. After that in Kibana viewed our different fields generated from our cloudtrail logs.
Similarly we are trying to achieve using ELK Stack setup with logstash S3 plugin as shown in above conversation. The main thing is to remove the text "Records" from logs to get all the fields and view it in kibana?
Any idea on this will be helpful.
---
### Author:  [@omkarr](https://discuss.elastic.co/u/omkarr)
#### Post date: [May 7, 2018, 10:41am UTC](https://discuss.elastic.co/t/how-to-remove-some-text-in-json-file-using-logstash-filter-plugin-which-plugin-to-use/130437/8 "2018-05-07T10:41:42Z")
Every 5 min a log file is generated in s3 bucket for every region. We need to analyze these logs to pull some important information out of these logs.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 4, 2018, 10:41am UTC](https://discuss.elastic.co/t/how-to-remove-some-text-in-json-file-using-logstash-filter-plugin-which-plugin-to-use/130437/9 "2018-06-04T10:41:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.