How Can I extract some fields from existence source in logstash

Elastic Stack Logstash
1

I'm trying to extract some fields from existence json source.I have s3 bucket as a source and elasticsearch as a destination.
logstash.config:-

input {
s3 {
bucket => "logstash-backup111"
access_key_id => "**************"
secret_access_key => "0l****************J9mqouRA"
}
}

filter {
json {
source => "message"
target => "json"

		}
	mutate {
                remove_field => [ "[substitution][substitutedProduct]"]
	        }
 }

output {

	elasticsearch {
  				hosts => [ "localhost:9200" ]
		}
	stdout  {
		 codec => rubydebug
                   }
   }

Here I'm trying to remove substitutedProduct list ,but in Elasticsearch I'm getting unchanged json.
please someone help how I extract particular fields.

2

I got answer. Removed all fields and add new fields:-

filter {
json {
source => "message"
target => "doc"
}
mutate {
add_field => [ "[doc][deliveryDate]", "%{[doc][orderDetails][storeId]}" ]
add_field => [ "[doc][orderStatus]", "%{[doc][orderDetails][orderStatus]}"]
add_field => [ "[doc][channel]" ,"ukghs:online"]
add_field =>["[doc][orderId]" ,"ordernumber%{[doc][orderDetails][customerNumber]}_%{[doc][orderDetails][storeId]} "]

      remove_field => "[doc][orderDetails]"
      remove_field => "[doc][orderLines]"
      remove_field => "[doc][clubCardStatements]"
      remove_field => "[doc][extendedOrder]"
      remove_field => "[doc][clubCardStatements]"
      remove_field => "[doc][paymentAndSavings]"
      remove_field => "[doc][deliveryDetails]"
      remove_field => "[doc][outOfStockProducts]"
      remove_field => "[doc][promotionSavings]"
      remove_field => "[doc][eCoupons]"
      remove_field =>"[doc][substitutions]"
      remove_field =>"[doc][customerDetails]"
}

}

if anyone has better solution please share

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.