Hi ,I'm trying to read json from s3 and put in Elasticsearch .I put filter in logstash config file.In input I have json, nested json ,list etc. I have to fetch only six fields out of 156 fields how can I put filter?
Hi! The prune filter allows you to create whitelists based on name or regex, but sadly it doesn't work on nested fields. Maybe you can still use it for most of your fields?
I have this problem too, but all my fields are nested since they come from a split filter, so I'm looking for a better solution.
Thank you for your reply.I'm using mutute plugin, but not able to remove fields.
my conf file:-
input {
s3 {
bucket => "logstash-backup111"
access_key_id => "A*********A"
secret_access_key => "0***************mqouRA"
}
}
filter {
json {
source => "message"
target => "doc"
}
mutate {
remove_field => [ "[extendedOrder][minimumBasketCharge]"]
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
stdout {
codec => rubydebug
}
}Your field [extendedOrder][minimumBasketCharge] is one of the fields created with the JSON filter, right? The whole field name might be [doc][extendedOrder][minimumBasketCharge] then. I think if you want to use [extendedOrder][minimumBasketCharge] you must not set the target in the JSON filter so the created fields stay at the root of the document.
https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html#plugins-filters-json-target
This is json that I'm reading from s3 :
{
"orderDetails": {
"orderNumber": "367388519",
"shortOrderNumber": "8647",
"customerNumber": "983e8d27-398f2bcf",
"deliveryDate": "01/11/2017",
"vanNumber": "2L",
"customerSupportNumber": "88888 4040",
"storeId": "9165",
"voucherPromotionName": "d clkujkubs"
},
"orderLines": [
{
"productId": "uid:1c1f0b3c-239e-4218-8441-acdf3677c9fb",
"productType": "Chilled",
"productDescription": " 120g ",
"productQuantity": "2",
"productUnitPrice": "1.20",
"productTotalPrice": "2.40",
"productSubGroup": "F16AB",
"pickingZone": "CH",
"isApplicableForVat": "N",
"useBy": "01/13/2017",
"flagToCustomer":"Poor Quality",
"productAmountAfterPromotion":"1.300"
},
{
"productId": "b00b6c-acfc-439d-833f-2633c39a516a",
"productType": "Cupboard"
"productSubGroup": "N53DG",
"pickingZone": "AM",
"isApplicableForVat": "N",
"flagToCustomer":"Poor Quality",
"productAmountAfterPromotion":"1.300"
},
{
"productId": "c78a9-8175-4976-df94e545a4a0",
"productType": "Frozen",
"productDescription": "Terate Orange 3 X 90ml *",
"productQuantity": "1",
"productUnitPrice": "2.99",
"productTotalPrice": "2.9900",
"productSubGroup": "F76AJ",
"pickingZone": "FZ",
"isApplicableForVat": "Y",
"flagToCustomer":"Poor Quality",
"productAmountAfterPromotion":"1.300"
},
{
"productId": "6ba31-e023-4dba-b4dd-d26acff38c72",
"productType": "Frozen",
"productDescription": " Chew Ice Cream 480ml *",
"productQuantity": "1",
"productUnitPrice": "2.00",
"productTotalPrice": "2.0000",
"productSubGroup": "F76DA",
"pickingZone": "FZ",
"isApplicableForVat": "Y",
"flagToCustomer":"Poor Quality",
"productAmountAfterPromotion":"1.300"
},
],
"substitutions": [
{
"orderLineId": "45878501",
"promotionName": "",
"orderedProductActualUnitPrice": "3.50",
"orderedProductPromoTotalPrice": "3.50",
"substitutedProducts": [
{
"productProvidedQuantity": "1",
"productProvidedActualUnitPrice": "3.50",
"productProvidedActualTotalPrice": "3.5000",
"productProvidedPromoUnitPrice": "3.5000",
"priceMatchRefunds": "0.00",
"isEssentialSubstitution": "N"
}
]
},
{
"orderLineId": "45878786",
"promotionName": "",
"orderedProductId": "18-8441-acdf3677c9fb",
"orderedProductType": "Chilled",
"orderedProductSubgroup": "F16AB",
"orderedProductPromoTotalPrice": "5.25",
"substitutedProducts": [
{
"productProvidedDescription": "Mixed Leaf 150g ",
"productProvidedQuantity": "1",
"productProvidedActualUnitPrice": "1.50",
"productProvidedActualTotalPrice": "3.00",
"productProvidedPromoUnitPrice": "2.45",
"priceMatchRefunds": "0.00",
"isEssentialSubstitution": "N"
}
]
},
{
"orderLineId": "45887889",
"promotionName": "",
"orderedProductId": "3c5-f418-4001-b0g",
"orderedProductQuantity": "1",
"orderedProductActualUnitPrice": "1.44",
"orderedProductPromoTotalPrice": "1.44",
"substitutedProducts": [
{
"productIdProvided": "d9b9309-cf5f-4a4d-8ab7-6b463a1e3b2c",
"productProvidedType": "Cupboard",
"productProvidedSubgroup": "G51AR",
"productProvidedPickingZone": "AM",
"productProvidedActualTotalPrice": "2.3900",
"productProvidedPromoUnitPrice": "1.4400",
"priceMatchRefunds": "0.95",
"isEssentialSubstitution": "N"
}
]
},
{
"orderLineId": "45887890",
"promotionName": "",
"orderedProductId": "4d11a-7b00-4695-9a21-8d1907510908",
"orderedProductType": "Cupboard",
"orderedProductActualUnitPrice": "2.88",
"orderedProductPromoTotalPrice": "2.88",
"substitutedProducts": [
{
"productIdProvided": "9b9309-cf5f-4a4d-8ab7-6b463a1e3b2c",
"productProvidedType": "Cupboard",
"productProvidedSubgroup": "G51AR",
"productProvidedPickingZone": "AM",
"productProvidedIsApplicableForVat": "Y",
"productProvidedDescription": "Nature Valley Crunchy Bars Oat & Hon 5pack 210g",
}
]
},
{
"orderLineId": "27295703",
"orderedProductPromoTotalPrice": "0.77",
"substitutedProducts": [
{
"productProvidedSubgroup": "B33ED",
"productProvidedDescription": " 6 Whlemeal Pitta Bread ",
"productProvidedActualTotalPrice": "0.5000
"isEssentialSubstitution": "N",
"productProvidedType": "Cupboard"
}
],
"orderedProductId": "a975d9-ff4b-4474-94bb-fa6baeb03034",
"orderedProductSubgroup": "B33ED",
"promotionName": "",
"orderedProductQuantity": "1",
"orderedProductActualUnitPrice": "0.38",
"orderedProductDescription": " White Pitta Bread "
}
],
"customerDetails": {
"title": "Mrs",
"firstName": "vivek",
"middleName": "",
"lastName": "Freeman-Bown",
"mobileNumber": "",
"emailId": ""
},
"outOfStockProducts": [
{
"productId": "81655524",
"productDescription": "Tv Times ",
"quantity": "1",
"isCustomerRequestedSubstitution": "TS"
},
{
"productDescription": " White Pitta Bread ",
"productId": "d:cba975d9-ff4b-4474-94bb-fa6baeb03034",
"quantity": "1",
"isCustomerRequestedSubstitution": "TS"
}
],
"eCoupons": {
"eCoupon": "0.0000"
},
"extendedOrder": {
"currency": {
"currencyCode": "GBP",
"currencySymbol": "£"
}
}
Now I try to fetch orderNumber,StoreId,emailId,firstname+lastname.
Ok that's quite like my problem in the end. You only want some nested fields but prune filter doesn't work on those. Maybe you could remove some not nested fields though, such as orderLines, substitutions, outOfStockProducts, eCoupons, extendedOrder, that would still help if it removes all the fields below those root fields.
But if you want exactly those 5 fields, you'll have to make a mutate filter with remove_field and a list of all the fields you do not want. This is supposed to work even for nested fields, so that's the best solution for now even if it's not nice to write.
I'm not able to remove any fields or tags.I can add new fields but can't remove or extract.May be issue with nested json.Can you please share with me logstash.conf file so I can see how to put mutate inside filter .
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.