Can't delete empty field

[hiba (chaieb)]

Hi,
i want to parse a json response and load data into elasticsearch via logstash ,
i try to delete arrDate and depDate fields if it's empty ,,,but i still got this error,

[2020-10-01T13:24:13,646][WARN ][logstash.outputs.elasticsearch][main][8dafa128c9ab419583f2bd9f8b39512559e9b500bbfe515cd29596f761e77443] Could not index event to Elasticsearch. {:status=>400, :action=>["update", {:_id=>"63198", :_index=>"invoices", :routing=>nil, :_type=>"_doc", :retry_on_conflict=>1}, #LogStash::Event:0x3a09cff0], :response=>{"update"=>{"_index"=>"invoices", "_type"=>"_doc", "_id"=>"63198", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [bookings.arrDate] of type [date] in document with id '63198'. Preview of field's value: ''", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"cannot parse empty date"}}}}}

my filter :
filter {
split { field => "[invoices]" }
mutate {
rename => {
"[invoices][invoiceRef]" => "invoiceRef"
"[invoices][invoiceId]" => "invoiceId"
"[invoices][invoiceType]" => "invoiceType"
"[invoices][invoiceStatus]" => "invoiceStatus"
"[invoices][invoiceDate]" => "invoiceDate"
"[invoices][amount]" => "amount"
"[invoices][currency]" => "currency"
"[invoices][billingTo]" => "billingTo"
"[invoices][customerId]" => "customerId"
"[invoices][customer]" => "customer"
"[invoices][distributorId]" => "distributorId"
"[invoices][distributor]" => "distributor"
"[invoices][bookings]" => "bookings"
}

}

if [bookings.arrDate] == "" {
mutate {
remove_field => ["bookings.arrDate"]
}
}

}

my data:
{
"invoices": [
{
"invoiceRef": "2020/B/000186",
"invoiceId": 63625,
"invoiceType": "Facture",
"invoiceStatus": "validée",
"invoiceDate": "2020-06-30",
"amount": 143.960,
"currency": "TND",
"billingTo": "Didi Travel",
"customerId": "",
"customer": "",
"distributorId": 13666,
"distributor": "DIDI TRAVEL",
"bookings": [
{
"bookingRef": 559986,
"bookingId": 559986,
"arrDate": "2020-06-27",
"depDate": "2020-06-28",
"detail": "Club Marmara Palm Beach Hammamet 4*
Tunisie,Hammamet
chambre double VP Standard Vue piscine - Soft All Inclusive
2 pax : 2 adultes
27 juin 2020 - 28 juin 2020 (1 nuit)
Amine Ben Ouali"
}
]
}]}

any hel please !

[Badger]

@hiba

[bookings.arrDate]

Both references should be [bookings][0][arrDate] since it is an array

[hiba (chaieb)]

Hi @Badger,

i change the filter to :

if [bookings][0][arrDate] == "" {
mutate {
remove_field => ["[bookings][0][arrDate]"]
}
}
if [bookings][0][depDate] == "" {
mutate {
remove_field => ["[bookings][0][depDate]"]
}
}

But the type of the fields depDate and arrDate change to "text".
So, i create an index with mapping where i put these two fields as a "date" type, but i got the same error of parse

[Badger]

Maybe you have events where the array has multiple entries.

[hiba (chaieb)]

yes i have

[Badger]

OK, so you have to remove all of the empty dates. You could try something like

ruby {
    code => '
        bookings = event.get("bookings")
        if bookings
            bookings.each_index { |x|
                if [bookings][x]["arrDate"] == ""
                    event.remove("[bookings][#{x}][arrDate]")
                end
                if [bookings][x]["depDate"] == ""
                    event.remove("[bookings][#{x}][depDate]")
                end
            }
        end
    '
}

[hiba (chaieb)]

Blockquote[2020-10-02T15:48:15,508][WARN ][logstash.outputs.elasticsearch][main][c063fff6f011cbeb262319fc0b3497018846d1d2ecb9baf70b73cdd0c43665ec] Could not index event to Elasticsearch. {:status=>400, :action=>["update", {:_id=>"63368", :_index=>"invoices", :routing=>nil, :_type=>"_doc", :retry_on_conflict=>1}, #LogStash::Event:0x4e3700c6], :response=>{"update"=>{"_index"=>"invoices", "_type"=>"_doc", "_id"=>"63368", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [bookings.arrDate] of type [date] in document with id '63368'. Preview of field's value: ''", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"cannot parse empty date"}}}}}

filter {
split { field => "[invoices]" }
mutate {
rename => {
"[invoices][invoiceRef]" => "invoiceRef"
"[invoices][invoiceId]" => "invoiceId"
"[invoices][invoiceType]" => "invoiceType"
"[invoices][invoiceStatus]" => "invoiceStatus"
"[invoices][invoiceDate]" => "invoiceDate"
"[invoices][amount]" => "amount"
"[invoices][currency]" => "currency"
"[invoices][billingTo]" => "billingTo"
"[invoices][customerId]" => "customerId"
"[invoices][customer]" => "customer"
"[invoices][distributorId]" => "distributorId"
"[invoices][distributor]" => "distributor"
"[invoices][bookings]" => "bookings"
}
remove_field => ["billingTo"]

}

ruby {
code => '
bookings = event.get("bookings")
if bookings
bookings.each_index { |x|
if [bookings][x ]["arrDate"] == ""
event.remove("[bookings][#{x}][arrDate]")
end
if [bookings][x ]["depDate"] == ""
event.remove("[bookings][#{x}][depDate]")
end
}
end
'
}

}