Loop through json arrays

Elastic Stack Logstash
1

Hi,
My JSON file looks like this :
{
"bookings": [
{
"bookingId": 553267,
"bookingRef": "VO20200108460",
"statusId": 1,
"status": "Confirmed",
"rooms": [
{
"board": "Demi pension",
"paxes": {
"adults": 2,
"infant": 0,
"children": 0
},
"quantity": 1,
"room": "DOUBLE"
},
{
"board": "Demi pension",
"paxes": {
"adults": 1,
"infant": 0,
"children": 0
},
"quantity": 1,
"room": "SINGLE"
}
],
"options": } }

How do I loop through all rooms.
my config file in logstash :

input {
http_poller {
urls => {
url => "XXX"
}
request_timeout => 60
schedule => { every => "60s" }
codec => "json"
}
}
filter {
split { field => "[bookings]" }
split { field => "[bookings][rooms]" }
mutate {
rename => {
"[bookings][bookingId]" => "bookingId"
"[bookings][status]" => "status"
"[bookings][hotelId]" => "hotelId"
"[bookings][hotelName]" => "hotelName"
"[bookings][hotelCity]" => "hotelCity"
"[bookings][hotelCountry]" => "hotelCountry"
"[bookings][arrDate]" => "arrDate"
"[bookings][depDate]" => "depDate"
"[bookings][price]" => "price"
"[bookings][currency]" => "currency"
"[bookings][purchasePrice]" => "purchasePrice"
"[bookings][partnerName]" => "partnerName"
"[bookings][partnerId]" => "partnerId"
"[bookings][firstName]" => "firstName"
"[bookings][lastName]" => "lastName"
"[bookings][channel]" => "channel"
"[bookings][supplierName]" => "supplierName"
"[bookings][rooms][board]" => "board"
"[bookings][rooms][paxes][adults]" => "adults"
"[bookings][rooms][paxes][infant]" => "infant"
"[bookings][rooms][paxes][children]" => "children"
"[bookings][rooms][quantity]" => "quantity"
"[bookings][rooms][room]" => "room"
}
remove_field => ["confirmedDate", "bookingRef", "bookings", "createdDate", "hotelAddress", "hotelPhonearrDate", "customerId", "title", "email", "city", "mobile", "supplierId","paxe", "payments", "options", "isXML"]
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "bookings"
action => "update"
doc_as_upsert => "true"
document_id => "%{bookingId}"
}
stdout { codec => rubydebug }
}

Any help please !

2

What do you not like about the resulting output?

3

i can't have all the data :

loop
loop1085×142 24.8 KB

in the document above there is no double room

{
"board": "Demi pension",
"paxes": {
"adults": 2,
"infant": 0,
"children": 0
},
"quantity": 1,
"room": "DOUBLE"
},

4

You have a split filter that is splitting "[bookings][rooms]", so you will only have one room per document.

5

what can i do to have all the rooms per document !?

6

Remove the second split filter:

# split { field => "[bookings][rooms]" }
7

lo
lo1080×282 40.8 KB

it displays no rooms :frowning:

8

That is because you remove the bookings field. Change your mutate+rename to include

"[bookings][rooms]" => "rooms"
9

it's work..
but I can't do aggregation on the field rooms .....
i need this field rooms to build my dashboards.....

lop
lop1288×139 26.2 KB

10

OK, let's backtrack. If you go back to your original configuration, if a booking has multiple rooms, then when you do the split on [bookings][rooms] you will get multiple events with the same [booking_id]. In your elasticsearch filter you are using

document_id => "%{bookingId}"

so it will update the document by overwriting the first room with the second room. If you want multiple documents for the same booking id then remove that.