Xml PARSING IN LOGSTASH

Elastic Stack Logstash
1

Hi I am new to ELK

Looking for extracting column from XML i need bedtype as a new column value as D1K from below xml D1K

My logstashsetting

Sample Logstash configuration for creating a simple

Beats -> Logstash -> Elasticsearch pipeline.

input {
beats {
port => 5044
}
}

filter {
csv {
separator => "@$@"
columns => ["date","est_cd","hotelhubcode","sessionid","searchid","requestid","request","response","parentmethodname","starttime","endtime","duration","exception","notes","channelpropertyid"]
}
mutate {convert => ["duration", "integer"]}

xml{
store_xml => false
source => "message"
xpath => ["/con/text()", "BedType"]
}
}

output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "cwtamadeuspropavaillog"
#user => "elastic"
#password => "changeme"
}
}

Below is my file beat setting

filebeat.inputs:

Each - is an input. Most options can be set at the input level, so

you can use different inputs for various configurations.

Below are the input specific configurations.

  • type: log

    Change to true to enable this input configuration.

    enabled: true

    Paths that should be crawled and fetched. Glob based paths.

    paths:

    • D:\Elk\Mysqllogs\GDSlogs\Amadeus*.csv

Below is My XML

"message": "2020-03-11 11:25:24,230@@HH154772@@HCO54742@@cf770bc3-b9f1-4e65-a75a-d0923ebda65d@@df77a585-8573-4a21-8e22-45e6345347ce@@c3becc14-c7a6-4591-9418-d2f16e607600@@http://localhost:6005/api/ratedetails/?echoToken=7d030973-6adb-44bd-8d30-f463edaae72e&type=json&version=1.0&language=EN&arrivalDate=2020-09-16&departureDate=2020-09-17&roomsCount=1&guestCount=1&room_adultCount=1&room_childCount=0&requesterId=DISCUKSTAFFOWN&hotelCode=HP051652&roomCode=&rateCode=&roomRateCode=&ratePlanId=1644373&searchId=f0f7a1f4-33fe-66ad-8a30-b9bb74474df1@$@<HotelCompleteAvailability_14>ILBA202009162020091711DV89337VILLAGE LEEDS SOUTH0E0GBP2NNNDBL9810DBLDTI0DIRECTTRAVEL@DTI00PN981001ST9810AT98100DIRECT TRAVEL INC ROOM ONLYDOUBLE ROOM SEALY KING BED FREE WIFITEA AND COFFEE MAKING FACILITIES SAT TV00000000000YYY11D1K</HotelMatch

2

You need to fix the formatting in your post so that we can correctly understand what your presenting. Using the backtick character to enclose single lines of code, like this. For blocks of code, encase the block in three backticks, this will preserve the indentation and prevent markdown from styling the text, like below

input {
  file {
  #Makin comments, bra!
  }
}
3

I understand i need to add xpath to parsh the new column from xml my new configuration

Sample Logstash configuration for creating a simple

Beats -> Logstash -> Elasticsearch pipeline.

input {
beats {
port => 5044
}
}

filter {
csv {
separator => "@$@"
columns => ["date","est_cd","hotelhubcode","sessionid","searchid","requestid","request","response","parentmethodname","starttime","endtime","duration","exception","notes","channelpropertyid"]
}
mutate {convert => ["duration", "integer"]}

xml{
store_xml => false
source => "message"
xpath => ["/con/text(/HotelCompleteAvailability_14/HotelCompleteAvailability/HotelMatch[1]/BedType)", "BedType"]
}
}

output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "cwtamadeuspropavaillog4"
#user => "elastic"
#password => "changeme"
}
}

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.