Hi team,
My log file is "|" separated but two fields contains XML data. please let me know how can i read this.
Please show an example.
My Xml file looks like:
1000|121|2017-07-18 15:14:25.893024|0|xxx.xxx.net|read|'<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"> <soapenv:Header/> <soapenv:Body> <web:getLookUpServiceDetails> <getLookUpService> <serviceRequester>iOBD</serviceRequester> <lineOfBusiness>mobility</lineOfBusiness> <lookupAttribute> <searchAttrValue>911425152231426</searchAttrValue> </lookupAttribute> </getLookUpService> </web:getLookUpServiceDetails> </soapenv:Body> </soapenv:Envelope>'|'<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"> <soapenv:Body> <ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com"> <getLookUpServiceReturn> <errorInfo> <ErrorCode>M136</ErrorCode> <ErrorMessage>While executing transform reformat for port out0: Index out of bounds in assignment for mapEntry.attributeValue. Index is 0 but vector has no elements. Source is the value "attribute_value_vec". </ErrorMessage> </errorInfo> <lookupResponseList/> </getLookUpServiceReturn> </ns:getLookUpServiceDetailsResponse> </soapenv:Body> </soapenv:Envelope> '|1500390865
As you can see my log is "|" separated and it has two xml data below are field in which my log need to be mapped.
txn_cd - 1000
user_id - 121
start_time - 2017-07-18 15:14:25.893024
status - 0
host -xxx.xxx.net
ser_name - read
request - '<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"> <soapenv:Header/> <soapenv:Body> <web:getLookUpServiceDetails> <getLookUpService> <serviceRequester>iOBD</serviceRequester> <lineOfBusiness>mobility</lineOfBusiness> <lookupAttribute> <searchAttrValue>911425152231426</searchAttrValue> </lookupAttribute> </getLookUpService> </web:getLookUpServiceDetails> </soapenv:Body> </soapenv:Envelope>'
response - '<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"> <soapenv:Body> <ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com"> <getLookUpServiceReturn> <errorInfo> <ErrorCode>M136</ErrorCode> <ErrorMessage>While executing transform reformat for port out0: Index out of bounds in assignment for mapEntry.attributeValue. Index is 0 but vector has no elements. Source is the value "attribute_value_vec". </ErrorMessage> </errorInfo> <lookupResponseList/> </getLookUpServiceReturn> </ns:getLookUpServiceDetailsResponse> </soapenv:Body> </soapenv:Envelope> '
txn_id -1500390865
please help me with the configuration file.
You should be able to use a csv filter to split up the line into separate fields.
Used the below config file to split the line
input {
file {
path => "abc/abc.dat"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => "|"
columns => ["txn_cd","usr_id","event_ts","status","host","service_name","request","response","txn_id"]
}
date{
match => ["event_ts", "yyyy-MM-dd HH:mm:ss.SSSSSS"]
target => "event_ts"
}
mutate {convert => ["status", "integer"]}
}
output {
elasticsearch {
hosts => "http://abc:9200"
index => "xml"
}
stdout {}
}
but the fields are not properly matched
but the fields are not properly matched
Please elaborate. What do you get?
My output looks like this in kibana after loading
@timestamp October 25th 2017, 16:02:31.500
t @version 1
t _id AV9Rk1UH8A4YAHFNXmUV
t _index xml
# _score -
t _type logs
t host abc.abc.net
t message ||2017-07-18 15:40:16.836493|1|abc,abc.net|read|'<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"> <soapenv:Header/> <soapenv:Body> <web:getLookUpServiceDetails> <getLookUpService> <serviceRequester>iOBD</serviceRequester> <lineOfBusiness>mobility</lineOfBusiness> <lookupAttribute> <searchAttrValue>911425152231426</searchAttrValue> </lookupAttribute> </getLookUpService> </web:getLookUpServiceDetails> </soapenv:Body> </soapenv:Envelope>'|'<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"> <soapenv:Body> <ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com"> <getLookUpServiceReturn> <errorInfo> <ErrorCode/> <ErrorMessage/> </errorInfo> <lookupResponseList> <mapEntry> <attributeName>region</attributeName> <attributeValue>["VIVO", "VIVO", "OPPO", "OPPO", "OPPO", "OPPO"]</attributeValue> </mapEntry> <mapEntry> <attributeName>msisdn</attributeName> <attributeValue>"Maharashtra"</attributeValue> </mapEntry> <mapEntry> <attributeName>imei</attributeName> <attributeValue>"917756870222"</attributeValue> </mapEntry> <mapEntry> <attributeName>imsi</attributeName> <attributeValue>"911425152231426"</attributeValue> </mapEntry> <mapEntry> <attributeName>device_vendor</attributeName> <attributeValue>"404909092353805"</attributeValue> </mapEntry> <mapEntry> <attributeName>device_type</attributeName> <attributeValue>"E311"</attributeValue> </mapEntry> <mapEntry> <attributeName>device_version</attributeName> <attributeValue>"T2222"</attributeValue> </mapEntry> <mapEntry> <attributeName>g3</attributeName> <attributeValue>"Yes"</attributeValue> </mapEntry> <mapEntry> <attributeName>mms</attributeName> <attributeValue>"Yes"</attributeValue> </mapEntry> <mapEntry> <attributeName>gprs</attributeName> <attributeValue>"Yes"</attributeValue> </mapEntry> <mapEntry> <attributeName>streaming</attributeName> <attributeValue>"Yes"</attributeValue> </mapEntry> <mapEntry> <attributeName>ota</attributeName> <attributeValue>"Yes"</attributeValue> </mapEntry> <mapEntry> <attributeName>wap</attributeName> <attributeValue>"Yes"</attributeValue> </mapEntry> <mapEntry> <attributeName>os</attributeName> <attributeValue>"Google"</attributeValue> </mapEntry> <mapEntry> <attributeName>version</attributeName> <attributeValue>"4.4.2"</attributeValue> </mapEntry> <mapEntry> <attributeName>camera</attributeName> <attributeValue>"Yes"</attributeValue> </mapEntry> <mapEntry> <attributeName>apn</attributeName> <attributeValue>""AIRTELGPRS.COM,AIRTELMMS.COM""</attributeValue> </mapEntry> </lookupResponseList> </getLookUpServiceReturn> </ns:getLookUpServiceDetailsResponse> </soapenv:Body> </soapenv:Envelope> '|1500392416
t path /abc/abc
t tags _csvparsefailure
Fields are not mapping according to the configuration file
As the _csvparsefailure tag indicates something goes wrong with the CSV parsing. The Logstash log should contain more information.
Could you please help me on what extra information that need to be put in ?
I meant that you should read the Logstash log file and look for clues about the CSV parse failure.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.