Could not index event to Elasticsearch

Elastic Stack Logstash
1

Hi I am getting this error in Elasticsearch (Please look at the bolded statements), any sort of help will be welcome. Thank you so much in advance:

[2022-05-03T12:41:26,871][WARN ][logstash.outputs.elasticsearch][rtt_kfk2es-metaswitch_vsbc_cdr][c354ec4cd692a6bf0e84ebd56a859c44b54bfe1e7981ee2a666788addd1f31b7] Could not index event to Elasticsearch.

{:status=>400, :action=>["index", {:_id=>nil, :_index=>"uc-metaswitch-vsbc-cdr"

and

{"index"=>{"_index"=>"uc-metaswitch-vsbc-cdr-2022.04.30-000039", "_type"=>"_doc", "_id"=>"AMbMioABqml3fPOHTpMD", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [cdr.call.QoS.gate.flowinfo.1.sd] of type [keyword] in document with id 'AMbMioABqml3fPOHTpMD'. Preview of field's value: '

{content=m=audio 0 RTP/SAVP 0 18 101 13\na=rtpmap:0 PCMU/8000\na=rtpmap:18 G729/8000\na=rtpmap:101 telephone-event/8000\na=rtpmap:13 CN/8000\na=ice-ufrag:sqijQaznPOo4C43c/9\na=ice-pwd:6WNYkcd1DZMUezTjz2HT9atd\na=rtcp-mux\na=fmtp:18 annexb=no\na=fmtp:101 0-16\na=crypto:2 AES_CM_128_HMAC_SHA1_80 inline:****************************************|2^31\na=ptime:10\n, direction=inbound}

'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:5352"}}}}}

[2022-05-03T12:41:26,871][WARN ][logstash.outputs.elasticsearch][rtt_kfk2es-metaswitch_vsbc_cdr][c354ec4cd692a6bf0e84ebd56a859c44b54bfe1e7981ee2a666788addd1f31b7] **Could not index event to Elasticsearch.**

**{:status=>400**, :action=>["index", {:_id=>nil, :_index=>"uc-metaswitch-vsbc-cdr", :routing=>nil}, {"message"=>"<e><st>Perimeta XML CDR</st><h>toro63-mdr-ssc01-v</h><t>1651596075822</t><tid>2814749767122557</tid><sid>43</sid><eid>CDR</eid><call starttime=\"1651595475182\" starttime_local=\"2022-05-03T16:31:15+0000\" record_type=\"full\" endtime=\"1651596075784\" endtime_local=\"2022-05-03T16:41:15+0000\" duration=\"600602\" release_side=\"term\" bcid=\"627158D32020202020202030302B3030303030300000005A\"><party type=\"orig\" phone=\"+15066943835\" domain=\"siptrunking.bell.ca\" sig_address=\"10.92.24.43\" sig_port=\"5060\" sig_transport=\"udp\" sip_call_id=\"BW123125799030522-990668301@10.92.24.43\"/><party type=\"term\" phone=\"5142750173\" domain=\"12345-12345.directrouting.bell.ca\" sig_address=\"52.114.132.46\" sig_port=\"5061\" sig_transport=\"tls\" sip_call_id=\"0gQAAC8WAAACBAAALxYAAD06v4MpWPDN4OYkUCG/fdm4ElO1W6f+u94dT+bQPO2wT6ywPUputxLpXlyxR5Rpyw--@toronto.directrouting.bell.ca\"/><adjacency type=\"orig\" name=\"MDR-CORE-ADJ\" account=\"BW-ACCT\" vpnid=\"0X00000001\" mediarealm=\"CoreMedia\"/><adjacency type=\"term\" name=\"MDR-PEER-TEAMS1-ADJ\" account=\"MDR-PEER-ACCT\" vpnid=\"0X00000002\" mediarealm=\"AccessMediaInternet\"/><connect time=\"1651595481809\" time_local=\"2022-05-03T16:31:21+0000\" ack=\"true\"/><disconnect time=\"1651596075784\" time_local=\"2022-05-03T16:41:15+0000\" reason=\"0\"/><post_dial_delay duration=\"1268\"/><QoS stream_id=\"1\" instance=\"0\" reservetime=\"1651595475182\" reservetime_local=\"2022-05-03T16:31:15+0000\" committime=\"1651595481809\" committime_local=\"2022-05-03T16:31:21+0000\" releasetime=\"1651596075821\" releasetime_local=\"2022-05-03T16:41:15+0000\"><gate><flowinfo><local address=\"10.92.161.51\" port=\"16462\"/><remote address=\"10.92.71.11\" port=\"43830\"/><sd direction=\"inbound\">m=audio 0 RTP/AVP 0 18 101\ra=rtpmap:0 PCMU/8000\ra=rtpmap:18 G729/8000\ra=rtpmap:101 telephone-event/8000\ra=fmtp:18 annexb=no\ra=fmtp:101 0-16\ra=ptime:10\r</sd><sd direction=\"outbound\">m=audio 0 RTP/AVP 0 18 101\ra=rtpmap:0 PCMU/8000\ra=rtpmap:18 G729/8000\ra=rtpmap:101 telephone-event/8000\ra=fmtp:18 annexb=no\ra=fmtp:101 0-16\ra=ptime:10\r</sd><RTCPstats>PS=58729, OS=7099400, PR=29676, OR=5935200, PD=0, OD=0, PL=0, JI=0, TOS=0, TOR=0, LA=196, PC/RPS=29622, PC/ROS=4739520, PC/RPR=58729, PC/RPL=0, PC/RJI=4, PC/RLA=196, RF=60, MOS=31, PC/RAJ=0, PC/RML=366</RTCPstats></flowinfo><flowinfo transport_type=\"srtp\"><local address=\"209.71.229.167\" port=\"16462\"/><remote address=\"52.112.77.235\" port=\"49319\"/><sd direction=\"inbound\">m=audio 0 RTP/SAVP 0 18 101 13\ra=rtpmap:0 PCMU/8000\ra=rtpmap:18 G729/8000\ra=rtpmap:101 telephone-event/8000\ra=rtpmap:13 CN/8000\ra=ice-ufrag:sqijQaznPOo4C43c/9\ra=ice-pwd:6WNYkcd1DZMUezTjz2HT9atd\ra=rtcp-mux\ra=fmtp:18 annexb=no\ra=fmtp:101 0-16\ra=crypto:2 AES_CM_128_HMAC_SHA1_80 inline:*****************************************|2^31\ra=ptime:10\r</sd><sd direction=\"outbound\">m=audio 0 RTP/SAVP 0 18 101 13\ra=rtpmap:0 PCMU/8000\ra=rtpmap:18 G729/8000\ra=rtpmap:101 telephone-event/8000\ra=rtpmap:13 CN/8000\ra=ice-ufrag:sqijQaznPOo4C43c/9\ra=ice-pwd:6WNYkcd1DZMUezTjz2HT9atd\ra=rtcp-mux\ra=fmtp:18 annexb=no\ra=fmtp:101 0-16\ra=crypto:2 AES_CM_128_HMAC_SHA1_80 inline:*****************************************|2^20\ra=ptime:10\r</sd><RTCPstats>PS=3414, OS=602112, PR=669, OR=139986, PD=30, OD=4052, PL=0, JI=0, TOS=0, TOR=0, LA=0, PC/RPS=669, PC/ROS=118578, PC/RPR=0, PC/RPL=0, PC/RJI=0, PC/RLA=0, RF=92, MOS=44, PC/RAJ=0, PC/RML=0</RTCPstats></flowinfo></gate></QoS></call></e>", "@version"=>"1", "@timestamp"=>2022-05-03T16:41:26.652Z, "cdr"=>{"h"=>["toro63-mdr-ssc01-v"], "tid"=>["2814749767122557"], "call"=>[{"bcid"=>"627158D32020202020202030302B3030303030300000005A", "disconnect"=>[

{"time"=>"1651596075784", "reason"=>"0", "time_local"=>"2022-05-03T16:41:15+0000"}

], "starttime_local"=>"2022-05-03T16:31:15+0000", "starttime"=>"1651595475182", "duration"=>"600602", "connect"=>[

{"time"=>"1651595481809", "ack"=>"true", "time_local"=>"2022-05-03T16:31:21+0000"}

], "adjacency"=>{"term"=>

{"mediarealm"=>"AccessMediaInternet", "account"=>"MDR-PEER-ACCT", "name"=>"MDR-PEER-TEAMS1-ADJ", "type"=>"term", "vpnid"=>"0X00000002"}, "orig"=>{"mediarealm"=>"CoreMedia", "account"=>"BW-ACCT", "name"=>"MDR-CORE-ADJ", "type"=>"orig", "vpnid"=>"0X00000001"}}, "record_type"=>"full", "endtime"=>"1651596075784", "e/RAJ=0, PC/RML=0</RTCPstats></flowinfo></gate></QoS></call></e>", "@version"=>"1", "@timestamp"=>2022-05-03T16:41:26.652Z, "cdr"=>{"h"=>["toro63-mdr-ssc01-v"], "tid"=>["2814749767122557"], "call"=>[\{"bcid"=>"627158D32020202020202030302B3030303030300000005A", "disconnect"=>[\{"time"=>"1651596075784", "reason"=>"0", "time_local"=>"2022-05-03T16:41:15+0000"}], "starttime_local"=>"2022-05-03T16:31:15+0000", "starttime"=>"1651595475182", "duration"=>"600602", "connect"=>["1651595481809"," "ack"=">"true"," "time_local"=">"2022-05-03T16:31:21+0000"}"" class="external-link" target="_blank" rel="nofollow noopener">\\\{"time"=>"1651595481809", "ack"=>"true", "time_local"=>"2022-05-03T16:31:21+0000"}](file://%7B/), "adjacency"=>{"term"=>{"mediarealm"=>"AccessMediaInternet", "account"=>"MDR-PEER-ACCT", "name"=>"MDR-PEER-TEAMS1-ADJ", "type"=>"term", "vpnid"=>"0X00000002"}

, "orig"=>{"mediarealm"=>"CoreMedia", "account"=>"BW-ACCT", "name"=>"MDR-CORE-ADJ", "type"=>"orig", "vpnid"=>"0X00000001"}}, "record_type"=>"full", "endtime"=>"1651596075784", "endtime_local"=>"2022-05-03T16:41:15+0000", "party"=>{"term"=>

{"domain"=>"12345-12345.directrouting.bell.ca", "sig_port"=>"5061", "sig_transport"=>"tls", "type"=>"term", "sip_call_id"=>"0gQAAC8WAAACBAAALxYAAD06v4MpWPDN4OYkUCG/fdm4ElO1W6f+u94dT+bQPO2wT6ywPUputxLpXlyxR5Rpyw--@toronto.directrouting.bell.ca", "sig_address"=>"52.114.132.46", "phone"=>"5142750173"}

, "orig"=>{"domain"=>"siptrunking.bell.ca", "sig_port"=>"5060", "sig_transport"=>"udp", "type"=>"orig", "sip_call_id"=>"BW123125799030522-990668301@10.92.24.43", "sig_address"=>"10.92.24.43", "phone"=>"+15066943835"}}, "release_side"=>"term", "QoS"=>[{"releasetime_local"=>"2022-05-03T16:41:15+0000", "committime"=>"1651595481809", "gate"=>[{"flowinfo"=>{"1"=>{"RTCPstats"=>

{"RF"=>"92", "TOS"=>"0", "PC/RJI"=>"0", "MOS"=>"44", "PC/RPR"=>"0", "JI"=>"0", "OD"=>"4052", "TOR"=>"0", "PC/RLA"=>"0", "PC/RPS"=>"669", "PR"=>"669", "OS"=>"602112", "PL"=>"0", "PC/ROS"=>"118578", "PS"=>"3414", "PD"=>"30", "LA"=>"0", "PC/RPL"=>"0", "PC/RML"=>"0", "OR"=>"139986", "PC/RAJ"=>"0"}

, "transport_type"=>"srtp", "remote"=>[

{"address"=>"52.112.77.235", "port"=>"49319"}

], "local"=>[

{"address"=>"209.71.229.167", "port"=>"16462"}

], "sd"=>[

{"content"=>"m=audio 0 RTP/SAVP 0 18 101 13\na=rtpmap:0 PCMU/8000\na=rtpmap:18 G729/8000\na=rtpmap:101 telephone-event/8000\na=rtpmap:13 CN/8000\na=ice-ufrag:sqijQaznPOo4C43c/9\na=ice-pwd:6WNYkcd1DZMUezTjz2HT9atd\na=rtcp-mux\na=fmtp:18 annexb=no\na=fmtp:101 0-16\na=crypto:2 AES_CM_128_HMAC_SHA1_80 inline:****************************************|2^31\na=ptime:10\n", "direction"=>"inbound"}

,

{"content"=>"m=audio 0 RTP/SAVP 0 18 101 13\na=rtpmap:0 PCMU/8000\na=rtpmap:18 G729/8000\na=rtpmap:101 telephone-event/8000\na=rtpmap:13 CN/8000\na=ice-ufrag:sqijQaznPOo4C43c/9\na=ice-pwd:6WNYkcd1DZMUezTjz2HT9atd\na=rtcp-mux\na=fmtp:18 annexb=no\na=fmtp:101 0-16\na=crypto:2 AES_CM_128_HMAC_SHA1_80 inline:****************************************|2^20\na=ptime:10\n", "direction"=>"outbound"}

]}, "0"=>{"RTCPstats"=>

{"RF"=>"60", "TOS"=>"0", "PC/RJI"=>"4", "MOS"=>"31", "PC/RPR"=>"58729", "JI"=>"0", "OD"=>"0", "TOR"=>"0", "PC/RLA"=>"196", "PC/RPS"=>"29622", "PR"=>"29676", "OS"=>"7099400", "PL"=>"0", "PC/ROS"=>"4739520", "PS"=>"58729", "PD"=>"0", "LA"=>"196", "PC/RPL"=>"0", "PC/RML"=>"366", "OR"=>"5935200", "PC/RAJ"=>"0"}

, "remote"=>[

{"address"=>"10.92.71.11", "port"=>"43830"}

], "local"=>[

{"address"=>"10.92.161.51", "port"=>"16462"}

], "sd"=>[

{"content"=>"m=audio 0 RTP/AVP 0 18 101\na=rtpmap:0 PCMU/8000\na=rtpmap:18 G729/8000\na=rtpmap:101 telephone-event/8000\na=fmtp:18 annexb=no\na=fmtp:101 0-16\na=ptime:10\n", "direction"=>"inbound"}

,

{"content"=>"m=audio 0 RTP/AVP 0 18 101\na=rtpmap:0 PCMU/8000\na=rtpmap:18 G729/8000\na=rtpmap:101 telephone-event/8000\na=fmtp:18 annexb=no\na=fmtp:101 0-16\na=ptime:10\n", "direction"=>"outbound"}

]}}}], "releasetime"=>"1651596075821", "stream_id"=>"1", "reservetime"=>"1651595475182", "instance"=>"0", "reservetime_local"=>"2022-05-03T16:31:15+0000", "committime_local"=>"2022-05-03T16:31:21+0000"}], "post_dial_delay"=>[

{"duration"=>"1268"}

]}], "t"=>["1651596075822"], "eid"=>["CDR"], "st"=>["Perimeta XML CDR"], "sid"=>["43"]}}], :**response=>{"index"=>{"_index"=>"uc-metaswitch-vsbc-cdr-2022.04.30-000039", "_type"=>"_doc", "_id"=>"AMbMioABqml3fPOHTpMD", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [cdr.call.QoS.gate.flowinfo.1.sd] of type [keyword] in document with id 'AMbMioABqml3fPOHTpMD'. Preview of field's value: '**

{content=m=audio 0 RTP/SAVP 0 18 101 13\na=rtpmap:0 PCMU/8000\na=rtpmap:18 G729/8000\na=rtpmap:101 telephone-event/8000\na=rtpmap:13 CN/8000\na=ice-ufrag:sqijQaznPOo4C43c/9\na=ice-pwd:6WNYkcd1DZMUezTjz2HT9atd\na=rtcp-mux\na=fmtp:18 annexb=no\na=fmtp:101 0-16\na=crypto:2 AES_CM_128_HMAC_SHA1_80 inline:****************************************|2^31\na=ptime:10\n, direction=inbound}

'", "caused_by"=>**{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:5352"**}}}}}
2

See this thread.

3

Check what Badger said and also check Elasticsearch log, the most likely it's something related to the field mapping-data types.

4

is this supposed to be done in the logstash conf. file? Im not using filebeat and am not familiar with it (Im a beginner to ELK), If you could elaborate abit, I would really appreciate it

5

It is related to that, and from what Ive gathered from what badger said "In Elasticsearch a field can be either an object or a value, it cannot be an object in some documents and a value in others" and this is the issue in my situation, however I dont understand where to apply as Im not using filebeats.. do I make these changes in the conf. file?:

Screenshot 2022-08-15 105304
Screenshot 2022-08-15 105304826×850 42.4 KB

6

This has nothing to do with filebeat. You need to make changes in the logstash.conf file.

7

So for the: {"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [cdr.call.QoS.gate.flowinfo.1.sd] of type [keyword] in document with id 'AMbMioABqml3fPOHTpMD '

We would need to convert the cdr.call.qos.gate.flowinfo.(0/1).sd into a string? like:

'event.set("[cdr][call][qos][gate][flowinfo][1][sd]", event.get("[cdr][call][qos][gate][flowinfo][1][sd]").to_s)'

and place it in between this in my conf. file? Thank you for your help

1111
11111030×624 51.2 KB

2222
22221072×597 44 KB

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.