Scripted field: Read field data from a document using doc[‘some_field’].value return null in some cases

hi,
on my document I have a field always populated. It's a string field and it can contains data like json or XML.
I created this scripted field to find problem.

def myField = doc['myfield'].value;
if (payload != null){
return payload;
}
else{
return "payload is null";
}

Now, in some cases I can see the payload data, but when I have an XML inside "myField", the scripted fields return "payload is null".
I don't know the reason, it's unbelievable.
Maybe the length of the string in XML, could it be?

but the payload is not null, this is the field content

any ideas?

Hi, I'm not able to replicate the issue.

  • Which version of Elasticsearch and Kibana are you running?
  • What is the mapping of your field? i.e. text or keyword?

If the field is mapped as text you might need to use params['_source']['myfield'] to get the value of the field. See the last paragraph on this page.

Tnx nick,
the versions are 6.3 for both.
I tried your
doc['event.payload].value --> works only in some cases, as described...
params['_source']['event.payload'] --> it doesn't work (I haven t errors but my fields seems to be always empty).

ELK:

KIBANA

In this case, doc['event.payload].value works (with this content):

In this case, doc['event.payload] is null (but it's not null!)

According to me the problem is related to the size(length) of field (event.payload); in the first example with XML, there are a lot of characters, could it be?
Tnx

Can you share the entire XML string that is failing? This may be a bug in the 6.3.0 version and I'd like to see if it has been fixed in subsequent releases.

Thanks nick,
I'm sharing the XML string that is failing, but the problem is related to all XML explicit content... Could the problem be related to XML declaration "<?xml version="1.0"...."

<?xml version='1.0' encoding='UTF-8'?>
<ZINT_SFDC_COMM>
<import>
<XT_SFDC>
<row id="0">
<ZID_RECORD>a0u1X000000gQ1p</ZID_RECORD>
<ZTIPOOPER>SWIN</ZTIPOOPER>
<DATA_EVENTO>2019-06-01</DATA_EVENTO>
<BP_CRM>0000004</BP_CRM>
<CC_CRM>BAN-0000001</CC_CRM>
<IMPIANTO_CRM>ABC-000001</IMPIANTO_CRM>
<CONTR_CRM>ASN-00000001</CONTR_CRM>
<BP_SAP></BP_SAP>
<CC_SAP></CC_SAP>
<POD_SAP>IT001E0000001</POD_SAP>
<CONTR_SAP></CONTR_SAP>
<IMPIANTO_SAP></IMPIANTO_SAP>
<BP_SAP_OLD></BP_SAP_OLD>
<CC_SAP_OLD></CC_SAP_OLD>
<CONTR_SAP_OLD></CONTR_SAP_OLD>
<TYPE>2</TYPE>
<NAME_FIRST></NAME_FIRST>
<NAME_LAST></NAME_LAST>
<NAME_ORG1>ENERGIA MOCK S.R.L.</NAME_ORG1>
<LEGAL_ENTITY>02</LEGAL_ENTITY>
<AUGRP>IT01</AUGRP>
<BPKIND>Z007</BPKIND>
<CF_TAXNUM></CF_TAXNUM>
<PI_TAXNUM>02000000001</PI_TAXNUM>
<CITY1_ESAZ>Milan</CITY1_ESAZ>
<COUNTRY_ESAZ>IT</COUNTRY_ESAZ>
<HOUSE_NUM1_ESAZ>12</HOUSE_NUM1_ESAZ>
<NAME_CO_ESAZ></NAME_CO_ESAZ>
<FLOOR_ESAZ></FLOOR_ESAZ>
<POST_CODE1_ESAZ>21052</POST_CODE1_ESAZ>
<REGION_ESAZ>VA</REGION_ESAZ>
<STREET_ESAZ>12 VIA SILVIO</STREET_ESAZ>
<ZBP_ISTATLOC>012026</ZBP_ISTATLOC>
<ROOMNUMBER_ESAZ></ROOMNUMBER_ESAZ>
<CITY1_CLIENTE>Milano</CITY1_CLIENTE>
<COUNTRY_CLIENTE>IT</COUNTRY_CLIENTE>
<HOUSE_NUM1_CLIENTE>12</HOUSE_NUM1_CLIENTE>
<NAME_CO_CLIENTE></NAME_CO_CLIENTE>
<FLOOR_CLIENTE></FLOOR_CLIENTE>
<POST_CODE1_CLIENTE>21052</POST_CODE1_CLIENTE>
<REGION_CLIENTE>VA</REGION_CLIENTE>
<STREET_CLIENTE>VIA SILVIO</STREET_CLIENTE>
<ROOMNUMBER_CLIENTE></ROOMNUMBER_CLIENTE>
<ZBP_ISTATLOC_CLIENTE>012026</ZBP_ISTATLOC_CLIENTE>
<TEL_NUMBER1></TEL_NUMBER1>
<SMTP_ADDR>testmail@gmail.com</SMTP_ADDR>
<PEC_SMTP_ADDR>testmail@gmail.com</PEC_SMTP_ADDR>
<PEC_SOLLECITI></PEC_SOLLECITI>
<CCNUM></CCNUM>
<DATA_INIZIO_CCNUM>0000-00-00</DATA_INIZIO_CCNUM>
<DATA_FINE_CCNUM>0000-00-00</DATA_FINE_CCNUM>
<BANKS></BANKS>
<BANKL></BANKL>
<BANKN></BANKN>
<BKONT></BKONT>
<IBAN></IBAN>
<Z_MANDATO></Z_MANDATO>
<STAT_MAND></STAT_MAND>
<COD_CUCSIA></COD_CUCSIA>
<CREDITORID></CREDITORID>
<DATASOTT>0000-00-00</DATASOTT>
<CODFISCFIRMA></CODFISCFIRMA>
<NOMEFIRMA></NOMEFIRMA>
<COGNOMEFIRMA></COGNOMEFIRMA>
<AB_MANDATO>0000-00-00</AB_MANDATO>
<BIS_MANDATO>0000-00-00</BIS_MANDATO>
<ZPUBB>N</ZPUBB>
<LGZUSATZ_BP></LGZUSATZ_BP>
<VBUND></VBUND>
<Z_COBRAND>IT01</Z_COBRAND>
<IKEY>Z2</IKEY>
<ZAHLKOND>DM30</ZAHLKOND>
<Z_MODINV>ZD</Z_MODINV>
<Z_SINTDETT>S</Z_SINTDETT>
<Z_DUALCODE>N</Z_DUALCODE>
<IPA_CODE></IPA_CODE>
<IPA_BEGDA>0000-00-00</IPA_BEGDA>
<Z_OU></Z_OU>
<Z_IOU></Z_IOU>
<Z_CUP></Z_CUP>
<Z_CIG></Z_CIG>
<Z_CONSIP></Z_CONSIP>
<Z_RIFAMM></Z_RIFAMM>
<EZAWE>B</EZAWE>
<ZCOD_ASS></ZCOD_ASS>
<ZCA_RES>Y</ZCA_RES>
<ZCA_TOP></ZCA_TOP>
<ZCOD_RES>AGESP</ZCOD_RES>
<SPARTE>E</SPARTE>
<GEMFAKT>2</GEMFAKT>
<KOFIZ>01</KOFIZ>
<ZCANACQ>AG</ZCANACQ>
<Z_KAM>Default</Z_KAM>
<Z_MERCRIE>SL</Z_MERCRIE>
<Z_PRODOTTO>Axpo99912</Z_PRODOTTO>
<Z_PRODOTTO_DESC>Listino Fisso 3Fasce</Z_PRODOTTO_DESC>
<Z_INIZIO>2019-06-01</Z_INIZIO>
<Z_LISTINO>RESEL_FI01386_1907FIX</Z_LISTINO>
<Z_MERCATO>ML</Z_MERCATO>
<Z_DEPOSITO>N</Z_DEPOSITO>
<ZTIP_OFF>B2B</ZTIP_OFF>
<Z_OFF_VERDE>N</Z_OFF_VERDE>
<Z_IMPDEP></Z_IMPDEP>
<ZAGENZIE></ZAGENZIE>
<ZGRUPPO></ZGRUPPO>
<ZCAPOGR></ZCAPOGR>
<ZCOD_OFFERTA_TOP></ZCOD_OFFERTA_TOP>
<ZFREQ>MENS</ZFREQ>
<ZTENS>220</ZTENS>
<Z_DISVENDITORE>Y</Z_DISVENDITORE>
<SERVICE>ZDIS</SERVICE>
<BRANCHE>35.23.00</BRANCHE>
<ER_TIPOUT_TF>DO</ER_TIPOUT_TF>
<ED_POTDIS>3.0</ED_POTDIS>
<ED_POTCON>3.0</ED_POTCON>
<ER_RESI_TF>Y</ER_RESI_TF>
<COANC0>1800</COANC0>
<COAND0></COAND0>
<EQ_COAND1></EQ_COAND1>
<EQ_COAND2></EQ_COAND2>
<EQ_COAND3></EQ_COAND3>
<DISDIS></DISDIS>
<EG_ENERG>N</EG_ENERG>
<EG_AUSIL>N</EG_AUSIL>
<EG_DOMD1>N</EG_DOMD1>
<EG_MIS2G></EG_MIS2G>
<EU_CLESEN></EU_CLESEN>
<DIS_CODE>D001</DIS_CODE>
<GU_CLIMP></GU_CLIMP>
<ER_ACC>ACCRES</ER_ACC>
<ED_ACCAUES></ED_ACCAUES>
<EQ_ACCFXES></EQ_ACCFXES>
<EU_STATOF>Attiva</EU_STATOF>
<AB_2G>0000-00-00</AB_2G>
<GF_CG>0.0000000</GF_CG>
<CITY1_FORN>PINEROLO</CITY1_FORN>
<COUNTRY_FORN>IT</COUNTRY_FORN>
<HOUSE_NUM1_FORN>45</HOUSE_NUM1_FORN>
<POST_CODE1_FORN>10064</POST_CODE1_FORN>
<REGION_FORN>TO</REGION_FORN>
<STREET_FORN>VIGONE</STREET_FORN>
<ROOMNUMBER></ROOMNUMBER>
<FLOOR></FLOOR>
<ZOGAL_ISTATLOC>000001</ZOGAL_ISTATLOC>
<LGZUSATZ></LGZUSATZ>
<Z_DAT_REMI></Z_DAT_REMI>
<ZINT_OPERAND>
<row id="0">
<ZCRM_CODE>E_VFXF</ZCRM_CODE>
<Z_DATA_INIZIO>2019-06-01</Z_DATA_INIZIO>
<Z_DATA_FINE>2852-08-31</Z_DATA_FINE>
<ZCRM_INPUT>EQP_VFXM</ZCRM_INPUT>
<Z_INPUT></Z_INPUT></row>
<row id="1">
<ZCRM_CODE>E_VFXF</ZCRM_CODE>
<Z_DATA_INIZIO>2019-06-01</Z_DATA_INIZIO>
<Z_DATA_FINE>2852-08-31</Z_DATA_FINE>
<ZCRM_INPUT>EQP_VFXF1</ZCRM_INPUT>
<Z_INPUT></Z_INPUT></row>
<row id="2">
<ZCRM_CODE>E_VFXF</ZCRM_CODE>
<Z_DATA_INIZIO>2019-06-01</Z_DATA_INIZIO>
<Z_DATA_FINE>2852-08-31</Z_DATA_FINE>
<ZCRM_INPUT>EQP_VFXF3</ZCRM_INPUT>
<Z_INPUT></Z_INPUT></row>
<row id="3">
<ZCRM_CODE>E_VFXF</ZCRM_CODE>
<Z_DATA_INIZIO>2019-06-01</Z_DATA_INIZIO>
<Z_DATA_FINE>2852-08-31</Z_DATA_FINE>
<ZCRM_INPUT>EQP_VFXF2</ZCRM_INPUT>
<Z_INPUT></Z_INPUT></row></ZINT_OPERAND>
<ZINT_ACCISE/></row></XT_SFDC>
<X_HEAD>
<ZID_FLUSSO>165420000000001</ZID_FLUSSO>
<ZCOD_FLUSSO>IN1</ZCOD_FLUSSO>
<ZPROCESS>ATT</ZPROCESS>
<ZDATA_RIC>2019-07-01</ZDATA_RIC>
</X_HEAD>
<X_N_RECORDS>1</X_N_RECORDS>
</import>
<export>
<YT_ESITI>
<row id="0">
<ZID_LETTURA>A0U1X000000GQ1P</ZID_LETTURA>
<ZCOD_ERR>INT_004</ZCOD_ERR>
<ZDESCR>ERRORE LISTA VALORI</ZDESCR>
<ZMOTIVAZIONE>Valore 35.23.00 non ammesso per campo BRANCHE</ZMOTIVAZIONE>
<ZID_LETT_INT>000000000000001</ZID_LETT_INT></row></YT_ESITI>
<Y_ESITO_HEAD>
<ZID_FLUSSO></ZID_FLUSSO>
<ZESITO>OKP</ZESITO>
<ZCOD_ERR></ZCOD_ERR>
<ZDESCR></ZDESCR>
<ZMOTIVAZIONE></ZMOTIVAZIONE>
</Y_ESITO_HEAD>
</export>
</ZINT_SFDC_COMM>

thanks

Hi nickpeihl,
is it allowed to upgrade only Kibana with a newver version?
(I don't wnt to change my elasticsearch version v. 6.3)
Many thanks

Just to update, this problem seems to exist in the 7.2 version
too.