Scripted field: Read field data from a document using doc[‘some_field’].value return null in some cases

Elastic Stack Kibana
1

hi,
on my document I have a field always populated. It's a string field and it can contains data like json or XML.
I created this scripted field to find problem.

def myField = doc['myfield'].value;
if (payload != null){
return payload;
}
else{
return "payload is null";
}

Now, in some cases I can see the payload data, but when I have an XML inside "myField", the scripted fields return "payload is null".
I don't know the reason, it's unbelievable.
Maybe the length of the string in XML, could it be?

image
image.png1074×382 23.3 KB

but the payload is not null, this is the field content

image
image.png852×572 39.7 KB

any ideas?

2

Hi, I'm not able to replicate the issue.

  • Which version of Elasticsearch and Kibana are you running?
  • What is the mapping of your field? i.e. text or keyword?

If the field is mapped as text you might need to use params['_source']['myfield'] to get the value of the field. See the last paragraph on this page.

3

Tnx nick,
the versions are 6.3 for both.
I tried your
doc['event.payload].value --> works only in some cases, as described...
params['_source']['event.payload'] --> it doesn't work (I haven t errors but my fields seems to be always empty).

ELK:

elk-conf
elk-conf.gif928×500 70.5 KB

KIBANA

kib
kib.gif869×457 28.7 KB

In this case, doc['event.payload].value works (with this content):

noproblem
noproblem.gif1231×652 71.2 KB

In this case, doc['event.payload] is null (but it's not null!)

problem
problem.gif1020×676 87.6 KB

According to me the problem is related to the size(length) of field (event.payload); in the first example with XML, there are a lot of characters, could it be?
Tnx

4

Can you share the entire XML string that is failing? This may be a bug in the 6.3.0 version and I'd like to see if it has been fixed in subsequent releases.

5

Thanks nick,
I'm sharing the XML string that is failing, but the problem is related to all XML explicit content... Could the problem be related to XML declaration "<?xml version="1.0"...."

<?xml version='1.0' encoding='UTF-8'?>
<ZINT_SFDC_COMM>
<import>
<XT_SFDC>
<row id="0">
<ZID_RECORD>a0u1X000000gQ1p</ZID_RECORD>
<ZTIPOOPER>SWIN</ZTIPOOPER>
<DATA_EVENTO>2019-06-01</DATA_EVENTO>
<BP_CRM>0000004</BP_CRM>
<CC_CRM>BAN-0000001</CC_CRM>
<IMPIANTO_CRM>ABC-000001</IMPIANTO_CRM>
<CONTR_CRM>ASN-00000001</CONTR_CRM>
<BP_SAP></BP_SAP>
<CC_SAP></CC_SAP>
<POD_SAP>IT001E0000001</POD_SAP>
<CONTR_SAP></CONTR_SAP>
<IMPIANTO_SAP></IMPIANTO_SAP>
<BP_SAP_OLD></BP_SAP_OLD>
<CC_SAP_OLD></CC_SAP_OLD>
<CONTR_SAP_OLD></CONTR_SAP_OLD>
<TYPE>2</TYPE>
<NAME_FIRST></NAME_FIRST>
<NAME_LAST></NAME_LAST>
<NAME_ORG1>ENERGIA MOCK S.R.L.</NAME_ORG1>
<LEGAL_ENTITY>02</LEGAL_ENTITY>
<AUGRP>IT01</AUGRP>
<BPKIND>Z007</BPKIND>
<CF_TAXNUM></CF_TAXNUM>
<PI_TAXNUM>02000000001</PI_TAXNUM>
<CITY1_ESAZ>Milan</CITY1_ESAZ>
<COUNTRY_ESAZ>IT</COUNTRY_ESAZ>
<HOUSE_NUM1_ESAZ>12</HOUSE_NUM1_ESAZ>
<NAME_CO_ESAZ></NAME_CO_ESAZ>
<FLOOR_ESAZ></FLOOR_ESAZ>
<POST_CODE1_ESAZ>21052</POST_CODE1_ESAZ>
<REGION_ESAZ>VA</REGION_ESAZ>
<STREET_ESAZ>12 VIA SILVIO</STREET_ESAZ>
<ZBP_ISTATLOC>012026</ZBP_ISTATLOC>
<ROOMNUMBER_ESAZ></ROOMNUMBER_ESAZ>
<CITY1_CLIENTE>Milano</CITY1_CLIENTE>
<COUNTRY_CLIENTE>IT</COUNTRY_CLIENTE>
<HOUSE_NUM1_CLIENTE>12</HOUSE_NUM1_CLIENTE>
<NAME_CO_CLIENTE></NAME_CO_CLIENTE>
<FLOOR_CLIENTE></FLOOR_CLIENTE>
<POST_CODE1_CLIENTE>21052</POST_CODE1_CLIENTE>
<REGION_CLIENTE>VA</REGION_CLIENTE>
<STREET_CLIENTE>VIA SILVIO</STREET_CLIENTE>
<ROOMNUMBER_CLIENTE></ROOMNUMBER_CLIENTE>
<ZBP_ISTATLOC_CLIENTE>012026</ZBP_ISTATLOC_CLIENTE>
<TEL_NUMBER1></TEL_NUMBER1>
<SMTP_ADDR>testmail@gmail.com</SMTP_ADDR>
<PEC_SMTP_ADDR>testmail@gmail.com</PEC_SMTP_ADDR>
<PEC_SOLLECITI></PEC_SOLLECITI>
<CCNUM></CCNUM>
<DATA_INIZIO_CCNUM>0000-00-00</DATA_INIZIO_CCNUM>
<DATA_FINE_CCNUM>0000-00-00</DATA_FINE_CCNUM>
<BANKS></BANKS>
<BANKL></BANKL>
<BANKN></BANKN>
<BKONT></BKONT>
<IBAN></IBAN>
<Z_MANDATO></Z_MANDATO>
<STAT_MAND></STAT_MAND>
<COD_CUCSIA></COD_CUCSIA>
<CREDITORID></CREDITORID>
<DATASOTT>0000-00-00</DATASOTT>
<CODFISCFIRMA></CODFISCFIRMA>
<NOMEFIRMA></NOMEFIRMA>
<COGNOMEFIRMA></COGNOMEFIRMA>
<AB_MANDATO>0000-00-00</AB_MANDATO>
<BIS_MANDATO>0000-00-00</BIS_MANDATO>
<ZPUBB>N</ZPUBB>
<LGZUSATZ_BP></LGZUSATZ_BP>
<VBUND></VBUND>
<Z_COBRAND>IT01</Z_COBRAND>
<IKEY>Z2</IKEY>
<ZAHLKOND>DM30</ZAHLKOND>
<Z_MODINV>ZD</Z_MODINV>
<Z_SINTDETT>S</Z_SINTDETT>
<Z_DUALCODE>N</Z_DUALCODE>
<IPA_CODE></IPA_CODE>
<IPA_BEGDA>0000-00-00</IPA_BEGDA>
<Z_OU></Z_OU>
<Z_IOU></Z_IOU>
<Z_CUP></Z_CUP>
<Z_CIG></Z_CIG>
<Z_CONSIP></Z_CONSIP>
<Z_RIFAMM></Z_RIFAMM>
<EZAWE>B</EZAWE>
<ZCOD_ASS></ZCOD_ASS>
<ZCA_RES>Y</ZCA_RES>
<ZCA_TOP></ZCA_TOP>
<ZCOD_RES>AGESP</ZCOD_RES>
<SPARTE>E</SPARTE>
<GEMFAKT>2</GEMFAKT>
<KOFIZ>01</KOFIZ>
<ZCANACQ>AG</ZCANACQ>
<Z_KAM>Default</Z_KAM>
<Z_MERCRIE>SL</Z_MERCRIE>
<Z_PRODOTTO>Axpo99912</Z_PRODOTTO>
<Z_PRODOTTO_DESC>Listino Fisso 3Fasce</Z_PRODOTTO_DESC>
<Z_INIZIO>2019-06-01</Z_INIZIO>
<Z_LISTINO>RESEL_FI01386_1907FIX</Z_LISTINO>
<Z_MERCATO>ML</Z_MERCATO>
<Z_DEPOSITO>N</Z_DEPOSITO>
<ZTIP_OFF>B2B</ZTIP_OFF>
<Z_OFF_VERDE>N</Z_OFF_VERDE>
<Z_IMPDEP></Z_IMPDEP>
<ZAGENZIE></ZAGENZIE>
<ZGRUPPO></ZGRUPPO>
<ZCAPOGR></ZCAPOGR>
<ZCOD_OFFERTA_TOP></ZCOD_OFFERTA_TOP>
<ZFREQ>MENS</ZFREQ>
<ZTENS>220</ZTENS>
<Z_DISVENDITORE>Y</Z_DISVENDITORE>
<SERVICE>ZDIS</SERVICE>
<BRANCHE>35.23.00</BRANCHE>
<ER_TIPOUT_TF>DO</ER_TIPOUT_TF>
<ED_POTDIS>3.0</ED_POTDIS>
<ED_POTCON>3.0</ED_POTCON>
<ER_RESI_TF>Y</ER_RESI_TF>
<COANC0>1800</COANC0>
<COAND0></COAND0>
<EQ_COAND1></EQ_COAND1>
<EQ_COAND2></EQ_COAND2>
<EQ_COAND3></EQ_COAND3>
<DISDIS></DISDIS>
<EG_ENERG>N</EG_ENERG>
<EG_AUSIL>N</EG_AUSIL>
<EG_DOMD1>N</EG_DOMD1>
<EG_MIS2G></EG_MIS2G>
<EU_CLESEN></EU_CLESEN>
<DIS_CODE>D001</DIS_CODE>
<GU_CLIMP></GU_CLIMP>
<ER_ACC>ACCRES</ER_ACC>
<ED_ACCAUES></ED_ACCAUES>
<EQ_ACCFXES></EQ_ACCFXES>
<EU_STATOF>Attiva</EU_STATOF>
<AB_2G>0000-00-00</AB_2G>
<GF_CG>0.0000000</GF_CG>
<CITY1_FORN>PINEROLO</CITY1_FORN>
<COUNTRY_FORN>IT</COUNTRY_FORN>
<HOUSE_NUM1_FORN>45</HOUSE_NUM1_FORN>
<POST_CODE1_FORN>10064</POST_CODE1_FORN>
<REGION_FORN>TO</REGION_FORN>
<STREET_FORN>VIGONE</STREET_FORN>
<ROOMNUMBER></ROOMNUMBER>
<FLOOR></FLOOR>
<ZOGAL_ISTATLOC>000001</ZOGAL_ISTATLOC>
<LGZUSATZ></LGZUSATZ>
<Z_DAT_REMI></Z_DAT_REMI>
<ZINT_OPERAND>
<row id="0">
<ZCRM_CODE>E_VFXF</ZCRM_CODE>
<Z_DATA_INIZIO>2019-06-01</Z_DATA_INIZIO>
<Z_DATA_FINE>2852-08-31</Z_DATA_FINE>
<ZCRM_INPUT>EQP_VFXM</ZCRM_INPUT>
<Z_INPUT></Z_INPUT></row>
<row id="1">
<ZCRM_CODE>E_VFXF</ZCRM_CODE>
<Z_DATA_INIZIO>2019-06-01</Z_DATA_INIZIO>
<Z_DATA_FINE>2852-08-31</Z_DATA_FINE>
<ZCRM_INPUT>EQP_VFXF1</ZCRM_INPUT>
<Z_INPUT></Z_INPUT></row>
<row id="2">
<ZCRM_CODE>E_VFXF</ZCRM_CODE>
<Z_DATA_INIZIO>2019-06-01</Z_DATA_INIZIO>
<Z_DATA_FINE>2852-08-31</Z_DATA_FINE>
<ZCRM_INPUT>EQP_VFXF3</ZCRM_INPUT>
<Z_INPUT></Z_INPUT></row>
<row id="3">
<ZCRM_CODE>E_VFXF</ZCRM_CODE>
<Z_DATA_INIZIO>2019-06-01</Z_DATA_INIZIO>
<Z_DATA_FINE>2852-08-31</Z_DATA_FINE>
<ZCRM_INPUT>EQP_VFXF2</ZCRM_INPUT>
<Z_INPUT></Z_INPUT></row></ZINT_OPERAND>
<ZINT_ACCISE/></row></XT_SFDC>
<X_HEAD>
<ZID_FLUSSO>165420000000001</ZID_FLUSSO>
<ZCOD_FLUSSO>IN1</ZCOD_FLUSSO>
<ZPROCESS>ATT</ZPROCESS>
<ZDATA_RIC>2019-07-01</ZDATA_RIC>
</X_HEAD>
<X_N_RECORDS>1</X_N_RECORDS>
</import>
<export>
<YT_ESITI>
<row id="0">
<ZID_LETTURA>A0U1X000000GQ1P</ZID_LETTURA>
<ZCOD_ERR>INT_004</ZCOD_ERR>
<ZDESCR>ERRORE LISTA VALORI</ZDESCR>
<ZMOTIVAZIONE>Valore 35.23.00 non ammesso per campo BRANCHE</ZMOTIVAZIONE>
<ZID_LETT_INT>000000000000001</ZID_LETT_INT></row></YT_ESITI>
<Y_ESITO_HEAD>
<ZID_FLUSSO></ZID_FLUSSO>
<ZESITO>OKP</ZESITO>
<ZCOD_ERR></ZCOD_ERR>
<ZDESCR></ZDESCR>
<ZMOTIVAZIONE></ZMOTIVAZIONE>
</Y_ESITO_HEAD>
</export>
</ZINT_SFDC_COMM>

thanks

6

Hi nickpeihl,
is it allowed to upgrade only Kibana with a newver version?
(I don't wnt to change my elasticsearch version v. 6.3)
Many thanks

7

Just to update, this problem seems to exist in the 7.2 version
too.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.