Problem with scripted field kibana when check doc['message.keyword'] empty

Elastic Stack Kibana
1

I want to check doc['message.keyword'] message field whether is exist or empty, so I use this statement :

if(doc['message.keyword'].empty ) {
    return "unavailable";
} else {
    return "available";
}

(this statement I read in this link in github: https://gist.github.com/jclosure/8e103dee2f7e9491845a2c0bb64c6b7a)

But I don't know why in some case it return unavailable but it actually is not. I saw it in Kibana dicover. This is two case example:

unavailable:

message:1600300703468|sendRequest_http10.20.30.1448090BO||wsdlUrl:http://10.20.30.400:5000/BOC2Services/boc2/importantProgramServiceImpl?wsdl: requestContent: <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ser="http://abcdde.abcd.bccs.viettel.com/">
<soapenv:Header>
      <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
         <wsse:UsernameToken>
         <wsse:Username>user</wsse:Username>
         <wsse:Password Type="http://docs.oas6is-open.org/ws/2006/01/oasis-200401-wss-username-token-profile-1.0#PasswordText">pass</wsse:Password>
        </wsse:UsernameToken>
 </wsse:Security>
</soapenv:Header>   <soapenv:Body><ser:getPrograms xmlns:ns2="http://service.boc2.bccs.viettel.com/"><inputSearchProgram><groupType>1</groupType><password>pass</password><username>user</username></inputSearchProgram></ser:getPrograms></soapenv:Body>
</soapenv:Envelope> output exception: java.net.ConnectException: Connection refused|1600300703480|12|1|Loi sendRequest exception 4

unavailable:

message:202009170658:1600300697981 CallableProcessor Error Exception with type = WARNING_CARE_BY_DAY gson.toJson(input) {"userName":"HANTT","ip":"10.20.30.400","staffId":268,"prepaid":true}
java.net.NoRouteToHostException: No route to host
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.7.0_05]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ~[?:1.7.0_05]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) ~[?:1.7.0_05]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) ~[?:1.7.0_05]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) ~[?:1.7.0_05]
	at java.net.Socket.connect(Socket.java:579) ~[?:1.7.0_05]
	at java.net.Socket.connect(Socket.java:528) ~[?:1.7.0_05]
	at java.net.Socket.<init>(Socket.java:425) ~[?:1.7.0_05]
	at java.net.Socket.<init>(Socket.java:280) ~[?:1.7.0_05]
	at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80) ~[commons-httpclient-3.1.jar:?]
	at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122) ~[commons-httpclient-3.1.jar:?]
	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) ~[commons-httpclient-3.1.jar:?]
	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) ~[commons-httpclient-3.1.jar:?]
	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) ~[commons-httpclient-3.1.jar:?]
	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) ~[commons-httpclient-3.1.jar:?]
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) ~[commons-httpclient-3.1.jar:?]
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) ~[commons-httpclient-3.1.jar:?]
	at com.viettel.bss.smartphonev2.bccsgw.DirectWebServiceClient.sendRequest(DirectWebServiceClient.java:235) ~[classes/:?]
	at com.viettel.bss.smartphonev2.bccsgw.DirectWebServiceClient.sendRequest(DirectWebServiceClient.java:153) ~[classes/:?]
	at com.viettel.bss.smartphonev2.bccsgw.DirectWebServiceClient.sendRequestNew(DirectWebServiceClient.java:129) ~[classes/:?]
	at com.viettel.bss.smartphonev2.bccsgw.DirectWebserviceMarshalUnMarshalObject.getResponseWhenCallServiceHaveSecurityHeader(DirectWebserviceMarshalUnMarshalObject.java:90) ~[classes/:?]
	at com.viettel.bss.smartphonev2.business.BocBusiness.findAlarmSalePointsByDayOfStaff(BocBusiness.java:1750) ~[classes/:?]
	at com.viettel.bss.smartphonev2.processor.business.WarningCareByDayBusiness.warningCareByDay(WarningCareByDayBusiness.java:43) ~[classes/:?]
	at com.viettel.bss.smartphonev2.processor.CallableProcessor.call(CallableProcessor.java:80) ~[classes/:?]
	at com.viettel.bss.smartphonev2.processor.CallableProcessor.call(CallableProcessor.java:39) ~[classes/:?]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[?:1.7.0_05]
	at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[?:1.7.0_05]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [?:1.7.0_05]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [?:1.7.0_05]
	at java.lang.Thread.run(Thread.java:722) [?:1.7.0_05]

available:

message:callId = 1600300697981 RESPONSE Exception :http://10.20.300.40:5000/abcdServices/boc2/SalePointsRouteService?wsdl detail: java.net.NoRouteToHostException: No route to host

Thank you.

2

Finally, I found that my message field is ignore_above:256, so although it display in UI, it is not index or stored.

3

Glad you figured it out - this is a bit confusing. Discover is using the _source object of a document in most cases, giving you the document in the form it was ingested. However, scripted fields (and visualizations using aggregations for that matter), rely on the values in the index. Those are often the same, but not always (like in the ignore_above case).

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.