Hi Everyone,
I am new to kibana. Need your help to solve the following problem which am facing from a day.
I am working on kibana scripted field approach to transform a field value by accessing it using "doc[some_field].value" statement and perform some computation using groovy script then finally return a new field.
Problem:
"doc[some_field].value" is returning null in-case if the field value is large.
Kibana version is : 6.0.1
field name: message.keyword => doc[message.keyword].value
language: painless
Type: String
Format: default(String)
New field name: channel
example script: if (doc['message.keyword'].value != null) {
return "data-present";
} else {
return "data-not-present";
}
for one set of events the above script is returning "channel" value as "data-present" and for rest of the events script is returning "data-not-present".
Especially "data-not-present" value is getting returned incase if the accessing field value is larger in length.
is this the expected behavior? if not - please could someone help me what is the mistake am doing here.
Sample long message => doc[message.keyword].value:
INFO 2017-02-01 17:01:12,628 c.i.c.f.SimpleJmsSender QW-1-my_Name Annamiyyya Sending to dstntn=abcd_bdc_kch prsst=true crrltn: 'null' prpty[J=1] pyld=<?xml version='1.0' encoding='UTF-8' standalone='yes'?><Order><TimeStamps><SubmittedAt>0</SubmittedAt><ReceivedAt>0</ReceivedAt><CompletedAt>0</CompletedAt><OSReceivedAt>1514826072209</OSReceivedAt><ManualStartAt>0</ManualStartAt><ManualProcessAt>0</ManualProcessAt><ConfirmedAt>0</ConfirmedAt><FinishedAt>0</FinishedAt></TimeStamps><RequestDetails><ExecutionInstruction>ABC</ExecutionInstruction><AccountCurrencyCode>#.</AccountCurrencyCode><AccountId>MY_Name</AccountId><AdvancedOrderType></AdvancedOrderType><AttachedOrderIndicator>330</AttachedOrderIndicator><BookingSystem>fooooooo</BookingSystem><BusinessArea>busy</BusinessArea><Channel>fjdnfj</Channel><ChannelId>6374j8485</ChannelId><ClientCategory></ClientCategory><ClientMarketGroup></ClientMarketGroup><CurrentLevel>0</CurrentLevel><DealId>hfjfnjkgbgdb</DealId><DealLevel>0</DealLevel><DealReference>hdgdjekifindj</DealReference><Deposit>0</Deposit><DepositAccountCurrency>0</DepositAccountCurrency><DigitalSignature></DigitalSignature><Direction>+</Direction><DisplayOrderLevel>748595</DisplayOrderLevel><DisplayOrderSize>1</DisplayOrderSize><DisplayProfitLoss></DisplayProfitLoss><ExpiryTime></ExpiryTime><ForceOpen>N</ForceOpen><ForeignOfficeAccount></ForeignOfficeAccount><IsControlledRisk>N</IsControlledRisk><IsSwapTrade>N</IsSwapTrade><IsReinject>N</IsReinject><Locale>en_GB</Locale><LimitDistance>0</LimitDistance><LimitLevel>0</LimitLevel><Market><MktCommod></MktCommod><EpicCode>hcdasjsdfsjfdsajfj</EpicCode><Period> jdsajfdk989</Period><DisplayPeriod> dfdkjkjvf89</DisplayPeriod><MarketName>ewjk/sdhjdsd</MarketName><DisplayMarketName>fjkf/kjds</DisplayMarketName><MarketStatus></MarketStatus><Level1></Level1><Level2></Level2><Level3></Level3><LotSize>10</LotSize><InstrumentLotSize>10</InstrumentLotSize><L2LookupCode>cd/dd</L2LookupCode><ContractId>dsvvddvd</ContractId><ContractVersion>2</ContractVersion><TradableId>dcddcdd</TradableId><TradableVersion>1</TradableVersion></Market><MarketFlipped>0</MarketFlipped><MessageId></MessageId><OrderCurrencyCode>#.</OrderCurrencyCode><PermissionGroup></PermissionGroup><PriceType>S</PriceType><RequestType>C</RequestType><SessionId></SessionId><SiteType>C</SiteType><SourceType>C</SourceType><Spread>0.0</Spread><Size>10</Size><StopDistance>0</StopDistance><StopLevel>0</StopLevel><TimeTillCancelled></TimeTillCancelled><TradingSystem>IGG</TradingSystem><TriggerLevel>8890</TriggerLevel><Type>1</Type><UKExpiryTime></UKExpiryTime><Unit>C</Unit><WebSiteId>igm</WebSiteId><PartialClose>N</PartialClose><PriceImprovement>N</PriceImprovement><RequestedDealLevel>0</RequestedDealLevel><POAName></POAName><TrailingStopDistance>0</TrailingStopDistance><TrailingStopIncrement>0</TrailingStopIncrement><TrailingTriggerDistance>0</TrailingTriggerDistance><TrailingTriggerIncrement>0</TrailingTriggerIncrement><ScalingFactor>10000</ScalingFactor><OrderType>D</OrderType><TmInForce>3</TmInForce></RequestDetails><Result><ActionType>D</ActionType><Action>REJECT</Action><FailedCheck>MARKET_OFFLINE</FailedCheck><Check><CheckName>MARKET_OFFLINE</CheckName><CheckValue></CheckValue><CheckUnit></CheckUnit></Check><UserCode></UserCode><ManualResult></ManualResult><DMAOrderId>dsds:50364~27</DMAOrderId><IGBid>9303.2</IGBid><IGAsk>8890</IGAsk></Result><Source><Internet><IPAddress>3340439434344</IPAddress></Internet></Source><Summary></Summary><ResultCode>normal</ResultCode><ResultDescription>MARKET_OFFLINE</ResultDescription><DisplayResult></DisplayResult><DisplayChannel>PureDMA</DisplayChannel><DisplayTime></DisplayTime><UpdateType>Order</UpdateType><DealResults/><ChangeInfo><Channel>PureDMA</Channel></ChangeInfo><CorpAction></CorpAction></Order>
Note: message.keyword is a searchable and aggregatable field on kibana.