Basic Help With Script Field

[gbarrett (Geoffrey Barrett)]

Hi all,

I'm new to the ELK Stack, and I'm struggling to create a scripted field properly.

I am polling uptime via SNMP and get a value back in seconds. (Actually in TIMETICKS but yay LogStash for allowing me to convert to seconds before it gets to ElasticSearch.) This is isolated to a specific polling index and if I search within that index using type_instance:prepp_uptime, I see results that look like this:

My goal is to create a scripted field called "Uptime" that is derived from prepp_uptime but displayed as a Duration that's nice and human readable. This is how I configure the top part of the scripted field form:

This is the script I'm attempting to use:

doc['type_instance'].value == 'prepp_uptime' ? doc['value'].value : 0

Here it is after creation:

And here's the error I get when I return to Discover:

Discover: runtime error

 Less Info
OK
Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"script_exception","reason":"runtime error","script_stack":["org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:301)","org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:115)","org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:279)","org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:88)","org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:85)","java.security.AccessController.doPrivileged(Native Method)","org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:85)","org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:39)","doc['type_instance'].value == 'prepp_uptime' ? doc['value'].value : 0","    ^---- HERE"],"script":"doc['type_instance'].value == 'prepp_uptime' ? doc['value'].value : 0","lang":"painless"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"fetch","grouped":true,"failed_shards":[{"shard":0,"index":"sand-poc_1-polling-2018.02.21","node":"md_sbcCUSRqgvhnYdPa_Lw","reason":{"type":"script_exception","reason":"runtime error","script_stack":["org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:301)","org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:115)","org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:279)","org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:88)","org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:85)","java.security.AccessController.doPrivileged(Native Method)","org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:85)","org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:39)","doc['type_instance'].value == 'prepp_uptime' ? doc['value'].value : 0","    ^---- HERE"],"script":"doc['type_instance'].value == 'prepp_uptime' ? doc['value'].value : 0","lang":"painless","caused_by":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [type_instance] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}}}]},"status":500}
    at http://127.0.0.1:9999/bundles/commons.bundle.js?v=16573:21:575797
    at Function.Promise.try (http://127.0.0.1:9999/bundles/commons.bundle.js?v=16573:21:474114)
    at http://127.0.0.1:9999/bundles/commons.bundle.js?v=16573:21:473502
    at Array.map (<anonymous>)
    at Function.Promise.map (http://127.0.0.1:9999/bundles/commons.bundle.js?v=16573:21:473460)
    at callResponseHandlers (http://127.0.0.1:9999/bundles/commons.bundle.js?v=16573:21:575375)
    at http://127.0.0.1:9999/bundles/commons.bundle.js?v=16573:21:564721
    at processQueue (http://127.0.0.1:9999/bundles/vendors.bundle.js?v=16573:123:132456)
    at http://127.0.0.1:9999/bundles/vendors.bundle.js?v=16573:123:133349
    at Scope.$digest (http://127.0.0.1:9999/bundles/vendors.bundle.js?v=16573:123:144239)
    at Scope.$apply (http://127.0.0.1:9999/bundles/vendors.bundle.js?v=16573:123:147018)
    at done (http://127.0.0.1:9999/bundles/vendors.bundle.js?v=16573:123:100026)
    at completeRequest (http://127.0.0.1:9999/bundles/vendors.bundle.js?v=16573:123:104697)
    at XMLHttpRequest.xhr.onload (http://127.0.0.1:9999/bundles/vendors.bundle.js?v=16573:123:105435)

Apologies in advance if this post is formatted incorrectly. First time posting here so I'm a n00b.

Any help would be much appreciated.

[tsullivan (Tim Sullivan)]

Thanks for including the full error message. I see some text in the error that says:

"Fielddata is disabled on text fields by default. Set fielddata=true on [type_instance] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."

The script_stack says:

            "doc['type_instance'].value == 'prepp_uptime' ? doc['value'].value : 0",
            "    ^---- HERE"

I think what this means is that the type_instance field is mapped as text. You might already have a type_instance.keyword field as one of that text's multi-field which you can use:

doc['type_instance.keyword'].value == 'prepp_uptime' ? doc['value'].value : 0

That should help a bit on visualizing this data as a metric. If it is an important metric that you'll keep around, I would suggest using the mutate plugin in Logstash to add_field the add the calculated field as a first-class field in your documents. That will give you much better performance at search time.

[gbarrett (Geoffrey Barrett)]

Hi Tim,

It turns out I did already have a type_instance.keyword field available and that fixed it:

Thank you for your help!

If I decide to write a LogStash mutate filter I'll share it here. For now this is "good enough".

G