Control Visualization List Filtering on scripted metric field

[silentfilm (Bruce Calvert)]

I have several Kibana dashboards that use Option List control filters to allow users to easily filter data. Many of the fields that I filter on for the list are scripted fields in the index. This works great most of the time.

However, there is one situation where they don't pull in all possible values. For example, all indexed records contain a hostname, but we use painless scripts to assign them as a ui_hostname, database_hostname, export_hostname, kafka_hostname, etc. depending on the type of data coming in. That way we can easily filter out the messages that don't have anything to do with the ui and just allow the user to select data that came in from ui_hosts. We have found that if hosta and hostb already exist, and data starts coming in from hostc in the middle of the day, that hostc does not appear on the filter pull-down list. After midnight, when a new index file is created, then hostc appears. It is almost as if Kibana only searches the first thousand or so possible values in the index for a field for the option list and then stops looking. Is there a way around this?

[matw (Matthias Wilhelm)]

Hi @silentfilm

Nice avatar you have! Which version of Kibana do you use, and did you configure the number of options in the editor.

Best,
Matthias

[silentfilm (Bruce Calvert)]

I'm using Kibana/Elasticsearch 6.4. We usually set the number of options to be 500, but there is never more than 25 distinct values.

[matw (Matthias Wilhelm)]

I see, and the problem is just with scripted fields?
thx!

[silentfilm (Bruce Calvert)]

Yes, just scripted fields, and just when a scripted field doesn't have any values in it until later in the day after an installation or after a new activation starts populating the scripted field. After the index is one day old, the problem disappears, presumably because the scripted value is continually being populated.

[matw (Matthias Wilhelm)]

Could you provide the REST request + response that's fetching the data of this field, when you go to Visualization and select the input control vis that uses the scripted fields, you can get it in Network Tab of the developer tools of your browser.
How many records do you collect during the day? Scripted fields are much more expensive performance wise.

Many thx!
Best,
Matthias

[silentfilm (Bruce Calvert)]

I do not have a system with the issue on it right now. It takes several hours to duplicate it and I don't currently have a lab system with a heavy load. Here is the page from a typical system with the network tab displayed.

At a customer site, there are 10K or many more records per day with a Kafka topic, but never more than 15-20 distinct values for the topic.

[silentfilm (Bruce Calvert)]

Any suggestions on this issue?

[silentfilm (Bruce Calvert)]

Still hoping for some advice as to how to get the scripted keyword values to populate.

[matw (Matthias Wilhelm)]

I've tried to reproduced that issue with 6.4, but I couldn't. could you share an example code for that scripted fields you're using? Is topic.keyword of you screenshot a scripted field?

thx & best,
Matthias

[silentfilm (Bruce Calvert)]

Actually, we are using Elasticsearch/Kibana 6.7.1. I apologize for stating 6.4 earlier.

"topic" is scripted in the pipeline, but in Kibana-> management, it does not show up as a scripted field. Does this mean that there is a difference between a pipeline scripted field and an index scripted field?

The metric definition:
{\"name\":\"topic\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"topic.keyword\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},

The Kafka metrics pipeline definition (excerpt):
{ "script": { "lang": "painless", "source": "if (ctx.kafka_metric_fields != null) { String application = ctx.kafka_metric_fields[0]; ctx.application = application; if (ctx.kafka_metric_fields[2] ==~ /Partition/) { String topic = ctx.kafka_metric_fields[7]; String partition = ctx.kafka_metric_fields[5]; ctx.topic = topic; ctx.partition = partition;} if (ctx.kafka_metric_fields[3] ==~ /BrokerState/){Integer state = ctx.value; String broker_state = ''; if (state.equals(0)){broker_state = 'NotRunning';} if (state.equals(1)) {broker_state = 'Starting'} if (state.equals(2)) {broker_state = 'RecoveringFromUncleanShutdown'} if (state.equals(3)) {broker_state = 'RunningAsBroker'} if (state.equals(6)) {broker_state = 'PendingControlledShutdown'} if (state.equals(7)) {broker_state = 'BrokerShuttingDown'} ctx.broker_state = broker_state} if (ctx.kafka_metric_fields[2] ==~ /BrokerTopicMetrics/) { if (ctx.kafka_metric_fields[3] ==~ /BytesOutPerSec/) { if ((ctx.kafka_metric_fields.size() > 4)){ctx.topic = ctx.kafka_metric_fields[5]; ctx.topic_bytes_out_per_sec_mean_rate = ctx.mean_rate;ctx.topic_bytes_out_per_sec_m1_rate = ctx.m1_rate;ctx.topic_bytes_out_per_sec_m5_rate = ctx.m5_rate;ctx.topic_bytes_out_per_sec_m15_rate = ctx.m15_rate;} else {ctx.broker_bytes_out_per_sec_mean_rate = ctx.mean_rate;ctx.broker_bytes_out_per_sec_m1_rate = ctx.m1_rate;ctx.broker_bytes_out_per_sec_m5_rate = ctx.m5_rate;ctx.broker_bytes_out_per_sec_m15_rate = ctx.m15_rate;}}} if (ctx.kafka_metric_fields[2] ==~ /BrokerTopicMetrics/) { if (ctx.kafka_metric_fields[3] ==~ /BytesInPerSec/) { if ((ctx.kafka_metric_fields.size() > 4)){ctx.topic = ctx.kafka_metric_fields[5]; ctx.topic_bytes_in_per_sec_mean_rate = ctx.mean_rate;ctx.topic_bytes_in_per_sec_m1_rate = ctx.m1_rate;ctx.topic_bytes_in_per_sec_m5_rate = ctx.m5_rate;ctx.topic_bytes_in_per_sec_m15_rate = ctx.m15_rate;} else {ctx.broker_bytes_in_per_sec_mean_rate = ctx.mean_rate;ctx.broker_bytes_in_per_sec_m1_rate = ctx.m1_rate;ctx.broker_bytes_in_per_sec_m5_rate = ctx.m5_rate;ctx.broker_bytes_in_per_sec_m15_rate = ctx.m15_rate;}} if (ctx.kafka_metric_fields[3] ==~ /MessagesInPerSec/) { if ((ctx.kafka_metric_fields.size() > 4)){**ctx.topic = ctx.kafka_metric_fields[5];** ctx.topic_messages_in_per_sec_mean_rate = ctx.mean_rate;ctx.topic_messages_in_per_sec_m1_rate = ctx.m1_rate;ctx.topic_messages_in_per_sec_m5_rate = ctx.m5_rate;ctx.topic_messages_in_per_sec_m15_rate = ctx.m15_rate;} else {ctx.broker_messages_in_per_sec_mean_rate = ctx.mean_rate;ctx.broker_messages_in_per_sec_m1_rate = ctx.m1_rate;ctx.broker_messages_in_per_sec_m5_rate = ctx.m5_rate;ctx.broker_messages_in_per_sec_m15_rate = ctx.m15_rate;}}}}" } }, { "script": { "lang": "painless", "source": "if (ctx.kafka_metric_fields != null && ctx.source != null && ctx.name != null && ctx.name.startsWith('kafka.cluster.Partition.UnderReplicated.partition') && ctx.value == 1) { **ctx.topic = ctx.kafka_metric_fields[7]**; ctx.under_replicated_partition = ctx.kafka_metric_fields[5];}" } },

[matw (Matthias Wilhelm)]

Yes there's a big difference, you're referring to an ingest pipeline field, so the field is already part of the Elasticsearch records, since it's calculated at ingestion. A scripted field of an index pattern is a computed field at query time. The pattern you're using is better. However about the field values you are missing, if they are in the records, they should be available in the list filtering, do you now have an environment where you could reproduce the behavior?

Thx & Best,
Matthias

[silentfilm (Bruce Calvert)]

I have reinstalled my system and reproduced the issue. As you can see from the "Topic Bytes In" chart there are certainly topics available. A check of Kibana -> Discover also shows topics populated. But nothing is available in the pull-down selection.

[silentfilm (Bruce Calvert)]

And today, with no kafka traffic on my system except for the 30 minutes I ran it yesterday, I have selections from the pull-down menu. As soon as a new index is created at midnight for the new day, the problem disappears.

[matw (Matthias Wilhelm)]

This is odd, may I ask which kind of mapping you're using for those fields?
thx & best,
Matthias

[silentfilm (Bruce Calvert)]

Is this what you mean?

[silentfilm (Bruce Calvert)]

Still looking for advice on this issue.

[silentfilm (Bruce Calvert)]

Keeping this topic open because I still don't have a possible solution.