Scripted fields vs real fields

[mtudisco]

Hi
i've defind some scripted fields for my index patterns, but have some doubts:

1. When you define them you get a warning message "Scripted fields can be used to display and aggregate calculated values. As such, they can be very slow, and if done incorrectly, can cause Kibana to be unusable. There's no safety net here. If you make a typo, unexpected exceptions will be thrown all over the place!"
   I'm using them for calculations, for instance storage.used=storage.size-storage.free_space, not for aggregations. Are they slow and it should be recomended to store the actual value (total_ammount)?
2. I'm collecting different types of metrics from a system and store all of them in an index, all related metrics are stored in a nested field, for instance storage.filesystem, storage.free_space, storage.size, cpu.used, cpu.free. Events are of type storage and cpu for instance, events of type cpu only have cpu.* fields not storage. But if I defin a scripted field storage.used=storage.size-storage.free_space then in events of type cpu i also gete a field storage.used when listing in kibana. Is there any way to make scripted field listed on certain types of documents?

thanks

[flash1293 (Joe Reuter)]

Hi @mtudisco,

thanks for your questions.

1. It depends on your use case - when you only work with small amounts of data, scripted fields can be fine. It is possible though you run into performance problems when scaling up your cluster and you suddenly have terabytes of data to go through. A common user story is to start with scripted fields to figure out what exactly you need and once you know switching to doing this calculation during ingest and storing the calculated value in a real field. This way you are flexible to make changes during the design phase of your setup but it's fast even for a lot of data in everyday use.
2. Scripted fields are defined on an index pattern level and thus will be executed for each document in this index pattern. One solution I can think of is to store the different events in different indices and using multiple index pattern like this: my-metric-indices-* when you need all events, my-metric-indices-cpu-* if you are interested in just the cpu events (on this index pattern the scripted fields for cpu stuff would be defined) and my-metric-indices-storage-* for the storage events.

[mtudisco]

Thanks Joe for you clear response!

1. Thats a good suggestion. In my case i will not have Terabytes of data, i expect to have monthly indexes of about 2 to 5GB and i'll be storing just one year of information at most.
2. Separating events in different indexes in my case would make me have severall small indexes, thats why i decided to store all the information in the same index, very similar to the way metricbeat does, one index for different metrics.