SavedObjectsClient doesn't return index patterns with their fields

brandtj · March 6, 2020, 11:40pm

For some reason this call below will return all the index patterns but only some of them have any fields in them even though the index has fields. What's really interesting is that if I do anything with these indices (like look at them from the KIbana UI management page) then the next request will return this index pattern with its fields. What is going on here?

savedObjectsClient.find({type: "index-pattern", fields: ["title", "fields"] perPage: 10000 })

brandtj · March 7, 2020, 4:54pm

Did some more research and it looks like some code in Kibana assumes these fields are going to be in the saved object while others have a comment that say it's an invalid index.

Here is some places where the code assumes that the saved object will have the fields populated
(Timelion)

github.com

elastic/kibana/blob/38fd963ed3916ccbe565ee22551ce0ff65182a85/src/plugins/timelion/server/series_functions/es/index.js#L122


  type: 'index-pattern',
  fields: ['title', 'fields'],
  search: `"${config.index}"`,
  search_fields: ['title'],
});
const indexPatternSavedObject = findResp.saved_objects.find(savedObject => {
  return savedObject.attributes.title === config.index;
});
let scriptedFields = [];
if (indexPatternSavedObject) {
  const fields = JSON.parse(indexPatternSavedObject.attributes.fields);
  scriptedFields = fields.filter(field => {
    return field.scripted;
  });
}


const esShardTimeout = tlConfig.esShardTimeout;


const body = buildRequest(config, tlConfig, scriptedFields, esShardTimeout);


const { callAsCurrentUser: callWithRequest } = tlConfig.esDataClient();

(Lens)

github.com

elastic/kibana/blob/d61ef267d530554cd048b5e602a0b6c2e362185f/x-pack/plugins/lens/server/routes/existing_fields.ts#L181


  .reduce((acc, f) => {
    acc[f.name] = f.path;
    return acc;
  }, {} as Record<string, string>);


const descriptorMap = fieldDescriptors.reduce((acc, f) => {
  acc[f.name] = f;
  return acc;
}, {} as Record<string, FieldDescriptor>);


return JSON.parse(indexPattern.attributes.fields).map(
  (field: { name: string; lang: string; scripted?: boolean; script?: string }) => {
    const path =
      aliasMap[field.name] || descriptorMap[field.name]?.subType?.multi?.parent || field.name;
    return {
      name: field.name,
      isScript: !!field.scripted,
      isAlias: !!aliasMap[field.name],
      path: path.split('.'),
      lang: field.lang,
      script: field.script,

IndexPatternSelect assumes fields not in the saved object must be an invalid index.

github.com

elastic/kibana/blob/096dda6f3460eee926661c7271a397d874e739a5/src/plugins/data/public/ui/index_pattern_select/index_pattern_select.tsx#L145


let savedObjects = await getIndexPatterns(savedObjectsClient, searchValue, savedObjectFields);


if (fieldTypes) {
  savedObjects = savedObjects.filter((savedObject: SimpleSavedObject<any>) => {
    try {
      const indexPatternFields = JSON.parse(savedObject.attributes.fields as any);
      return indexPatternFields.some((field: any) => {
        return fieldTypes?.includes(field.type);
      });
    } catch (err) {
      // Unable to parse fields JSON, invalid index pattern
      return false;
    }
  });
}


if (!this.isMounted) {
  return;
}


// We need this check to handle the case where search results come back in a different

However it's not an invalid index pattern. It's just that the saved objects client is not returning all the data. It seems like the index patterns don't have fields in them until someone calls getFieldsForWildcard which does seem like what happens when you look at the index in the index management page.

So for instance I go to use Timelion and it doesn't work for an index. I hit the index pattern management page look at the index pattern and see that it has fields. Go back to Timelion and it now works. This happens because the saved object now has fields in it because going to the index pattern management page and clicking the index pattern calls that getFieldsForWildcard which then populates the saved object with its fields.

What I don't understand is why is some Kibana code assuming the fields are going to exist and others think that if they don't then the index pattern is invalid. Again it's not that the index pattern is invalid but that the saved object is not populated correctly. I also really don't understand why the index pattern saved object is not ALWAYS populated with the fields. Would really appreciate the Kibana team to look into this. I don't want to have to keep telling my users, "Oh go look at the index pattern from the management page and then go back to what you were trying to do. It will work then."

tylersmalley · March 10, 2020, 3:10am

The saved objects API is returning the data as persisted. When adding an index pattern through the UI, clicking refresh on the index pattern page, or as you discovered any other time getFieldsForWildcard is called - the fields property is populated. We have for a while wanted to remove this field cache kibana#6498 entirely and recently discussed changing how index patterns work kibana#35481.

Are you programmatically creating these index patterns using the saved object API?

system · April 7, 2020, 3:10am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Kibana 7.16.2 - fields array empty for saved_object api call Kibana	4	474	March 3, 2022
Create index patterns / saved objects in plugin Kibana	4	538	November 15, 2019
Using kibana saved_object _find api and return only selected field from nested json Kibana	3	545	April 10, 2019
Are scripted fields saved as objects anywhere Kibana	2	689	February 21, 2018
Retrieve stored index-patterns Kibana	4	1503	December 11, 2017

SavedObjectsClient doesn't return index patterns with their fields

Related topics