What is excluded column in kibana index pattern?

Matthew_Field · October 20, 2020, 10:35am

I am trying to create an index pattern in kibana and setting the time field to be @timestamp.
The index pattern apparently creates OK with @timestamp as the time filter, but when i refresh the fields, the time filter gets changed to timestamp (I have both timestamp and @timestamp fields in data).

When i look at the fields, I can see that @timestamp has a green dot in the "excluded" column. Why is that, and what does it mean?

Thanks for any help.

matw · October 20, 2020, 10:56am

Hi

Those are the fields excluded by source filters

So have a look if you've some Source filters configured.

Best,
Matthias

llamskt · October 21, 2020, 4:18am

I've noticed this too for a recently created index pattern. If I recreate the index pattern it will work for a short while in the discover tab, but after some time I receive an error, discover becomes unusable for the index pattern, and @timestamp will appear in the index pattern's source filter all by itself. I haven't configured any source filters.

Could you explain in greater detail why this source filter is appearing by itself?
Is it possible to prevent this from happening?
currently v7.9.2 elastic/kibana

matw · October 21, 2020, 7:25am

It should no appear by itself, could you provide some details / screens of the error that makes Discover unusable after a while? thx!

Matthew_Field · October 21, 2020, 8:16am

Thanks for your help Matthias. In my case, the error is reproducible using the following procedure:

I create an index pattern selecting time filter @timestamp from drop down.

Screenshot from 2020-10-21 10-06-031453×603 54.6 KB
The pattern is created (apparently) correctly with no source filters.
I press the refresh field list button (top right corner of sceen)
The time filter has changed from @timestamp to timestamp and a source filter has been applied (it wasn't me...)

Version 7.6.1

matw · October 21, 2020, 8:42am

I see, is @timestamp an alias of timestamp

Matthew_Field · October 21, 2020, 9:50am

Not as far as i know... there is no alias setup in the mapping.

eg. GET myindex/_mapping/field/@timestamp
Doesnt show up any alias set up there.

Is there any thing else i should run to check that?

matw · October 21, 2020, 3:58pm

So, refreshing the index pattern should not exclude fields, just refresh the fieldllist. I wonder if an external source is creating or updating the index pattern? I think I've seen the wazuh-alerts-* index before, how are the log messages ingested in this case? thx

matw · October 21, 2020, 4:00pm

you could also record and share HAR file when refreshing fields, so we could have a closer look, what's happening.

llamskt · October 21, 2020, 5:10pm

Steps to reproduce:

Create index pattern from filebeat-7.9.2 index (doesn't matter if I designate @timestamp as time-filter or not)
Wait about 5 minutes (index pattern will work during this time and not have source filter present or @timestamp field marked as excluded)
View index pattern in Discover tab, receive following error:

 FieldParamType/_this.deserialize@https://some.url.com/33984/bundles/plugin/data/data.plugin.js:9:345453
setParams/<@https://some.url.com/33984/bundles/plugin/data/data.plugin.js:9:362647
setParams@https://some.url.com/33984/bundles/plugin/data/data.plugin.js:9:362156
set@https://some.url.com/33984/bundles/plugin/data/data.plugin.js:9:368734
setType@https://some.url.com/33984/bundles/plugin/data/data.plugin.js:9:368146
AggConfig@https://some.url.com/33984/bundles/plugin/data/data.plugin.js:9:361885
AggConfigs/<@https://some.url.com/33984/bundles/plugin/data/data.plugin.js:9:375134
AggConfigs/<@https://some.url.com/33984/bundles/plugin/data/data.plugin.js:9:375555
AggConfigs@https://some.url.com/33984/bundles/plugin/data/data.plugin.js:9:375516
createAggConfigs@https://some.url.com/33984/bundles/plugin/data/data.plugin.js:14:318274
_callee2$@https://some.url.com/33984/bundles/plugin/visualizations/visualizations.plugin.js:9:304414
l@https://some.url.com/33984/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155323
s/o._invoke</<@https://some.url.com/33984/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155077
_/</e[t]@https://some.url.com/33984/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155680
vis_asyncGeneratorStep@https://some.url.com/33984/bundles/plugin/visualizations/visualizations.plugin.js:9:300183
_next@https://some.url.com/33984/bundles/plugin/visualizations/visualizations.plugin.js:9:300519

Check index pattern and find @timestamp field is excluded and source filter has been created.

error21189×206 13.2 KB

The issue has appeared around the time of upgrading filebeat elastic and kibana to 7.9.2. I have deleted the .kibana system index and optimize folders and restarted kibana as troubleshooting steps already. I have a separate filebeat instance running version 7.8 that is unaffected by this issue and is running fine. Both filebeat instances have similar configurations and the same ILM policy. I have also tried deleting the index itself and starting the filebeat service again with no luck. The only debug error I see from kibana is:

 {"type":"log","@timestamp":"2020-10-21T16:48:26Z","tags":["debug","plugins","usageCollection","collector-set"],"pid":2065,"message":"not sending [kibana_settings] monitoring document because [undefined] is null or invalid."}

Its also worth mentioning I'm using Wazuh's filebeat index template here. The wazuh-alerts index pattern works fine, here is my filebeat config for reference:

## Wazuh - Filebeat configuration file

filebeat.modules:
  - module: wazuh
    alerts:
      enabled: true
    archives:
      enabled: false
# OwlH Module
  - module: owlh
    events:
      enabled: true

filebeat.config.modules:
  enabled: true
  path: ${path.config}/modules.d/*.yml
## OWLH pipeline sync
filebeat.overwrite_pipelines: true


#setup.template:
#  name: "filebeat"
#  pattern: "filebeat-custom-*
#  settings:
setup.template.settings.index.number_of_shards: 1
setup.template.settings.index.number_of_replicas: 0

setup.ilm.enabled: auto
setup.ilm.pattern: "{now/M{yyyy.MM}}-001"
setup.ilm.overwrite: false
setup.ilm.rollover_alias: "filebeat-%{[agent.version]}-custom"
setup.ilm.policy_name: "filebeat-custom"

matw · October 21, 2020, 5:28pm

Thanks for the excellent summary, seems the index pattern changes for some reason? would it be possible, that you could share an exported index pattern shared object before and after the error occurs? you could export it in the saved object part of our stack management. that would be great!
Many thx!

Matthew_Field · October 22, 2020, 6:28am

In my case, the logs are coming from Logstash. Unfortunately the environment is completely locked down and i am not permitted to share a HAR file, but thanks for your offer.

matw · October 22, 2020, 6:40am

Yes, sharing such data is not possible. There is one thing you could do, export the saved object containing the index pattern before and after the change, diff both exports , share what changed (of course after checking it doesn't contain sensitive data, yes). This would help a lot.

matw · October 22, 2020, 3:21pm

thx @llamskt for sharing you saved objects, so comparing pre (no troubles) and post (troubles) state

It's clear, that the index pattern was updated, the question is, how

matw · October 22, 2020, 3:25pm

diging a bit deeper, it seems that wazuh-kibana-app could be the source of this update

it looks like it's updating the index pattern, adding formatting and sourceFilter:

github.com

wazuh/wazuh-kibana-app/blob/b1aff90794dea780db5a210fd9a01328c0eebaf1/server/lib/elastic-wrapper.js#L281-L300


const data = await this.elasticRequest.callWithInternalUser('update', {
  index: this.WZ_KIBANA_INDEX,
  type: '_doc',
  id: id.includes('index-pattern:') ? id : 'index-pattern:' + id,
  body: {
    doc: {
      type: 'index-pattern',
      'index-pattern': {
        timeFieldName: 'timestamp',
        fields: currentFieldsString,
        fieldFormatMap: `{
            "data.virustotal.permalink":{"id":"url"},
            "data.vulnerability.reference":{"id":"url"},
            "data.url":{"id":"url"}
          }`,
        sourceFilters: '[{"value":"@timestamp"}]'
      }
    }
  }
});

So the solution to this problem might be the configuration of wazuh?

llamskt · October 22, 2020, 3:44pm

Thank you for the insight! I'll check and update here if I find a solution there.

edit: deleted

Matthew_Field · October 23, 2020, 7:52am

Yes, thankyou Matthias and Matthew (that makes 3 of us!) for your help we are using this Kibana app, i think you have found the issue.

system · November 20, 2020, 7:52am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Different time field instead of @timestamp and filters Kibana	3	4355	June 23, 2021
Can't use anymore @timestamp after a time filter change Kibana	3	577	December 25, 2019
Unable to browse another index pattern Kibana	3	748	July 6, 2017
@timestamp not appearing for a particular index in kibana Kibana	5	5245	March 20, 2017
Index pattern empty when choosing @timestamp Kibana	2	654	November 8, 2019

What is excluded column in kibana index pattern?

Related topics