Many extra fields displaying after upgrading to 7.8.0

Elastic Stack Kibana
1

After upgrading the full stack to 7.8.0, Kibana now shows numerous fields in a document that it did not before. These are for the filebeat index and all the extra fields are empty.

image
image1375×957 63 KB

Looking at the json document inside Kibana there is a field named fields that has all of these:

"fields": {
    "cef.extensions.flexDate1": [],
    "netflow.flow_end_microseconds": [],
    "netflow.system_init_time_milliseconds": [],
    "netflow.flow_end_nanoseconds": [],
    "misp.observed_data.last_observed": [],
    "netflow.max_flow_end_microseconds": [],
    "file.mtime": [],
    "aws.cloudtrail.user_identity.session_context.creation_date": [],
    "netflow.min_flow_start_seconds": [],
    "misp.intrusion_set.first_seen": [],
    "file.created": [],
    "misp.threat_indicator.valid_from": [],
    "process.parent.start": [],
    "azure.auditlogs.properties.activity_datetime": [],
    "crowdstrike.event.ProcessStartTime": [],
    "zeek.ocsp.update.this": [],
    "crowdstrike.event.IncidentStartTime": [],
    "netflow.observation_time_microseconds": [],
    "event.start": [],
    "cef.extensions.agentReceiptTime": [],
    "cef.extensions.oldFileModificationTime": [],
    "checkpoint.subs_exp": [],
    "event.end": [],
    "netflow.max_flow_end_milliseconds": [],
    "netflow.min_flow_start_nanoseconds": [],
    "zeek.smb_files.times.changed": [],
    "crowdstrike.event.StartTimestamp": [],
    "netflow.flow_start_nanoseconds": [],
    "netflow.flow_start_seconds": [],
    "crowdstrike.event.ProcessEndTime": [],
    "zeek.x509.certificate.valid.until": [],
    "misp.observed_data.first_observed": [],
    "netflow.exporter.timestamp": [],
    "netflow.monitoring_interval_start_milli_seconds": [],
    "cef.extensions.oldFileCreateTime": [],
    "event.ingested": [],
    "@timestamp": [
      "2020-06-21T16:24:56.340Z"
    ],
    "zeek.ocsp.update.next": [],
    "crowdstrike.event.UTCTimestamp": [],
    "tls.server.not_before": [],
    "cef.extensions.startTime": [],
    "netflow.min_flow_start_milliseconds": [],
    "azure.signinlogs.properties.created_at": [],
    "cef.extensions.endTime": [],
    "suricata.eve.tls.notbefore": [],
    "zeek.kerberos.valid.from": [],
    "cef.extensions.fileCreateTime": [],
    "misp.threat_indicator.valid_until": [],
    "crowdstrike.event.EndTimestamp": [],
    "misp.campaign.last_seen": [],
    "cef.extensions.deviceReceiptTime": [],
    "netflow.observation_time_seconds": [],
    "crowdstrike.metadata.eventCreationTime": [],
    "cef.extensions.fileModificationTime": [],
    "tls.client.not_before": [],
    "zeek.smb_files.times.created": [],
    "zeek.smtp.date": [],
    "netflow.collection_time_milliseconds": [],
    "zeek.pe.compile_time": [],
    "netflow.max_flow_end_seconds": [],
    "tls.client.not_after": [],
    "netflow.flow_start_milliseconds": [],
    "event.created": [],
    "package.installed": [],
    "zeek.kerberos.valid.until": [],
    "suricata.eve.flow.end": [],
    "netflow.observation_time_milliseconds": [],
    "netflow.flow_start_microseconds": [],
    "tls.server.not_after": [],
    "netflow.flow_end_seconds": [],
    "process.start": [],
    "suricata.eve.tls.notafter": [],
    "zeek.snmp.up_since": [],
    "azure.enqueued_time": [],
    "netflow.max_flow_end_nanoseconds": [],
    "misp.intrusion_set.last_seen": [],
    "netflow.min_flow_start_microseconds": [],
    "netflow.observation_time_nanoseconds": [],
    "cef.extensions.managerReceiptTime": [],
    "file.accessed": [],
    "netflow.flow_end_milliseconds": [],
    "misp.campaign.first_seen": [],
    "netflow.min_export_seconds": [],
    "suricata.eve.flow.start": [],
    "suricata.eve.timestamp": [
      "2020-06-21T16:24:56.340Z"
    ],
    "cef.extensions.deviceCustomDate1": [],
    "cef.extensions.deviceCustomDate2": [],
    "netflow.monitoring_interval_end_milli_seconds": [],
    "file.ctime": [],
    "crowdstrike.event.IncidentEndTime": [],
    "zeek.smb_files.times.accessed": [],
    "zeek.ocsp.revoke.time": [],
    "zeek.x509.certificate.valid.from": [],
    "netflow.max_export_seconds": [],
    "zeek.smb_files.times.modified": [],
    "kafka.block_timestamp": [],
    "misp.report.published": []
  },

Calling the ElasticSearch api those fields do not exist on the doc. I have attempted to delete the Index Pattern and have Filebeat recreate, but that did not fix the issue. Where did this come from and how can I fix the issue?

2

This is an known issue that can be tracked with https://github.com/elastic/kibana/issues/69545

3

@Nathan_Reese that does appear to be the issue I am running into. Is there any known workarounds or do I have to wait for the next release?

4

The same question here: is there a workaround to hide the fields that are not used/empty?

Thanks in advance.

Regards,
Alejandro

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

6

@Ronin and @aguida79 if you had subscribed to this issue or checked again a day later you would have discovered the follow up in https://github.com/elastic/kibana/issues/69545#issuecomment-648061949.

We generally are very detailed in these issues and add tags or links to related information.
Especially once you found which one is the problem you are facing it is easiest just to subscribe to it in GitHub and get updates on the progress automatically.

The regression here was at the Elasticsearch side and Kibana just displayed what Elasticsearch returned. This is fixed in 7.8.1 (https://github.com/elastic/elasticsearch/pull/58418) so the solution is not far out.