Fielddata is disabled on [host.hostname] in [filebeat-8.10.4-2023.10.17]

[whanklee]

Hi there,

I got the following error code:

Index: filebeat-8.10.4-2023.10.17
Type: illegal_argument_exception
Node: rQ6bZR_YQ3Ken74LUR-Oaw
Reason: Fielddata is disabled on [host.hostname] in [filebeat-8.10.4-2023.10.17]. Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [host.hostname] in order to load field data by uninverting the inverted index. Note that this can use significant memory.

Request:

{
  "aggs": {
    "2": {
      "terms": {
        "field": "user_agent.name",
        "order": {
          "1": "desc"
        },
        "size": 5,
        "shard_size": 25
      },
      "aggs": {
        "1": {
          "cardinality": {
            "field": "source.address"
          }
        },
        "3": {
          "terms": {
            "field": "user_agent.version",
            "order": {
              "1": "desc"
            },
            "size": 5,
            "shard_size": 25
          },
          "aggs": {
            "1": {
              "cardinality": {
                "field": "source.address"
              }
            }
          }
        }
      }
    }
  },
  "size": 0,
  "fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    {
      "field": "aws.cloudtrail.digest.end_time",
      "format": "date_time"
    },
    {
      "field": "aws.cloudtrail.digest.newest_event_time",
      "format": "date_time"
    },
    {
      "field": "aws.cloudtrail.digest.oldest_event_time",
      "format": "date_time"
    },
    {
      "field": "aws.cloudtrail.digest.start_time",
      "format": "date_time"
    },
    {
      "field": "aws.cloudtrail.user_identity.session_context.creation_date",
      "format": "date_time"
    },
    {
      "field": "azure.auditlogs.properties.activity_datetime",
      "format": "date_time"
    },
    {
      "field": "azure.enqueued_time",
      "format": "date_time"
    },
    {
      "field": "azure.signinlogs.properties.created_at",
      "format": "date_time"
    },
    {
      "field": "cef.extensions.agentReceiptTime",
      "format": "date_time"
    },
    {
      "field": "cef.extensions.deviceCustomDate1",
      "format": "date_time"
    },
    {
      "field": "cef.extensions.deviceCustomDate2",
      "format": "date_time"
    },
    {
      "field": "cef.extensions.deviceReceiptTime",
      "format": "date_time"
    },
    {
      "field": "cef.extensions.endTime",
      "format": "date_time"
    },
    {
      "field": "cef.extensions.fileCreateTime",
      "format": "date_time"
    },
    {
      "field": "cef.extensions.fileModificationTime",
      "format": "date_time"
    },
    {
      "field": "cef.extensions.flexDate1",
      "format": "date_time"
    },
    {
      "field": "cef.extensions.managerReceiptTime",
      "format": "date_time"
    },
    {
      "field": "cef.extensions.oldFileCreateTime",
      "format": "date_time"
    },
    {
      "field": "cef.extensions.oldFileModificationTime",
      "format": "date_time"
    },
    {
      "field": "cef.extensions.startTime",
      "format": "date_time"
    },
    {
      "field": "checkpoint.subs_exp",
      "format": "date_time"
    },
    {
      "field": "cisco.amp.threat_hunting.incident_end_time",
      "format": "date_time"
    },
    {
      "field": "cisco.amp.threat_hunting.incident_start_time",
      "format": "date_time"
    },
    {
      "field": "cisco.amp.timestamp_nanoseconds",
      "format": "date_time"
    },
    {
      "field": "code_signature.timestamp",
      "format": "date_time"
    },
    {
      "field": "crowdstrike.event.EndTimestamp",
      "format": "date_time"
    },
    {
      "field": "crowdstrike.event.IncidentEndTime",
      "format": "date_time"
    },
    {
      "field": "crowdstrike.event.IncidentStartTime",
      "format": "date_time"
    },
    {
      "field": "crowdstrike.event.ProcessEndTime",
      "format": "date_time"
    },
    {
      "field": "crowdstrike.event.ProcessStartTime",
      "format": "date_time"
    },
    {
      "field": "crowdstrike.event.StartTimestamp",
      "format": "date_time"
    },
    {
      "field": "crowdstrike.event.Timestamp",
      "format": "date_time"
    },
    {
      "field": "crowdstrike.event.UTCTimestamp",
      "format": "date_time"
    },
    {
      "field": "crowdstrike.metadata.eventCreationTime",
      "format": "date_time"
    },
    {
      "field": "cyberarkpas.audit.iso_timestamp",
      "format": "date_time"
    },
    {
      "field": "dll.code_signature.timestamp",
      "format": "date_time"
    },
    {
      "field": "elf.creation_date",
      "format": "date_time"
    },
    {
      "field": "event.created",
      "format": "date_time"
    },
    {
      "field": "event.end",
      "format": "date_time"
    },
    {
      "field": "event.ingested",
      "format": "date_time"
    },
    {
      "field": "event.start",
      "format": "date_time"
    },
    {
      "field": "file.accessed",
      "format": "date_time"
    },
    {
      "field": "file.code_signature.timestamp",
      "format": "date_time"
    },
    {
      "field": "file.created",
      "format": "date_time"
    },
    {
      "field": "file.ctime",
      "format": "date_time"
    },
    {
      "field": "file.elf.creation_date",
      "format": "date_time"
    },
    {
      "field": "file.mtime",
      "format": "date_time"
    },
    {
      "field": "file.x509.not_after",
      "format": "date_time"
    },
    {
      "field": "file.x509.not_before",
      "format": "date_time"
    },
    {
      "field": "google_workspace.admin.email.log_search_filter.end_date",
      "format": "date_time"
    },
    {
      "field": "google_workspace.admin.email.log_search_filter.start_date",
      "format": "date_time"
    },
    {
      "field": "google_workspace.admin.user.birthdate",
      "format": "date_time"
    },
    {
      "field": "juniper.srx.elapsed_time",
      "format": "date_time"
    },
    {
      "field": "juniper.srx.epoch_time",
      "format": "date_time"
    },
    {
      "field": "juniper.srx.timestamp",
      "format": "date_time"
    },
    {
      "field": "kafka.block_timestamp",
      "format": "date_time"
    },
    {
      "field": "microsoft.defender_atp.lastUpdateTime",
      "format": "date_time"
    },
    {
      "field": "microsoft.defender_atp.resolvedTime",
      "format": "date_time"
    },
    {
      "field": "microsoft.m365_defender.alerts.creationTime",
      "format": "date_time"
    },
    {
      "field": "microsoft.m365_defender.alerts.lastUpdatedTime",
      "format": "date_time"
    },
    {
      "field": "microsoft.m365_defender.alerts.resolvedTime",
      "format": "date_time"
    },
    {
      "field": "misp.campaign.first_seen",
      "format": "date_time"
    },
    {
      "field": "misp.campaign.last_seen",
      "format": "date_time"
    },
    {
      "field": "misp.intrusion_set.first_seen",
      "format": "date_time"
    },
    {
      "field": "misp.intrusion_set.last_seen",
      "format": "date_time"
    },
    {
      "field": "misp.observed_data.first_observed",
      "format": "date_time"
    },
    {
      "field": "misp.observed_data.last_observed",
      "format": "date_time"
    },
    {
      "field": "misp.report.published",
      "format": "date_time"
    },
    {
      "field": "misp.threat_indicator.valid_from",
      "format": "date_time"
    },
    {
      "field": "misp.threat_indicator.valid_until",
      "format": "date_time"
    },
    {
      "field": "netflow.collection_time_milliseconds",
      "format": "date_time"
    },
    {
      "field": "netflow.exporter.timestamp",
      "format": "date_time"
    },
    {
      "field": "netflow.flow_end_microseconds",
      "format": "date_time"
    },
    {
      "field": "netflow.flow_end_milliseconds",
      "format": "date_time"
    },
    {
      "field": "netflow.flow_end_nanoseconds",
      "format": "date_time"
    },
    {
      "field": "netflow.flow_end_seconds",
      "format": "date_time"
    },
    {
      "field": "netflow.flow_start_microseconds",
      "format": "date_time"
    },
    {
      "field": "netflow.flow_start_milliseconds",
      "format": "date_time"
    },
    {
      "field": "netflow.flow_start_nanoseconds",
      "format": "date_time"
    },
    {
      "field": "netflow.flow_start_seconds",
      "format": "date_time"
    },
    {
      "field": "netflow.max_export_seconds",
      "format": "date_time"
    },
    {
      "field": "netflow.max_flow_end_microseconds",
      "format": "date_time"
    },
    {
      "field": "netflow.max_flow_end_milliseconds",
      "format": "date_time"
    },
    {
      "field": "netflow.max_flow_end_nanoseconds",
      "format": "date_time"
    },
    {
      "field": "netflow.max_flow_end_seconds",
      "format": "date_time"
    },
    {
      "field": "netflow.min_export_seconds",
      "format": "date_time"
    },
    {
      "field": "netflow.min_flow_start_microseconds",
      "format": "date_time"
    },
    {
      "field": "netflow.min_flow_start_milliseconds",
      "format": "date_time"
    },
    {
      "field": "netflow.min_flow_start_nanoseconds",
      "format": "date_time"
    },
    {
      "field": "netflow.min_flow_start_seconds",
      "format": "date_time"
    },
    {
      "field": "netflow.monitoring_interval_end_milli_seconds",
      "format": "date_time"
    },
    {
      "field": "netflow.monitoring_interval_start_milli_seconds",
      "format": "date_time"
    },
    {
      "field": "netflow.observation_time_microseconds",
      "format": "date_time"
    },
    {
      "field": "netflow.observation_time_milliseconds",
      "format": "date_time"
    },
    {
      "field": "netflow.observation_time_nanoseconds",
      "format": "date_time"
    },
    {
      "field": "netflow.observation_time_seconds",
      "format": "date_time"
    },
    {
      "field": "netflow.system_init_time_milliseconds",
      "format": "date_time"
    },
    {
      "field": "okta.debug_context.debug_data.suspicious_activity.timestamp",
      "format": "date_time"
    },
    {
      "field": "package.installed",
      "format": "date_time"
    },
    {
      "field": "panw.panos.factorcompletiontime",
      "format": "date_time"
    },
    {
      "field": "pensando.dfw.timestamp",
      "format": "date_time"
    },
    {
      "field": "postgresql.log.session_start_time",
      "format": "date_time"
    },
    {
      "field": "process.code_signature.timestamp",
      "format": "date_time"
    },
    {
      "field": "process.elf.creation_date",
      "format": "date_time"
    },
    {
      "field": "process.end",
      "format": "date_time"
    },
    {
      "field": "process.parent.code_signature.timestamp",
      "format": "date_time"
    },
    {
      "field": "process.parent.elf.creation_date",
      "format": "date_time"
    },
    {
      "field": "process.parent.end",
      "format": "date_time"
    },
    {
      "field": "process.parent.start",
      "format": "date_time"
    },
    {
      "field": "process.start",
      "format": "date_time"
    },
    {
      "field": "rsa.internal.lc_ctime",
      "format": "date_time"
    },
    {
      "field": "rsa.internal.time",
      "format": "date_time"
    },
    {
      "field": "rsa.time.effective_time",
      "format": "date_time"
    },
    {
      "field": "rsa.time.endtime",
      "format": "date_time"
    },
    {
      "field": "rsa.time.event_queue_time",
      "format": "date_time"
    },
    {
      "field": "rsa.time.event_time",
      "format": "date_time"
    },
    {
      "field": "rsa.time.expire_time",
      "format": "date_time"
    },
    {
      "field": "rsa.time.recorded_time",
      "format": "date_time"
    },
    {
      "field": "rsa.time.stamp",
      "format": "date_time"
    },
    {
      "field": "rsa.time.starttime",
      "format": "date_time"
    },
    {
      "field": "snyk.vulnerabilities.disclosure_time",
      "format": "date_time"
    },
    {
      "field": "snyk.vulnerabilities.introduced_date",
      "format": "date_time"
    },
    {
      "field": "snyk.vulnerabilities.publication_time",
      "format": "date_time"
    },
    {
      "field": "sophos.xg.date",
      "format": "date_time"
    },
    {
      "field": "sophos.xg.eventtime",
      "format": "date_time"
    },
    {
      "field": "sophos.xg.start_time",
      "format": "date_time"
    },
    {
      "field": "sophos.xg.starttime",
      "format": "date_time"
    },
    {
      "field": "sophos.xg.timestamp",
      "format": "date_time"
    },
    {
      "field": "suricata.eve.alert.created_at",
      "format": "date_time"
    },
    {
      "field": "suricata.eve.alert.updated_at",
      "format": "date_time"
    },
    {
      "field": "suricata.eve.tls.notafter",
      "format": "date_time"
    },
    {
      "field": "suricata.eve.tls.notbefore",
      "format": "date_time"
    },
    {
      "field": "threat.enrichments.indicator.file.accessed",
      "format": "date_time"
    },
    {
      "field": "threat.enrichments.indicator.file.code_signature.timestamp",
      "format": "date_time"
    },
    {
      "field": "threat.enrichments.indicator.file.created",
      "format": "date_time"
    },
    {
      "field": "threat.enrichments.indicator.file.ctime",
      "format": "date_time"
    },
    {
      "field": "threat.enrichments.indicator.file.elf.creation_date",
      "format": "date_time"
    },
    {
      "field": "threat.enrichments.indicator.file.mtime",
      "format": "date_time"
    },
    {
      "field": "threat.enrichments.indicator.file.x509.not_after",
      "format": "date_time"
    },
    {
      "field": "threat.enrichments.indicator.file.x509.not_before",
      "format": "date_time"
    },
    {
      "field": "threat.enrichments.indicator.first_seen",
      "format": "date_time"
    },
    {
      "field": "threat.enrichments.indicator.last_seen",
      "format": "date_time"
    },
    {
      "field": "threat.enrichments.indicator.modified_at",
      "format": "date_time"
    },
    {
      "field": "threat.enrichments.indicator.x509.not_after",
      "format": "date_time"
    },
    {
      "field": "threat.enrichments.indicator.x509.not_before",
      "format": "date_time"
    },
    {
      "field": "threat.indicator.file.accessed",
      "format": "date_time"
    },
    {
      "field": "threat.indicator.file.code_signature.timestamp",
      "format": "date_time"
    },
    {
      "field": "threat.indicator.file.created",
      "format": "date_time"
    },
    {
      "field": "threat.indicator.file.ctime",
      "format": "date_time"
    },
    {
      "field": "threat.indicator.file.elf.creation_date",
      "format": "date_time"
    },
    {
      "field": "threat.indicator.file.mtime",
      "format": "date_time"
    },
    {
      "field": "threat.indicator.file.x509.not_after",
      "format": "date_time"
    },
    {
      "field": "threat.indicator.file.x509.not_before",
      "format": "date_time"
    },
    {
      "field": "threat.indicator.first_seen",
      "format": "date_time"
    },
    {
      "field": "threat.indicator.last_seen",
      "format": "date_time"
    },
    {
      "field": "threat.indicator.modified_at",
      "format": "date_time"
    },
    {
      "field": "threat.indicator.x509.not_after",
      "format": "date_time"
    },
    {
      "field": "threat.indicator.x509.not_before",
      "format": "date_time"
    },
    {
      "field": "tls.client.not_after",
      "format": "date_time"
    },
    {
      "field": "tls.client.not_before",
      "format": "date_time"
    },
    {
      "field": "tls.client.x509.not_after",
      "format": "date_time"
    },
    {
      "field": "tls.client.x509.not_before",
      "format": "date_time"
    },
    {
      "field": "tls.server.not_after",
      "format": "date_time"
    },
    {
      "field": "tls.server.not_before",
      "format": "date_time"
    },
    {
      "field": "tls.server.x509.not_after",
      "format": "date_time"
    },
    {
      "field": "tls.server.x509.not_before",
      "format": "date_time"
    },
    {
      "field": "x509.not_after",
      "format": "date_time"
    },
    {
      "field": "x509.not_before",
      "format": "date_time"
    },
    {
      "field": "zeek.kerberos.valid.from",
      "format": "date_time"
    },
    {
      "field": "zeek.kerberos.valid.until",
      "format": "date_time"
    },
    {
      "field": "zeek.ntp.org_time",
      "format": "date_time"
    },
    {
      "field": "zeek.ntp.rec_time",
      "format": "date_time"
    },
    {
      "field": "zeek.ntp.ref_time",
      "format": "date_time"
    },
    {
      "field": "zeek.ntp.xmt_time",
      "format": "date_time"
    },
    {
      "field": "zeek.ocsp.revoke.time",
      "format": "date_time"
    },
    {
      "field": "zeek.ocsp.update.next",
      "format": "date_time"
    },
    {
      "field": "zeek.ocsp.update.this",
      "format": "date_time"
    },
    {
      "field": "zeek.pe.compile_time",
      "format": "date_time"
    },
    {
      "field": "zeek.smb_files.times.accessed",
      "format": "date_time"
    },
    {
      "field": "zeek.smb_files.times.changed",
      "format": "date_time"
    },
    {
      "field": "zeek.smb_files.times.created",
      "format": "date_time"
    },
    {
      "field": "zeek.smb_files.times.modified",
      "format": "date_time"
    },
    {
      "field": "zeek.smtp.date",
      "format": "date_time"
    },
    {
      "field": "zeek.snmp.up_since",
      "format": "date_time"
    },
    {
      "field": "zeek.x509.certificate.valid.from",
      "format": "date_time"
    },
    {
      "field": "zeek.x509.certificate.valid.until",
      "format": "date_time"
    },
    {
      "field": "zoom.meeting.start_time",
      "format": "date_time"
    },
    {
      "field": "zoom.participant.join_time",
      "format": "date_time"
    },
    {
      "field": "zoom.participant.leave_time",
      "format": "date_time"
    },
    {
      "field": "zoom.phone.answer_start_time",
      "format": "date_time"
    },
    {
      "field": "zoom.phone.call_end_time",
      "format": "date_time"
    },
    {
      "field": "zoom.phone.connected_start_time",
      "format": "date_time"
    },
    {
      "field": "zoom.phone.date_time",
      "format": "date_time"
    },
    {
      "field": "zoom.phone.ringing_start_time",
      "format": "date_time"
    },
    {
      "field": "zoom.recording.recording_file.recording_end",
      "format": "date_time"
    },
    {
      "field": "zoom.recording.recording_file.recording_start",
      "format": "date_time"
    },
    {
      "field": "zoom.recording.start_time",
      "format": "date_time"
    },
    {
      "field": "zoom.timestamp",
      "format": "date_time"
    },
    {
      "field": "zoom.webinar.start_time",
      "format": "date_time"
    }
  ],
  "script_fields": {},
  "stored_fields": [
    "*"
  ],
  "runtime_mappings": {},
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "bool": {
            "should": [
              {
                "match": {
                  "event.dataset": "apache.access"
                }
              }
            ],
            "minimum_should_match": 1
          }
        },
        {
          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2023-10-17T20:11:10.335Z",
              "lte": "2023-10-17T20:26:10.335Z"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

Response:

{
  "took": 4,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 1,
    "skipped": 1,
    "failed": 1,
    "failures": [
      {
        "shard": 0,
        "index": "filebeat-8.10.4-2023.10.17",
        "node": "rQ6bZR_YQ3Ken74LUR-Oaw",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "Fielddata is disabled on [host.hostname] in [filebeat-8.10.4-2023.10.17]. Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [host.hostname] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
        }
      }
    ]
  },
  "hits": {
    "total": 0,
    "max_score": 0,
    "hits": []
  }
}

I would like to use apache.yml, mysql.yml and system.yml. Of course, these ones are enabled.

[stephenb (Stephen Brown)]

Hi @whanklee Welcome to the community!

What you are seeing is that the mapping (schema) is incorrect for that index and field. This is very common issue when getting started with the Elastic Stack.

Why is the mapping incorrect ... because the correct template is not being used.

Why is the correct template not being used?

Because most likely the setup command for filebeat was not run, which loads the correct template which then will create the correct mappings which then leads to the correct data types in the index which then allows the proper search and aggregations / filters / dashboards etc.

Filebeat quick start: installation and configuration | Filebeat Reference...

Make sure you look at the Enable Modules Section

And Equally important the Setup Command

This is key as this loads all the assets (templates pipeline dashboards etc ) for the enabled modules

Do clean up / delete your index follow the commands and give it another try.

Then your searches / aggregations should work...

Note you can check the current mapping with

GET filebeat-8.10.4-2023.10.17

and you will see the fields with the default multifields of both text and keyword which is what is provided when a template is not provided.

You can not aggregate on a text field, you will get that error.

keyword is for filtering and aggregating / exact match
text is for full-text search

I would suggest perhaps take a look at them both in the field types

Keyword type family | Elasticsearch Guide [8.10] | Elastic

Text type family | Elasticsearch Guide [8.10] | Elastic

[whanklee]

I used that installation guide to setup Elastic Stack. How To Install Elasticsearch, Logstash, and Kibana (Elastic Stack) on Ubuntu 22.04 | DigitalOcean

I ran 'sudo filebeat setup -e' and output:

{"log.level":"info","@timestamp":"2023-10-18T07:48:32.107+0200","log.origin":{"file.name":"instance/beat.go","file.line":783},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-18T07:48:32.118+0200","log.origin":{"file.name":"instance/beat.go","file.line":791},"message":"Beat ID: 202c3af2-5cf1-4a58-b50a-57732da92f2a","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-18T07:48:32.137+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1303},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/etc/filebeat","data":"/var/lib/filebeat","home":"/usr/share/filebeat","logs":"/var/log/filebeat"},"type":"filebeat","uuid":"202c3af2-5cf1-4a58-b50a-57732da92f2a"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-10-18T07:48:32.143+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1312},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"10b198c985eb95c16405b979c63847881a199aba","libbeat":"8.10.4","time":"2023-10-11T19:23:15.000Z","version":"8.10.4"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-10-18T07:48:32.143+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1315},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":4,"version":"go1.20.8"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-10-18T07:48:32.145+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1321},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-10-17T21:32:30+02:00","containerized":false,"name":"ultron.keves.net","ip":["127.0.0.1","38.242.147.9"],"kernel_version":"5.15.0-86-generic","mac":["00:50:56:46:f5:d2"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"22.04.3 LTS (Jammy Jellyfish)","major":22,"minor":4,"patch":3,"codename":"jammy"},"timezone":"CEST","timezone_offset_sec":7200,"id":"52948f666ef335a02661c7d762cde228"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-10-18T07:48:32.149+0200","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1350},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","perfmon","bpf","checkpoint_restore"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","perfmon","bpf","checkpoint_restore"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","perfmon","bpf","checkpoint_restore"],"ambient":null},"cwd":"/usr/share/filebeat/bin","exe":"/usr/share/filebeat/bin/filebeat","name":"filebeat","pid":79937,"ppid":79936,"seccomp":{"mode":"disabled","no_new_privs":false},"start_time":"2023-10-18T07:48:30.930+0200"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-10-18T07:48:32.149+0200","log.origin":{"file.name":"instance/beat.go","file.line":329},"message":"Setup Beat: filebeat; Version: 8.10.4","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-18T07:48:32.163+0200","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":105},"message":"Beat name: ultron.keves.net","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-18T07:48:32.172+0200","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":135},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-10-18T07:48:32.172+0200","log.origin":{"file.name":"beater/filebeat.go","file.line":193},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-10-18T07:48:32.180+0200","log.origin":{"file.name":"instance/beat.go","file.line":1278},"message":"Exiting: index management requested but the Elasticsearch output is not configured/enabled","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: index management requested but the Elasticsearch output is not configured/enabled

filebeat.yml:

filebeat.inputs:

- type: filestream

  id: my-filestream-id

  enabled: false

  paths:
    - /var/log/*.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

setup.kibana:

output.logstash:
  hosts: ["localhost:5044"]

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~```

[stephenb (Stephen Brown)]

Ahh so you put logstash in the middle... Perfectly fine architecture That requires additional work and configuration and steps.

And following another guide.. not highly recommended.

I / we don't debug other guides.. but a quick look tells me that died will not work because you are using 8.X The guide is for 7.X and there are some significant changes between the versions..

What I would suggest is get it working directly from Filebeat to Elasticsearch directly first.

Your setup is failing because you have filebeat pointed at lo gstash not elasticsearch which is where it needs to point to to load the templates that will create correct. Mappings.

That is why I suggested you to follow the quick start first.

Then after you get that working with logstash in the middle.

You will also need a proper logstash configuration...to pass through the beats data.

See this page fur the proper logstash

Use ingest pipelines for parsing | Logstash Reference [8.10] | Elastic

Get it working directly first then come back and we can help with logstash... Is there a specific reason you want to use logstash?

[whanklee]

I'm sorry, the description I linked is actually about installation version 7, but I changed it to version 8 before installing it. I was using this repo: echo "deb [signed-by=/usr/share/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list
So, during the installation, I made sure that all components were 8.

I do try that you show up now. I will be back and share experiences.

[whanklee]

It works well. I skipped Logstash, filebeat connects to Elasticsearch directly and there is no error in Dashboard.

I used the following commands to install:
sudo filebeat setup --pipelines --modules system apache mysql

(then edit these files in /etc/filebeat/modules.d/{apache, mysql,system}.yml)

For example:

# Module: apache
# Docs: https://www.elastic.co/guide/en/beats/filebeat/8.10/filebeat-module-apache.html

- module: apache
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/apache2/*access.log*"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/apache2/*error.log*"]

After that
sudo filebeat setup --pipelines -e && sudo systemctl start filebeat && sudo systemctl enable filebeat