Winlogbeat authorization error on AWS ELK

Elastic Stack Beats
1

Hi there,

I have an AWS Elasticsearch stack - version 7.10 and I would like to ship our fleet of EC2's Windows event logs to it.

I've followed the documentation and used an earlier version of Winlogbeat - version 7.12.1 however I can't seem to get it connected to Kibana to setup the dashboard. The following is the error I get: Exiting: 1 error: error loading index pattern: returned 401 to import file: . Response: {"statusCode":401,"error":"Unauthorized","message":"Authentication required"}

I would appreciate if anybody can help.

Thanks and regards,
Juan

2

Welcome to our community! :smiley:

We may not be able to help with this as the aws service telling you there is an access control problem, you will be have to chat to them about that.

But from a Beats point of view, you will need to make sure you have defined the authentication details in the config.

3

Thanks for the reply.

Yes, the authentication details are in the config and the credentials are correct because it can connect to ES.

Unfortunately AWS is also saying it is outside of their remit.

4

Can you post the logs of Winlogbeat please?

5

Hi Mark,

This is the latest log from Winlogbeat.

2021-08-17T23:52:47.628Z	INFO	instance/beat.go:660	Home path: [C:\Program Files\Winlogbeat] Config path: [C:\Program Files\Winlogbeat] Data path: [C:\Program Files\Winlogbeat\data] Logs path: [C:\Program Files\Winlogbeat\logs]
2021-08-17T23:52:47.632Z	INFO	instance/beat.go:668	Beat ID: b7dead8d-7ba2-48ab-b7c1-3893289c3310
2021-08-17T23:52:47.725Z	INFO	[beat]	instance/beat.go:996	Beat info	{"system_info": {"beat": {"path": {"config": "C:\\Program Files\\Winlogbeat", "data": "C:\\Program Files\\Winlogbeat\\data", "home": "C:\\Program Files\\Winlogbeat", "logs": "C:\\Program Files\\Winlogbeat\\logs"}, "type": "winlogbeat", "uuid": "b7dead8d-7ba2-48ab-b7c1-3893289c3310"}}}
2021-08-17T23:52:47.727Z	INFO	[beat]	instance/beat.go:1005	Build info	{"system_info": {"build": {"commit": "651a2ad1225f3d4420a22eba847de385b71f711d", "libbeat": "7.12.1", "time": "2021-04-20T21:18:27.000Z", "version": "7.12.1"}}}
2021-08-17T23:52:47.727Z	INFO	[beat]	instance/beat.go:1008	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":2,"version":"go1.15.9"}}}
2021-08-17T23:52:47.729Z	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:105	add_cloud_metadata: hosting provider type detected as aws, metadata={"account":{"id":"accountid"},"availability_zone":"us-east-1c","image":{"id":"ami-0f93c815788872c5d"},"instance":{"id":"i-0a461d23fba9d38f7"},"machine":{"type":"m4.large"},"provider":"aws","region":"us-east-1"}
2021-08-17T23:52:47.734Z	INFO	[beat]	instance/beat.go:1012	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-06-07T04:13:15.66Z","name":"EC2AMAZ-TJPFT03","ip":["fe80::dc43:ac65:a217:8bba/64","10.1.1.60/20","::1/128","127.0.0.1/8"],"kernel_version":"10.0.17763.1935 (WinBuild.160101.0800)","mac":["12:92:fe:dc:6d:df"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows Server 2019 Datacenter","version":"10.0","major":10,"minor":0,"patch":0,"build":"17763.1935"},"timezone":"GMT","timezone_offset_sec":0,"id":"d8914117-0efc-45b2-a11c-9592746c174e"}}}
2021-08-17T23:52:47.735Z	INFO	[beat]	instance/beat.go:1041	Process info	{"system_info": {"process": {"cwd": "C:\\Program Files\\Winlogbeat", "exe": "C:\\Program Files\\Winlogbeat\\winlogbeat.exe", "name": "winlogbeat.exe", "pid": 11092, "ppid": 12828, "start_time": "2021-08-17T23:52:47.365Z"}}}
2021-08-17T23:52:47.735Z	INFO	instance/beat.go:304	Setup Beat: winlogbeat; Version: 7.12.1
2021-08-17T23:52:47.735Z	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'winlogbeat-7.12.1' as ILM is enabled.
2021-08-17T23:52:47.735Z	INFO	eslegclient/connection.go:99	elasticsearch url: https://endpoint.us-east-1.es.amazonaws.com:443
2021-08-17T23:52:47.735Z	INFO	[publisher]	pipeline/module.go:113	Beat name: EC2AMAZ-TJPFT03
2021-08-17T23:52:47.736Z	INFO	beater/winlogbeat.go:69	State will be read from and persisted to C:\Program Files\Winlogbeat\data\.winlogbeat.yml
2021-08-17T23:52:47.785Z	WARN	[cfgwarn]	registered_domain/registered_domain.go:61	BETA: The registered_domain processor is beta.
2021-08-17T23:52:47.842Z	WARN	[cfgwarn]	registered_domain/registered_domain.go:61	BETA: The registered_domain processor is beta.
2021-08-17T23:52:47.854Z	INFO	kibana/client.go:119	Kibana url: https://endpoint.us-east-1.es.amazonaws.com:443/_plugin/kibana
2021-08-17T23:52:48.086Z	INFO	kibana/client.go:119	Kibana url: https://endpoint.us-east-1.es.amazonaws.com:443/_plugin/kibana
2021-08-17T23:52:48.132Z	ERROR	instance/beat.go:971	Exiting: 1 error: error loading index pattern: returned 401 to import file: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"Authentication required"}

Thanks and regards,
Juan

6

OK, what does your Winlogbeat config look like?

7

Here's the config. Thanks for looking.

###################### Winlogbeat Configuration Example ########################

# This file is an example configuration file highlighting only the most common
# options. The winlogbeat.reference.yml file from the same directory contains
# all the supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/winlogbeat/index.html

# ======================== Winlogbeat specific options =========================

# event_logs specifies a list of event logs to monitor as well as any
# accompanying options. The YAML data type of event_logs is a list of
# dictionaries.
#
# The supported keys are name (required), tags, fields, fields_under_root,
# forwarded, ignore_older, level, event_id, provider, and include_xml. Please
# visit the documentation for the complete details of each option.
# https://go.es.io/WinlogbeatConfig

winlogbeat.event_logs:
  - name: Application
    ignore_older: 72h

  - name: System

  - name: Security
    processors:
      - script:
          lang: javascript
          id: security
          file: ${path.home}/module/security/config/winlogbeat-security.js

  - name: Microsoft-Windows-Sysmon/Operational
    processors:
      - script:
          lang: javascript
          id: sysmon
          file: ${path.home}/module/sysmon/config/winlogbeat-sysmon.js

  - name: Windows PowerShell
    event_id: 400, 403, 600, 800
    processors:
      - script:
          lang: javascript
          id: powershell
          file: ${path.home}/module/powershell/config/winlogbeat-powershell.js

  - name: Microsoft-Windows-PowerShell/Operational
    event_id: 4103, 4104, 4105, 4106
    processors:
      - script:
          lang: javascript
          id: powershell
          file: ${path.home}/module/powershell/config/winlogbeat-powershell.js

  - name: ForwardedEvents
    tags: [forwarded]
    processors:
      - script:
          when.equals.winlog.channel: Security
          lang: javascript
          id: security
          file: ${path.home}/module/security/config/winlogbeat-security.js
      - script:
          when.equals.winlog.channel: Microsoft-Windows-Sysmon/Operational
          lang: javascript
          id: sysmon
          file: ${path.home}/module/sysmon/config/winlogbeat-sysmon.js
      - script:
          when.equals.winlog.channel: Windows PowerShell
          lang: javascript
          id: powershell
          file: ${path.home}/module/powershell/config/winlogbeat-powershell.js
      - script:
          when.equals.winlog.channel: Microsoft-Windows-PowerShell/Operational
          lang: javascript
          id: powershell
          file: ${path.home}/module/powershell/config/winlogbeat-powershell.js

# ====================== Elasticsearch template settings =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false


# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "https://endpoint.us-east-1.es.amazonaws.com:443/_plugin/kibana"
  username: "userid"
  password: "password"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Winlogbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["https://endpoint.us-east-1.es.amazonaws.com:443"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "userid"
  password: "password"

# ------------------------------ Logstash Output -------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Winlogbeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Winlogbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:
setup.ilm.enabled: false
# ============================== Instrumentation ===============================

# Instrumentation support for the winlogbeat.
#instrumentation:
    # Set to true to enable instrumentation of winlogbeat.
    #enabled: false

    # Environment in which winlogbeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    #  - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:


# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true
8

Anything wrong with the config?

9

It looks ok, does the username and password you used work with a curl against your aws instance?

10

Using curl, I've got output for Elasticsearch but not for Kibana. Please see the attached screenshot.

2021-08-20 12_24_41-mRemoteNG - MremoteNG.xml - Dev
2021-08-20 12_24_41-mRemoteNG - MremoteNG.xml - Dev959×357 78.5 KB

11

Please don't post pictures of text or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them :slight_smile:

If those details are working with curl then they should definitely be working with the aws service endpoint.

12

Thanks. Noted about screenshots.

Yeah the credentials are fine but for some reason Kibana is not liking it when setting up dashboards. Looking for alternative solutions now since this one won't work.

13

Oh it's a dashboard thing, then yeah that won't work because the aws service doesn't have the functionality it needs.

14

I see. That's interesting and makes sense. Then the instructions I found online for AWS ELK with dashboards setup are misleading then. Thanks for that Mark.

Does that mean dashboards for it will have to be created manually?

15

Yes, sorry to say.

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.