Winlogbeat is not Connecting to Logstash Which is not connecting to Elasticsearch

I have a small lab that consists of 2 ES nodes with basic security enabled using a TLS certificate, Kibana, Logstash, and a Windows 10 machine. The thing is that i'm not able to connect winbeat to logstash.

Logstash Error from logstash-plain.log

[2023-03-28T04:25:54,416][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://192.168.120.23:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://192.168.120.23:9200/'"}
[2023-03-28T04:25:58,003][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>33, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}, {"thread_id"=>34, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:95:in `sleep'"}, {"thread_id"=>35, "name"=>"[main]>worker2", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:95:in `sleep'"}, {"thread_id"=>36, "name"=>"[main]>worker3", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2023-03-28T04:25:59,421][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://192.168.120.23:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://192.168.120.23:9200/'"}


Winlogbeat when running .\winlogbeat setup

Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at http://192.168.120.23:5044: Get "http://192.168.120.23:5044": dial tcp 192.168.120.23:5044: connectex: No connection could be made because the target machine actively refused it.]

logstash.yml

# X-Pack Monitoring
# https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html
#xpack.monitoring.enabled: false
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: *********
#xpack.monitoring.elasticsearch.proxy: ["http://proxy:port"]
xpack.monitoring.elasticsearch.hosts: ["http://192.168.120.23:9200", "http://192.168.120.24:9200"]
# an alternative to hosts + username/password settings is to use cloud_id/cloud_auth
#xpack.monitoring.elasticsearch.cloud_id: monitoring_cluster_id:xxxxxxxxxx
#xpack.monitoring.elasticsearch.cloud_auth: logstash_system:password
# another authentication alternative is to use an Elasticsearch API key
#xpack.monitoring.elasticsearch.api_key: "id:api_key"
#xpack.monitoring.elasticsearch.ssl.certificate_authority: [ "/path/to/ca.crt" ]
#xpack.monitoring.elasticsearch.ssl.truststore.path: path/to/file
#xpack.monitoring.elasticsearch.ssl.truststore.password: password
#xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file
#xpack.monitoring.elasticsearch.ssl.keystore.password: password
#xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
#xpack.monitoring.elasticsearch.sniffing: false
#xpack.monitoring.collection.interval: 10s
#xpack.monitoring.collection.pipeline.details.enabled: true
input {
  beats {
    port => 5044
  }
}


output {
  elasticsearch {
    hosts => ["http://192.168.120.23:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
     user => "elastic"
     password => "**********"
  }
}


Elasticsearch.yml


 ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
node.roles: [master, data]




#action.destructive_requires_name: true
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
#

Running ES, kibana &winlogbeat v8.6 and logstash v7.17

Your Logstash error is an authentication error, the code 401 means that the request was unauthorized, check if the username and the password is correct.

What is your winglobeat.yml ? You didn't share it.

This logs means that Winlogbeat tried to connect to Elasticsearch on port 5044, but this port is not used by Elasticsearch, which listen on port 9200 per default. Something is wrong in your configuration.

Here's my beat.yml configs

winlogbeat.event_logs:
  - name: Application
    #ignore_older: 72h

  - name: System

  - name: Security
  - name: Microsoft-Windows-Windows Firewall With Advanced Security/Firewall

  #- name: Microsoft-Windows-Sysmon/Operational

  #- name: Windows PowerShell
    #event_id: 400, 403, 600, 800

  #- name: Microsoft-Windows-PowerShell/Operational
    #event_id: 4103, 4104, 4105, 4106

  #- name: ForwardedEvents
   # tags: [forwarded]

# ====================== Elasticsearch template settings =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false


# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
setup.dashboards.enabled: true

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "192.168.120.23:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Winlogbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  Array of hosts to connect to.
  hosts: ["192.168.120.23:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "zg9tpovAiTYOPiOgM4XP"

  # Pipeline to route events to security, sysmon, or powershell pipelines.
  pipeline: "winlogbeat-%{[agent.version]}-routing"

# ------------------------------ Logstash Output -------------------------------
output.logstash:
   The Logstash hosts
  hosts: ["192.168.120.23:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~

i'm not sure why it's trying to ES on port 5044

The beats only support one output, you can't have both output.elasticsearch and output.logstash uncommented.

To resolve the issue:
Make sure that Logstash is running and you can connect to it. ...
Verify that the config file for Winlogbeat specifies the correct port where Logstash is running.
Make sure that the Elasticsearch output is commented out in the config file and the Logstash output is uncommented.

Regards,
Rachel Gomez

Now with that logstash was able to start and connect to ES as i can see from the logs, beat still giving me hard time processing the configuration file.

Logstash logs:

[2023-03-29T01:11:42,892][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2023-03-29T01:11:42,908][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.17.0", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2023-03-29T01:11:42,919][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[2023-03-29T01:11:44,544][WARN ][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml
[2023-03-29T01:11:45,915][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Got response code '401' contacting Elasticsearch at URL 'http://192.168.120.23:9200/_xpack'"}
[2023-03-29T01:11:45,942][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
[2023-03-29T01:11:46,209][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-03-29T01:11:47,350][INFO ][org.reflections.Reflections] Reflections took 67 ms to scan 1 urls, producing 119 keys and 417 values
[2023-03-29T01:11:48,335][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://192.168.120.23:9200", "http://192.168.120.24:9200"]}
[2023-03-29T01:11:48,362][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@192.168.120.23:9200/, http://elastic:xxxxxx@192.168.120.24:9200/]}}
[2023-03-29T01:11:48,645][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@192.168.120.23:9200/"}
[2023-03-29T01:11:48,668][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.6.2) {:es_version=>8}
[2023-03-29T01:11:48,682][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2023-03-29T01:11:49,728][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@192.168.120.24:9200/"}
[2023-03-29T01:11:49,813][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2023-03-29T01:11:49,822][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2023-03-29T01:11:49,890][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:disabled}
[2023-03-29T01:11:49,942][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/beats.conf"], :thread=>"#<Thread:0xfbc2d4c run>"}
[2023-03-29T01:11:51,410][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.47}
[2023-03-29T01:11:51,442][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2023-03-29T01:11:51,499][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-03-29T01:11:51,578][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2023-03-29T01:11:51,948][INFO ][org.logstash.beats.Server][main][e026f35db2f06468275417bcb3cbd40622c94f720c9088491c89e8c2ded9d28f] Starting server on port: 5044
[2023-03-29T01:12:16,031][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2023-03-29T01:12:16,031][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2023-03-29T01:12:16,956][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["http://192.168.120.23:9200", "http://192.168.120.24:9200"]}
[2023-03-29T01:12:16,963][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://logstash_system:xxxxxx@192.168.120.23:9200/, http://logstash_system:xxxxxx@192.168.120.24:9200/]}}
[2023-03-29T01:12:16,975][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"http://logstash_system:xxxxxx@192.168.120.23:9200/"}
[2023-03-29T01:12:16,979][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch version determined (8.6.2) {:es_version=>8}
[2023-03-29T01:12:16,980][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2023-03-29T01:12:16,997][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"http://logstash_system:xxxxxx@192.168.120.24:9200/"}
[2023-03-29T01:12:17,012][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2023-03-29T01:12:17,013][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2023-03-29T01:12:17,014][WARN ][logstash.javapipeline    ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[2023-03-29T01:12:17,017][INFO ][logstash.javapipeline    ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x77dd88d0 run>"}
[2023-03-29T01:12:17,026][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>0.01}
[2023-03-29T01:12:17,036][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2023-03-29T01:12:17,044][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:main, :".monitoring-logstash"], :non_runnin

Winlogbeat error:
Exiting: error loading config file: yaml: line 130: mapping values are not allowed in this context

Line 130
hosts: ["192.168.120.23:5044"]
128 output.logstash:
  129 The Logstash hosts
  130 hosts: ["192.168.120.23:5044"]

figured it out, needed to comment line #129

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.