Can't create winlogbeat-*

Elastic Stack Beats
1

Hi,

I have succesfully installed winlogbeat on a Windows Server 2016. Yesterday I was able to create the index for the winlogbeat data. Today I deleted everything and I wanted to start again also with a new index name.

Unfortunately it doesn't work anymore. Yesterday I also don't know exactly why it was working. I send the data thru Logstash so Logstash is as output defined.

This is my Logstash Input/Output File:

input {
#  file {
#    path => "/tmp/in.log"
#    type => "logfile"  # a type to identify those logs (will need this later)
#    start_position => "beginning"
#   }
syslog {
    port => 5514
    type => "syslog"
}
udp {
    type => "pfsense"
    port => 5140
}
beats {
    type => "log"
    port => 5044
}
udp {
    type => "syslog"
    port => 5515

}
}

output {
#  file {
#    path => "/tmp/out.log"
#  }
 if [@metadata][beats] {
  elasticsearch {
  hosts => "localhost:9200"
  index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  document_type => "%{[@metadata][type]}"
 }
  } else {
  elasticsearch {
     hosts => "localhost:9200"
     manage_template => false
     index => "logstash-%{+YYYY.MM.dd}"
   }
 }
#  if [@metadata][cthostmeta] == "ELKSTACK" {
#    file {
#       codec => rubydebug { metadata => true}
#       path => "/tmp/logstash_out.log"
#    }
#   }
}

I'm getting this error when I want to do the Invoke-Webrequest. I think this is one of the problems....

> Invoke-WebRequest : Die Verbindung mit dem Remoteserver kann nicht hergestellt werden.
> In Zeile:1 Zeichen:1
> + Invoke-WebRequest -Method Put -InFile winlogbeat.template.json -Uri   ...
> + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>     + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], WebException
>     + FullyQualifiedErrorId : System.Net.WebException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

Can someone give some hints? I have already an index called: Logstash-* where also data is comming in succesfully.

Thanks

2

Setting type on the input does nothing. Beats input plugin | Logstash Reference [8.11] | Elastic

Should that say "beat" instead of "beats"?

Sounds like a connection issue. Are you sure Elasticsearch is up and running at that address?

3

Hi Andrew,

I have now changed it to:

input {
#  file {
#    path => "/tmp/in.log"
#    type => "logfile"  # a type to identify those logs (will need this later)
#    start_position => "beginning"
#   }
syslog {
    port => 5514
    type => "syslog"
}
udp {
    type => "pfsense"
    port => 5140
}
beats {
    port => 5044
}
udp {
    type => "syslog"
    port => 5515

}
}

output {
#  file {
#    path => "/tmp/out.log"
#  }
 if [@metadata][beat] {
  elasticsearch {
  hosts => "localhost:9200"
  index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  document_type => "%{[@metadata][type]}"
 }
  } else {
  elasticsearch {
     hosts => "localhost:9200"
     manage_template => false
     index => "logstash-%{+YYYY.MM.dd}"
   }
 }
#  if [@metadata][cthostmeta] == "ELKSTACK" {
#    file {
#       codec => rubydebug { metadata => true}
#       path => "/tmp/logstash_out.log"
#    }
#   }
}

Yes Elasticsearch is up and running.

{
  "name" : "tePVrUc",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "zUnx4gplQAaWUoYSBz2PLA",
  "version" : {
    "number" : "5.5.0",
    "build_hash" : "260387d",
    "build_date" : "2017-06-30T23:16:05.735Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.0"
  },
  "tagline" : "You Know, for Search"
}

And yes it says in english: The connection to the remoteserver can't be established. I don't know why. Firewall is offline from the Windows Server. And yes the invoke command should work because it's 2016. Do I need that Invoke command for creating the first index?

4

Can you describe you setup more. Is Logstash and Elasticsearch running on the same machine as Winlogbeat?

What's the Winlogbeat config being used?

What is the exact Invoke-WebRequest command you are running?

The command you are running installs the index template to Elasticsearch. The template must be in place before the index is created to ensure that the incoming data is mapped to the proper types.

5

I use this command:

PS C:\program files\winlogbeat> Invoke-WebRequest -Method Put -InFile winlogbeat.template.json -Uri http://192.168.0.13:9200/_template/winlogbeat?pretty

The ELK stack is running on: 192.168.0.13 CentOS 7 minimal installation. Winlogbeat is on different server (192.168.0.29).
This is the Winlogbeat configuration:

###################### Winlogbeat Configuration Example ##########################

# This file is an example configuration file highlighting only the most common
# options. The winlogbeat.full.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/winlogbeat/index.html

#======================= Winlogbeat specific options ==========================

# event_logs specifies a list of event logs to monitor as well as any
# accompanying options. The YAML data type of event_logs is a list of
# dictionaries.
#
# The supported keys are name (required), tags, fields, fields_under_root,
# forwarded, ignore_older, level, event_id, provider, and include_xml. Please
# visit the documentation for the complete details of each option.
# https://go.es.io/WinlogbeatConfig
winlogbeat.event_logs:
  - name: Application
    ignore_older: 72h
  - name: Security
  - name: System
  - name: Microsoft-Windows-Windows Firewall With Advanced Security/Firewall
  - name: Microsoft-Windows-TWinUI/Operational
  - name: Microsoft-Windows-Hyper-V-Config-Admin

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  # hosts: 192.168.0.13:9200

  # Optional protocol and basic auth credentials.
  #protocol: "http"
  #username: "admin"
  #password: "test"

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["192.168.0.13:5044"]
  template.name: "winlogbeat"
  template.path: "winlogbeat.template.json"
  template.overwrite: true

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]
6

Are you able to browse to http://192.168.0.13:9200/ from the Windows machine?

By default Elasticsearch listens on localhost. If you want it to accept connections from the network you need to modify the network.host setting. https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html

Since you are sending the Winlogbeat data to Logstash rather then ES. You might want to consider copying the winlogbeat.template.json over to the ES server and installing it from there. This avoids the need to expose ES to the network. If you copy the file over you could use curl to install it.

curl -XPUT http://localhost:9200/_template/winlogbeat -d@winlogbeat.template.json

7

I can't connect to the url: http://192.168.0.13:9200 so this is probably the problem. When I understand that correct, i need to copy from the Windows Server the file: winlogbeat.template to the ELK server and then I have to run that curl command?

EDIT:
I have uploaded the file winlogbeat.template.json into the conf.d folder. I'm with the CLI in this folder. Now i used:
curl -XPUT http://localhost:9200/_template/winlogbeat -d@winlogbeat.template.json

But nothing happens...?

8

Try it with the ?pretty which will make the output more readable. I also added the content-type header. You'll see the "acknowledge: true" output on success.

$ curl -H 'Content-Type: application/json' -XPUT http://localhost:9200/_template/winlogbeat?pretty -d@winlogbeat.template.json 
{
  "acknowledged" : true
}

Do you mean it hangs and doesn't return?

You can check to see if the template is installed with

curl -H 'Content-Type: application/json' http://localhost:9200/_template/winlogbeat?pretty

That will return the full template.

9

This worked both. I got the aknowledged: true response. I also can see the full template.

With: GET /_cat/indices?v I still can't see the new index in Kibana. Probably something is still missing?

10

Check the Winlogbeat logs to see if it is successfully delivering events to Logstash. Then check the Logstash and Elasticsearch logs to see if there are any errors.

11

I see the following error in the winlogbeat log.

2017-09-12T16:20:08+02:00 ERR Failed to publish events caused by: write tcp 192.168.0.29:59543->192.168.0.13:5044: wsasend: Eine vorhandene Verbindung wurde vom Remotehost geschlossen.
2017-09-12T16:20:08+02:00 INFO Error publishing events (retrying): write tcp 192.168.0.29:59543->192.168.0.13:5044: wsasend: Eine vorhandene Verbindung wurde vom Remotehost geschlossen.

It means in english: An already existing connection have been closed from the remotehost.

So probably it's something with Winlogbeat.

12

Are there any log lines containing metrics (like "Non-zero metrics in the last 30s")? That can tell you if any data was sent to Logstash.

But it sounds like there's an issue with Logstash or the network path to Logstash. Are you sure LS is listening on 5044? From the Logstash host, what does sudo netstat -anp | grep 5044 show? Another test is to check if you can telnet from Windows to port 5044 on the LS host.

13

It looks good to me concering logstash:

tcp6 0 0 :::5044 :::* LISTEN 24162/java

I can't telnet from Windows to 5044 !

Logstash Input/Output file at the moment looks like that:

input {
#  file {
#    path => "/tmp/in.log"
#    type => "logfile"  # a type to identify those logs (will need this later)
#    start_position => "beginning"
#   }
syslog {
    port => 5514
    type => "syslog"
}
udp {
    type => "pfsense"
    port => 5140
}
beats {
    port => 5044
}
udp {
    type => "syslog"
    port => 5515

}
}

output {
#  file {
#    path => "/tmp/out.log"
#  }
 if [@metadata][beat] {
  elasticsearch {
  hosts => "localhost:9200"
  index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  document_type => "%{[@metadata][type]}"
 }
  } else {
  elasticsearch {
     hosts => "localhost:9200"
     manage_template => false
     index => "logstash-%{+YYYY.MM.dd}"
   }
 }
#  if [@metadata][cthostmeta] == "ELKSTACK" {
#    file {
#       codec => rubydebug { metadata => true}
#       path => "/tmp/logstash_out.log"
#    }
#   }
}

And yes I found the following line:

2017-09-12T16:20:19+02:00 DBG  EventLog[System] Read() returned 0 records
2017-09-12T16:20:19+02:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=7 libbeat.logstash.publish.read_bytes=54 libbeat.logstash.publish.write_bytes=14757 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_and_acked_events=26 libbeat.logstash.published_but_not_acked_events=1 libbeat.publisher.published_events=26 msg_file_cache.ApplicationMisses=1 msg_file_cache.ApplicationSize=1 msg_file_cache.SecurityHits=23 msg_file_cache.SecurityMisses=1 msg_file_cache.SecuritySize=1 msg_file_cache.SystemHits=1 msg_file_cache.SystemMisses=1 msg_file_cache.SystemSize=1 published_events.Application=1 published_events.Security=23 published_events.System=2 published_events.total=26 uptime={"server_time":"2017-09-12T14:20:19.3943421Z","start_time":"2017-09-12T10:18:19.5091167Z","uptime":"4h1m59.8852254s","uptime_ms":"14519885225"}
2017-09-12T16:20:19+02:00 DBG  WinEventLog[Security] No more events
14

This indicates that it was able to send 26 events to Logstash.

I would uncomment that file output you have in the Logstash config to double check that the events are being received in LS. Then check the LS logs to see what's happening that they never make it to ES.

15

Something is now going wrong :(. I just played around with the inputoutput file and now the logstash pipeline can't start anymore.

[2017-09-12T19:33:57,828][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 192, column 1 (byte 4770) after "}

Can you also help with this logstash problem?

output {
#  file {
#    path => "/tmp/out.log"
#  }
 if [@metadata][beat] {
  elasticsearch {
  hosts => "localhost:9200"
  index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  document_type => "%{[@metadata][type]}"
 }
  } else {
  elasticsearch {
     hosts => "localhost:9200"
     manage_template => false
     index => "logstash-%{+YYYY.MM.dd}"
   }
 }
#  if [@metadata][cthostmeta] == "ELKSTACK" {
#    file {
#       codec => rubydebug { metadata => true}
#       path => "/tmp/logstash_out.log"
#    }
#   }
}
16

Hmm, not sure w/o seeing the whole file. Try this. (tip: Using a consistent indentation style will help make it easier to see mismatched braces.)

input {
  syslog {
    port => 5514
    type => "syslog"
  }
  udp {
    type => "pfsense"
    port => 5140
  }
  beats {
    port => 5044
  }
  udp {
    type => "syslog"
    port => 5515
  }
}

output {
  file {
    path => "/tmp/logstash-events.out"
  }

  if [@metadata][beat] {
    elasticsearch {
      hosts => "localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
    }   
  } else {
    elasticsearch {
      hosts => "localhost:9200"
      manage_template => false
      index => "logstash-%{+YYYY.MM.dd}"
    }   
  }
}
17

I got it working back again.
I had to delete the .json file from the folder. This was stupid to pick that folder from me. I'm sorry.

Now I was able to create the index file in Kibana :). So far it looks good. I check tomorrow how it is working.

18

This topic was automatically closed after 21 days. New replies are no longer allowed.