Winlogbeats indices created but docs appear as deleted

Elastic Stack Beats
1

Hi

Problem:
Indices are created, and I see traffic on port 5044 but all docs are created as deleted

winlogbeat-deleted-docs.PNG937×578 28.4 KB

Configuration
I configured the system as follows:

Winlogbeats --> Logstash --> Elasticsearch --> Kibana

Installed the winlogbeats on the server Windows 2012 R2 64bit

and invoked this command on Powershell as per doucmentation and it passed

Invoke-WebRequest -Method Put -InFile winlogbeat.template.json -Uri http://elk:9200/_template/winlogbeat?pretty

Winlogbeats

winlogbeat.event_logs:
  - name: Application
    #ignore_older: 72h
  - name: Security
  - name: System

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["elk:5044"]

#================================ Logging =====================================
logging.to_files: true
logging.files:
    path: C:\ProgramData\winlogbeat\Logs
logging.level: info

Logstash

input
{
    beats
    {
        ssl => false
        host => "0.0.0.0"
        port => 5044
    }
    gelf
    {
        host => "0.0.0.0"
        port => 12201
    }
    http
    {
        ssl => false
        host => "0.0.0.0"
        port => 8888
    }
    tcp
    {
        mode => "server"
        host => "0.0.0.0"
        port => 5010
    }
    udp
    {
        host => "0.0.0.0"
        port => 5000
    }
}
filter {
        if [type] == "syslog"
        {
                mutate { add_tag => "syslog_tag" }
                grok
                {
                        match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
                        add_field => [ "received_at", "%{@timestamp}" ]
                        add_field => [ "received_from", "%{host}" ]
                }
                syslog_pri { }
                date
                {
                        match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
                }
        }
        if [type] == "eventlog"
        {
                grok
                {
                        match => [ "TimeCreated", "Date\(%{NUMBER:timestamp}\)" ]
                }
                date
                {
                        match => [ "timestamp", "UNIX_MS" ]
                }
        }
}
output
{
        if "syslog_tag" in [tags]
        {
                elasticsearch
                        {
                                hosts => ["localhost:9200"]
                                sniffing => true
                                manage_template => false
                                index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
                                document_type => "%{[@metadata][type]}"
                        }
        }
        if [type] == "wineventlog"
        {
                elasticsearch
                {
                        hosts => ["127.0.0.1:9200"]
                        document_id => "%{logstash_checksum}"
                        index => "winlogbeat-%{+YYYY.MM.dd}"
                }
        }
        if [type] == "eventlog"
        {
                elasticsearch
                {
                        hosts => ["127.0.0.1:9200"]
                        index => "winbeatlog-%{+YYYY.MM.dd}"
                }
        }
        elasticsearch
        {
                hosts => ["127.0.0.1:9200"]
                document_id => "%{logstash_checksum}"
                index => "logstash-%{+YYYY.MM.dd}"
        }

}

ElasticSearch

cluster.name: elasticsearch
# ---------------------------------- Network -----------------------------------
network.host: 0.0.0.0
network.publish_host: 192.168.10.137
http.port: 9200
transport.tcp.port: 9300
2

Try removing the document_id from the elasticsearch output for the winlogbeat type. In accordance with the docs you should use:

  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" 
    document_type => "%{[@metadata][type]}" 
  }

So add manage_template and document_type and remove document_id.

3

Thank you, It worked like charm

Would you please tell me why the document_id was making a problem?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.