No indices - but winlogbeat thinks it's working

Elastic Stack Beats
1

Winlogbeat is pretty happy, but I'm not able to see anything in Kibana. I have some other beats reporting data.

Here's a snip from winlogbeat:
2019-07-19T11:13:25.816-0400 INFO beater/eventlogger.go:76 EventLog[ForwardedEvents] successfully published 12 events
2019-07-19T11:13:26.825-0400 INFO beater/eventlogger.go:76 EventLog[ForwardedEvents] successfully published 34 events
2019-07-19T11:13:28.871-0400 INFO beater/eventlogger.go:76 EventLog[ForwardedEvents] successfully published 29 events
2019-07-19T11:13:29.901-0400 INFO beater/eventlogger.go:76 EventLog[ForwardedEvents] successfully published 17 events
2019-07-19T11:13:30.198-0400 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10625,"time":{"ms":188}},"total":{"ticks":356078,"time":{"ms":454},"value":356078},"user":{"ticks":345453,"time":{"ms":266}}},"handles":{"open":194},"info":{"ephemeral_id":"c07ef45e-1e91-4ff7-b774-d9bb70dfb9eb","uptime":{"ms":750173}},"memstats":{"gc_next":9007840,"memory_alloc":4518488,"memory_total":34495644784,"rss":-634880},"runtime":{"goroutines":20}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":601,"batches":26,"total":601},"read":{"bytes":156},"write":{"bytes":215963}},"pipeline":{"clients":1,"events":{"active":0,"published":594,"total":594},"queue":{"acked":601}}},"published_events":{"ForwardedEvents":601,"total":601}}}}
2019-07-19T11:13:31.964-0400 INFO beater/eventlogger.go:76 EventLog[ForwardedEvents] successfully published 96 events
2019-07-19T11:13:32.950-0400 INFO beater/eventlogger.go:76 EventLog[ForwardedEvents] successfully published 4 events
2019-07-19T11:13:37.000-0400 INFO beater/eventlogger.go:76 EventLog[ForwardedEvents] successfully published 10 events
2019-07-19T11:13:39.148-0400 INFO beater/eventlogger.go:76 EventLog[ForwardedEvents] successfully published 9 events
2019-07-19T11:13:40.173-0400 INFO beater/eventlogger.go:76 EventLog[ForwardedEvents] successfully published 28 events
EventLog[ForwardedEvents] successfully published 58 events
2019-07-19T11:13:49.345-0400 INFO beater/eventlogger.go:76 EventLog[ForwardedEvents] successfully published 11 events
2019-07-19T11:13:58.460-0400 INFO beater/eventlogger.go:113 EventLog[ForwardedEvents] Stop processing.
2019-07-19T11:13:58.519-0400 INFO [monitoring] log/log.go:153 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10781,"time":{"ms":10781}},"total":{"ticks":356406,"time":{"ms":356406},"value":356406},"user":{"ticks":345625,"time":{"ms":345625}}},"handles":{"open":183},"info":{"ephemeral_id":"c07ef45e-1e91-4ff7-b774-d9bb70dfb9eb","uptime":{"ms":778493}},"memstats":{"gc_next":8819936,"memory_alloc":8108184,"memory_total":34544121936,"rss":88092672},"runtime":{"goroutines":14}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":818613,"active":10,"batches":655,"total":818623},"read":{"bytes":3924},"type":"logstash","write":{"bytes":245063516}},"pipeline":{"clients":0,"events":{"active":10,"published":818623,"retry":2048,"total":818623},"queue":{"acked":818613}}},"published_events":{"ForwardedEvents":818613,"total":818613},"system":{"cpu":{"cores":1}}}}}
2019-07-19T11:13:58.523-0400 INFO [monitoring] log/log.go:154 Uptime: 12m58.4984278s
2019-07-19T11:13:58.523-0400 INFO [monitoring] log/log.go:131 Stopping metrics logging.
2019-07-19T11:13:58.573-0400 INFO instance/beat.go:431 winlogbeat stopped.

The logstash logs are pretty silent (except when I stop the winlogbeat service)

2019-07-19T10:52:06,258][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://10.10.8.18:9200/"}
[2019-07-19T10:52:06,306][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
[2019-07-19T10:52:06,308][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2019-07-19T10:52:06,353][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://10.10.8.18:9200"]}
[2019-07-19T10:52:06,386][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-07-19T10:52:06,442][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refr
esh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms
"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}],
 "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"
=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2019-07-19T10:52:06,779][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for k
ey: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
[2019-07-19T10:52:06,803][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.
max_inflight"=>125, :thread=>"#<Thread:0x5736aeac run>"}
[2019-07-19T10:52:08,735][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2019-07-19T10:52:08,845][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"main"}
[2019-07-19T10:52:08,931][INFO ][logstash.inputs.snmptrap ] It's a Trap! {:Port=>1062, :Community=>["public"], :Host=>"0.0.0.0"}
[2019-07-19T10:52:09,296][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-07-19T10:52:09,890][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2019-07-19T10:52:13,797][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2019-07-19T10:59:13,064][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: 10.10.8.5:63492] Handling exception: Connection reset by peer

Screenshot from Kibana (and no, not sure why they're yellow yet :blush: )

kibana
kibana.png1053×480 41.2 KB

List of open ports from firewall-cmd:
5601/tcp 9200/tcp 9300/tcp 5044/tcp 1062/tcp 9600/tcp

What else do you need to figure out what I'm doing wrong? :slight_smile:

2

Can you please share your LS config? Did you start with the config from here?

https://www.elastic.co/guide/en/elastic-stack-get-started/7.2/get-started-elastic-stack.html#logstash-setup

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}
1 Like
3

It's a custom config - I was using something similar back on 6.6 but I tried to transplant it into a new install.

input {
    beats {
        port => 5044
    }
    snmptrap {
        type => "snmptrap"
        host => "0.0.0.0"
        port => 1062
    }
}
 
output {
    if [type] == "wineventlog" {
        elasticsearch { hosts => ["http://10.10.8.18:9200"]
            index => "logstash-winlogbeat-%{+YYYY.MM.dd}"
        } 
    } else if [type] == "packetbeat" {
        if [host][name] == "HAI-WEB01" {
        elasticsearch { hosts => ["http://10.10.8.18:9200"]
                index => "logstash-packetbeat-%{+YYYY.MM.dd}"
}
}
    } else if [tags] == "metricbeat" {
        elasticsearch { hosts => ["http://10.10.8.18:9200"]
        index => "logstash-metricbeat-%{+YYYY.MM.dd}"
}
    } else if [type] == "snmptrap" {
        elasticsearch { hosts => ["http://10.10.8.18:9200"]
        index => "logstash-snmp-%{+YYYY.MM.dd}"
}
    } else if [host][name] == "HAI-ADM01" {
        elasticsearch { hosts => ["http://10.10.8.18:9200"]
            index => "logstash-adm01-%{+YYYY.MM.dd}"
}
    } else if [host][name] == "HAI-RDS" {
        elasticsearch { hosts => ["http://10.10.8.18:9200"]
            index => "logstash-rds-%{+YYYY.MM.dd}"
}
    }
}

Though I will say I like the whole "%{[@metadata][beat]}, I didn't know that was an option

4

I think type was removed. Better check the breaking changes docs. Breaking changes in 7.0 | Beats Platform Reference [7.0] | Elastic

1 Like
5

oh boy, that would certainly do it :blush:

Thanks, I'll take a look!

6

So I don't see it mentioned specifically - but clearly that's the issue.
I see that 'type' was renamed in winlogbeat but that shouldn't impact logstash? Or am I misunderstanding how things correlate.

And I guess I don't need to do that anymore due to the whole metadata/beat thing. Is there a complete listing of those options?

7

The type field was coming from the Beat and it's not there anymore so you cannot use it as the basis for routing events in Logstash. If you wanted an equivalent config you'd need to change it to if [winlog][api] == "wineventlog". But I think it would be better to change it to something like

if [@metadata][beat] { 
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
} else if [type] == "snmptrap" ...
2 Likes
8

Beautiful, thank you so much.

Is there no full list of what [substitutions] are available?

9

Any field in the event can be used. So if the event contains {winlog: {channel: "Application"}} then you can reference that field with [winlog][channel].

You can dump the full event from Logstash with the stdout output.

output { 
  stdout { 
    codec  => rubydebug {
      metadata => true
    }
  }
}
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.