Stopping pipeline {:id=>"main"}

Elastic Stack Logstash
1

Hi,
I am having an issue when starting logstash. I have checked the other solutions in the forum but they dont seem to be related to the issue I am having. Can someone please help? Thank you.

[2018-06-01T12:32:23,733][DEBUG][logstash.filters.grok ] replacement_pattern => (?:(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])...)(?![0-9]))
[2018-06-01T12:32:23,733][DEBUG][logstash.filters.grok ] replacement_pattern => (?:\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))(.?|\b))
[2018-06-01T12:32:23,751][DEBUG][logstash.filters.grok ] Grok compiled OK {:pattern=>"%{IPORHOST:clientip}", :expanded_pattern=>"(?IPORHOST:clientip(?:(?:(?:(?:((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?)|(?:(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])...)(?![0-9]))))|(?:\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))(\.?|\b))))"}
[2018-06-01T12:32:23,756][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.3.1-java/vendor/GeoLite2-City.mmdb"}
[2018-06-01T12:32:23,847][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125}
[2018-06-01T12:32:23,885][INFO ][logstash.pipeline ] Pipeline main started
[2018-06-01T12:32:23,886][DEBUG][logstash.pipeline ] Input plugins stopped! Will shutdown filter/output workers.
[2018-06-01T12:32:23,987][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2018-06-01T12:32:23,989][DEBUG][logstash.pipeline ] Pushing shutdown {:thread=>"#<Thread:0xf1626d2 run>"}
[2018-06-01T12:32:23,989][DEBUG][logstash.pipeline ] Shutdown waiting for worker thread #Thread:0xf1626d2
[2018-06-01T12:32:24,027][DEBUG][logstash.filters.grok ] closing {:plugin=>"LogStash::Filters::Grok"}
[2018-06-01T12:32:24,173][DEBUG][logstash.filters.geoip ] closing {:plugin=>"LogStash::Filters::GeoIP"}
[2018-06-01T12:32:24,174][DEBUG][logstash.outputs.elasticsearch] closing {:plugin=>"LogStash::Outputs::ElasticSearch"}
[2018-06-01T12:32:24,174][DEBUG][logstash.outputs.elasticsearch] Stopping sniffer
[2018-06-01T12:32:24,383][DEBUG][logstash.outputs.elasticsearch] Stopping resurrectionist
[2018-06-01T12:32:24,383][DEBUG][logstash.outputs.elasticsearch] Waiting for in use manticore connections
[2018-06-01T12:32:24,383][DEBUG][logstash.outputs.elasticsearch] Closing adapter #LogStash::Outputs::ElasticSearch::HttpClient::ManticoreAdapter:0x4f45b0b3
[2018-06-01T12:32:24,384][DEBUG][logstash.pipeline ] Pipeline main has been shutdown
[2018-06-01T12:32:24,390][DEBUG][logstash.agent ] Starting puma
[2018-06-01T12:32:24,390][DEBUG][logstash.agent ] Trying to start WebServer {:port=>9600}
[2018-06-01T12:32:24,391][DEBUG][logstash.api.service ] [api-service] start
[2018-06-01T12:32:24,483][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-06-01T12:32:27,458][DEBUG][logstash.instrument.periodicpoller.os] PeriodicPoller: Stopping
[2018-06-01T12:32:27,458][DEBUG][logstash.instrument.periodicpoller.jvm] PeriodicPoller: Stopping
[2018-06-01T12:32:27,459][DEBUG][logstash.instrument.periodicpoller.persistentqueue] PeriodicPoller: Stopping
[2018-06-01T12:32:27,459][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] PeriodicPoller: Stopping
[2018-06-01T12:32:27,467][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
[2018-06-01T12:32:27,468][DEBUG][logstash.pipeline ] Closing inputs
[2018-06-01T12:32:27,468][DEBUG][logstash.pipeline ] Closed inputs

/etc/logstash/conf.d/11-iis-log-filter.conf

 filter {
  if [type] == "iis" {
grok {
  match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:S-SiteName} %{NOTSPACE:S-ComputerName} %{IPORHOST:S-IP} %{WORD:CS-Method} %{URIPATH:CS-URI-Stem} (?:-|\"%{URIPATH:CS-URI-Query}\") %{NUMBER:S-Port} %{NOTSPACE:CS-Username} %{IPORHOST:C-IP} %{NOTSPACE:CS-Version} %{NOTSPACE:CS-UserAgent} %{NOTSPACE:CS-Cookie} %{NOTSPACE:CS-Referer} %{NOTSPACE:CS-Host} %{NUMBER:SC-Status} %{NUMBER:SC-SubStatus} %{NUMBER:SC-Win32-Status} %{NUMBER:SC-Bytes} %{NUMBER:CS-Bytes} %{NUMBER:Time-Taken}"}
}
  }
}

30-elasticsearch-output.conf

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
2

And what input plugins are you using?

3

filebeat is the agent I am using. I am new to this. thanks.

4

Yes, but you do have a beats input in your Logstash configuration don't you?

5

got it thank you for the direction i was missing the input.

input {
  beats {
    port => 5443
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

Adding this fixed it. But somehow logstash still hasnt picked up anything from the iis server. Which check on that. Thanks again.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.