I have enabled xpack in elasticsearch and kibana which is working fine. I tried to enable it in logstash and used filebeat to ingest the data.
But while starting the logstash it is spitting out the following error: ( I tried running pipelines.yml also which doesn't help )
logstash.bat -f E:\ELK\6.8vELK\logstash-6.8.0\config\logstash-sample.conf
Sending Logstash logs to E:/ELK/6.8vELK/logstash-6.8.0/logs which is now configured via log4j2.properties
[2019-06-17T12:13:32,604][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["mdm_contact_pipeline"], :poll_interval=>"5000000000ns"}
[2019-06-17T12:13:37,701][ERROR][logstash.configmanagement.elasticsearchsource] Configuration Management is not available: basic is not a valid license for this feature.
[2019-06-17T12:13:37,740][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::LicenseChecker::LicenseError: Configuration Management is not available: basic is not a valid license for this feature.>, :backtrace=>["E:/ELK/6.8vELK/logstash-6.8.0/x-pack/lib/license_checker/licensed.rb:67:in `with_license_check'", "E:/ELK/6.8vELK/logstash-6.8.0/x-pack/lib/config_management/elasticsearch_source.rb:46:in `initialize'", "E:/ELK/6.8vELK/logstash-6.8.0/x-pack/lib/config_management/hooks.rb:41:in `after_bootstrap_checks'", "org/logstash/execution/EventDispatcherExt.java:71:in `execute'", "E:/ELK/6.8vELK/logstash-6.8.0/logstash-core/lib/logstash/runner.rb:304:in `execute'", "E:/ELK/6.8vELK/logstash-6.8.0/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "E:/ELK/6.8vELK/logstash-6.8.0/logstash-core/lib/logstash/runner.rb:237:in `run'", "E:/ELK/6.8vELK/logstash-6.8.0/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "E:\\ELK\\6.8vELK\\logstash-6.8.0\\lib\\bootstrap\\environment.rb:73:in `<main>'"]}
[2019-06-17T12:13:37,772][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped
processing because of an error: (SystemExit) exit
I'm attaching the config files below:
logstash.yml
xpack.management.enabled: true
xpack.management.pipeline.id: ["mdm_contact_pipeline"]
xpack.management.elasticsearch.username: elastic
xpack.management.elasticsearch.password: LP5gC}!mv
xpack.management.elasticsearch.hosts: "http://localhost:9200/"
elasticsearch.yml
cluster.name: ElasticsearchCluster
xpack.security.audit.enabled: true
xpack.license.self_generated.type : basic
xpack.security.enabled: true
logstash-sample.conf
input {
beats {
type => beats
port => 5044
}
}
filter {
grok {
match => [
"message", '%{IPV4:remoteIP}\s+%{IPV4:localIP}\s+%{INT:throughtputData:int}\s+%{INT}\s+%{IPV4:remoteHostIP}\s+%{DATA:httpProtocol}\s+%{DATA:remoteLogicalUserName}\s+%{DATA:requestMethod}\s+%{DATA:port}\s+%{DATA:queryString}\s+%{DATA:requestMethod2}\s+%{DATA:requestURI}\s+%{DATA:requestProtocol2}\s+%{INT:requestStatusCode}\s+%{DATA:userSessionID}\s+\[%{HTTPDATE:logTimeStamp}\]\s+%{DATA:remoteUser}\s+%{DATA:requestedURL}\s+%{DATA:serverName}\s+%{INT:timeTakenInMilliSec:int}\s+%{NUMBER:timeTakenInSec}\s+default\s+task-%{INT:taskID:int}\s+"%{DATA:authorization}"\s+"%{DATA:contentType}"\s+"%{DATA:referer}"\s+"%{DATA:userAgent}"\s+"%{DATA:cookie}"\s+"%{DATA:setCookie}"'
]
}
if "_grokparsefailure" in [tags] {
drop {}
}
if "_groktimeout" in [tags] {
drop {}
}
date {
match => ["logTimeStamp" ,"dd/MMM/yyyy:HH:mm:ss Z" ]
}
mutate {
remove_field => ["message","host","input","type","@version","prospector","beat","garbageData","offset"]
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "logs-%{+YYYY.MM.dd}"
stdout {
codec => rubydebug
}
# user => "logstash_system"
# password => "LP5gC}!mv"
}
}
pipelines.yml
- pipeline.id: mdm_contact_pipeline
path.config: "E:/ELK/6.8vELK/logstash-6.8.0/config/logstash-sample.conf"