ERROR: Starting the Logstash with Xpack

I have enabled xpack in elasticsearch and kibana which is working fine. I tried to enable it in logstash and used filebeat to ingest the data.

But while starting the logstash it is spitting out the following error: ( I tried running pipelines.yml also which doesn't help )

logstash.bat -f E:\ELK\6.8vELK\logstash-6.8.0\config\logstash-sample.conf
Sending Logstash logs to E:/ELK/6.8vELK/logstash-6.8.0/logs which is now configured via log4j2.properties
[2019-06-17T12:13:32,604][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["mdm_contact_pipeline"], :poll_interval=>"5000000000ns"}
[2019-06-17T12:13:37,701][ERROR][logstash.configmanagement.elasticsearchsource] Configuration Management is not available: basic is not a valid license for this feature.
[2019-06-17T12:13:37,740][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::LicenseChecker::LicenseError: Configuration Management is not available: basic is not a valid license for this feature.>, :backtrace=>["E:/ELK/6.8vELK/logstash-6.8.0/x-pack/lib/license_checker/licensed.rb:67:in `with_license_check'", "E:/ELK/6.8vELK/logstash-6.8.0/x-pack/lib/config_management/elasticsearch_source.rb:46:in `initialize'", "E:/ELK/6.8vELK/logstash-6.8.0/x-pack/lib/config_management/hooks.rb:41:in `after_bootstrap_checks'", "org/logstash/execution/EventDispatcherExt.java:71:in `execute'", "E:/ELK/6.8vELK/logstash-6.8.0/logstash-core/lib/logstash/runner.rb:304:in `execute'", "E:/ELK/6.8vELK/logstash-6.8.0/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "E:/ELK/6.8vELK/logstash-6.8.0/logstash-core/lib/logstash/runner.rb:237:in `run'", "E:/ELK/6.8vELK/logstash-6.8.0/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "E:\\ELK\\6.8vELK\\logstash-6.8.0\\lib\\bootstrap\\environment.rb:73:in `<main>'"]}
[2019-06-17T12:13:37,772][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped
processing because of an error: (SystemExit) exit

I'm attaching the config files below:

logstash.yml

xpack.management.enabled: true
xpack.management.pipeline.id: ["mdm_contact_pipeline"]
xpack.management.elasticsearch.username: elastic
xpack.management.elasticsearch.password: LP5gC}!mv
xpack.management.elasticsearch.hosts: "http://localhost:9200/"

elasticsearch.yml

cluster.name: ElasticsearchCluster
xpack.security.audit.enabled: true
xpack.license.self_generated.type : basic
xpack.security.enabled: true

logstash-sample.conf

input {
  beats {
    type => beats
    port => 5044
  }
}

filter {
  grok {
      match => [ 
            "message", '%{IPV4:remoteIP}\s+%{IPV4:localIP}\s+%{INT:throughtputData:int}\s+%{INT}\s+%{IPV4:remoteHostIP}\s+%{DATA:httpProtocol}\s+%{DATA:remoteLogicalUserName}\s+%{DATA:requestMethod}\s+%{DATA:port}\s+%{DATA:queryString}\s+%{DATA:requestMethod2}\s+%{DATA:requestURI}\s+%{DATA:requestProtocol2}\s+%{INT:requestStatusCode}\s+%{DATA:userSessionID}\s+\[%{HTTPDATE:logTimeStamp}\]\s+%{DATA:remoteUser}\s+%{DATA:requestedURL}\s+%{DATA:serverName}\s+%{INT:timeTakenInMilliSec:int}\s+%{NUMBER:timeTakenInSec}\s+default\s+task-%{INT:taskID:int}\s+"%{DATA:authorization}"\s+"%{DATA:contentType}"\s+"%{DATA:referer}"\s+"%{DATA:userAgent}"\s+"%{DATA:cookie}"\s+"%{DATA:setCookie}"'
        ]
    }
    
      if "_grokparsefailure" in [tags] {
	  	drop {}
	  }
	  
	  if "_groktimeout" in [tags] {
	  	drop {}
	  }
	 
	date {
		match => ["logTimeStamp" ,"dd/MMM/yyyy:HH:mm:ss Z" ]
	}

	mutate {
	remove_field => ["message","host","input","type","@version","prospector","beat","garbageData","offset"]
	}
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "logs-%{+YYYY.MM.dd}"
    stdout { 
      codec => rubydebug 
      }
    # user => "logstash_system"
    # password => "LP5gC}!mv"
  }
}

pipelines.yml

 - pipeline.id: mdm_contact_pipeline
   path.config: "E:/ELK/6.8vELK/logstash-6.8.0/config/logstash-sample.conf"

Hi @Sundaramoorthy_Anand,

You are using Logstash pipeline management feature. I would like to inform that this feature is not under Basic license as it is under GOLD or PLATINUM. This is the reason its showing "Configuration Management is not available: basic is not a valid license for this feature." error.

Please refer below link for subscription details:
https://www.elastic.co/subscriptions

Regards,
Harsh Bajaj

Now, I have commented the xpack.management.pipeline.id: ["mdm_contact_pipeline"] line and tried.. The error is :

  E:\ELK\6.8vELK\logstash-6.8.0\bin>logstash.bat -f E:\ELK\6.8vELK\logstash-6.8.0\config\logstash-sample.conf
Sending Logstash logs to E:/ELK/6.8vELK/logstash-6.8.0/logs which is now configured via log4j2.properties
[2019-06-17T14:03:15,806][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["main"], :poll_interval=>"5000000000ns"}
[2019-06-17T14:03:15,885][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<ArgumentError: You must set the password using the "xpack.management.elasticsearch.password" in logstash.yml>, :backtrace=>["E:/ELK/6.8vELK/logstash-6.8.0/x-pack/lib/config_management/elasticsearch_source.rb:39:in `initialize'", "E:/ELK/6.8vELK/logstash-6.8.0/x-pack/lib/config_management/hooks.rb:41:in `after_bootstrap_checks'", "org/logstash/execution/EventDispatcherExt.java:71:in `execute'", "E:/ELK/6.8vELK/logstash-6.8.0/logstash-core/lib/logstash/runner.rb:304:in `execute'",
"E:/ELK/6.8vELK/logstash-6.8.0/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "E:/ELK/6.8vELK/logstash-6.8.0/logstash-core/lib/logstash/runner.rb:237:in `run'", "E:/ELK/6.8vELK/logstash-6.8.0/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "E:\\ELK\\6.8vELK\\logstash-6.8.0\\lib\\bootstrap\\environment.rb:73:in `<main>'"]}
[2019-06-17T14:03:15,915][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Hi @Sundaramoorthy_Anand,

Please read below document for Configuring Centralized Pipeline Management in which mentioned you need to enable x-pack security with license based subscription otherwise you need to remove all those settings related to this.

https://www.elastic.co/guide/en/logstash/6.6/configuring-centralized-pipelines.html

Regards,
Harsh Bajaj

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.