Centralized pipeline management - pipeline terminating


(Ramesh Pendela) #1

Hello community,

I am trying to use the centralize pipeline management feature , I followed the document https://www.elastic.co/guide/en/logstash/6.0/logstash-centralized-pipeline-management.html
I have a x-pack licence aswell. but after creating the pipeline using kibana UI -> logstash ->pipeline when I start the logstash I am getting this

Pipeline terminated {"pipeline.id"=>".monitoring-logstash"} logstash_test exited with code 0

`    logstash.config.source.local.configpathloader] No config files found in path {:path=>"/usr/share/logstash/pipeline/*"}`

I know this error is coming because of no pipeline configuration files in the logstash pipeline directory but I guess that is not the case if I am using centralized pipeline management

and

if I use the same pipeline configuration in logstash/pipeline directory it is working fine. logstash not giving any errors. what might be the cause?

here is my elasticsearch.yml file

**node.name: elastic-test **
cluster.name: "elk-cluster"
network.host: 0.0.0.0
network.bind_host: 0.0.0.0
transport.tcp.port: 9300
http.port: 9200
network.publish_host: ...
discovery.zen.minimum_master_nodes: 1
discovery.type: single-node
xpack.security.enabled: false

Kibana.yml

server.host: "0"
server.name: kibana-test
elasticsearch.url: "http://elasticsearch_test:9200"
elasticsearch.username: ######
elasticsearch.password:######
xpack.monitoring.ui.container.elasticsearch.enabled: true
#xpack.security.enabled: true

Logstash.yml

node.name: logstash-1
http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline
config.reload.automatic: true
#xpack.monitoring.enabled: true
#xpack.monitoring.elasticsearch.url: "http://#.#.#.#:9200"
#xpack.monitoring.elasticsearch.username: logstash_system
#xpack.monitoring.elasticsearch.password: changeme
xpack.management.enabled: true
xpack.management.elasticsearch.url: "http://#.#.#.#:9200/"
xpack.management.elasticsearch.username: logstash_admin_user
xpack.management.elasticsearch.password: elastic
xpack.management.logstash.poll_interval: 5s
xpack.management.pipeline.id: ["paloalto", "winlogbeats"]

My docker-compose file

`version: '2.2'

services:

######## ELASTICSEARCH ######################

elasticsearch:
build: /usr/share/elk/elasticsearch/
container_name: elasticsearch_test
environment:
- cluster.name=elk-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- ELASTIC_PASSWORD=changeme
ulimits:
nofile:
soft: 65536
hard: 65536
memlock:
soft: -1
hard: -1
mem_limit: 1g
ports:
- 9300:9300
- 9200:9200
restart: always
volumes:
- es_data1:/usr/share/elasticsearch/data
- /usr/share/elk/configs/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
networks:
- elk

########### LOGSTASH ###########

logstash:
build: /usr/share/elk/logstash/
container_name: logstash_test
environment:
- cluster.name=elk-cluster
- bootstrap.memory_lock=true
- "LS_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
nofile:
soft: 65536
hard: 65536
memlock:
soft: -1
hard: -1
mem_limit: 1g
ports:
- 5000:5000
- 1514:1514
- 5044:5044
- 7575:7575
volumes:
- ls_data1:/usr/share/logstash/data
- /usr/share/elk/configs/logstash.yml:/usr/share/config/logstash.yml
- /usr/share/elk/logstash/pipeline:/usr/share/logstash/pipeline
networks:
- elk
depends_on:
- elasticsearch

################ KIBANA ###############

kibana:
build: /usr/share/elk/kibana/
container_name: kibana_test
environment:
- cluster.name=elk-cluster
#- "ELASTICSEARCH_URL=http://elasticsearch:9200"
mem_limit: 1g
volumes:
- kibana_data1:/usr/share/kibana/data
- /usr/share/elk/configs/kibana.yml:/usr/share/kibana/config/kibana.yml
ports:
- "5601:5601"
networks:
- elk
depends_on:
- elasticsearch

volumes:
es_data1:
driver: local
ls_data1:
driver: local
kibana_data1:
driver: local
networks:
elk:
driver: bridge`


(Shaunak Kashyap) #2

When you are using centralized configuration management, can you try commenting out the path.config line in your logstash.yml file?


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.