# Docker ELK (multipipelines and pipeline error)

**URL:** <https://discuss.elastic.co/t/docker-elk-multipipelines-and-pipeline-error/245936>\
**Category:** Logstash\
**Tags:** docker\
**Created:** [August 21, 2020, 4:04pm UTC](https://discuss.elastic.co/t/docker-elk-multipipelines-and-pipeline-error/245936 "2020-08-21T16:04:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![nb03briceno](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nb03briceno/32/45777_2.png) [@nb03briceno](https://discuss.elastic.co/u/nb03briceno)\
**Post date:** [August 21, 2020, 4:04pm UTC](https://discuss.elastic.co/t/docker-elk-multipipelines-and-pipeline-error/245936/1 "2020-08-21T16:04:40Z")

</div>

Hello everyone,

I'm working with ELK in docker. I was able to run Elasticsearch, KIbana, Logstash, and Filebeat with docker containers through a docker-compose file. Ingesting appear to works when I use simple pipelines, yet in the moment I forward that with more complex pipelines it is giving me some errors.

## This is my docker-compose file:

version: "2.1"  
services:

# The environment variable "ELASTIC\_VERSION" is used throughout this file to

# specify the version of the images to run. The default is set in the

# '.env' file in this folder. It can be overridden with any normal

# technique for setting environment variables, for example:

# 

# ELASTIC\_VERSION=5.5.1 docker-compose up

# 

# Additionally, the user can control:

# \* the total memory assigned to the ES container through the variable ES\_MEM\_LIMIT e.g. ES\_MEM\_LIMIT=2g

# \* the memory assigned to the ES JVM through the variable ES\_JVM\_HEAP e.g. ES\_JVM\_SIZE=1024m

# \* the password used for the elastic, logstash\_system and kibana accounts through the variable ES\_PASSWORD

# \* the mysql root password through the var MYSQL\_ROOT\_PASSWORD

# \* the default index pattern used in kibana via the var DEFAULT\_INDEX\_PATTERN

# \* the ES heap size through tt

# REF: [https://docs.docker.com/compose/compose-file/#variable-substitution](https://docs.docker.com/compose/compose-file/#variable-substitution)

# 

elasticsearch:  
container\_name: elasticsearch  
hostname: elasticsearch  
image: "[docker.elastic.co/elasticsearch/elasticsearch:](http://docker.elastic.co/elasticsearch/elasticsearch:){ELASTIC\_VERSION}" environment: - http.host=0.0.0.0 - transport.host=127.0.0.1 - bootstrap.memory\_lock=true - "ES\_JAVA\_OPTS=-Xms{ES\_JVM\_HEAP} -Xmx${ES\_JVM\_HEAP}"  
mem\_limit: {ES\_MEM\_LIMIT} ulimits: memlock: soft: -1 hard: -1 volumes: - ./config/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/elasticsearch.yml - esdata:/usr/share/elasticsearch/data #Port 9200 is available on the host ports: ['9200:9200'] #Healthcheck to confirm availability of ES. Other containers wait on this. healthcheck: test: ["CMD", "curl","-s" ,"-f", "-u", "elastic:{ES\_PASSWORD}", "[http://localhost:9200/\_cat/health](http://localhost:9200/_cat/health)"]  
#Internal network for the containers  
networks: ['stack']  
kibana:  
container\_name: kibana  
hostname: kibana  
image: "[docker.elastic.co/kibana/kibana:](http://docker.elastic.co/kibana/kibana:){ELASTIC\_VERSION}" volumes: - ./config/kibana/kibana.yml:/usr/share/kibana/kibana.yml #Port 5601 accessible on the host ports: ['5601:5601'] networks: ['stack'] #We don't start Kibana until the ES instance is ready depends\_on: ['elasticsearch'] environment: - "ELASTICSEARCH\_PASSWORD={ES\_PASSWORD}"  
healthcheck:  
test: ["CMD", "curl", "-s", "-f", "[http://localhost:5601/login](http://localhost:5601/login)"]  
retries: 6  
#Logstash container  
logstash:  
container\_name: logstash  
hostname: logstash  
image: "[docker.elastic.co/logstash/logstash:](http://docker.elastic.co/logstash/logstash:){ELASTIC\_VERSION}" volumes: - ./config/logstash/logstash.yml:/usr/share/logstash/config/logstash.yml:ro - ./config/logstash/pipeline:/usr/share/logstash/pipeline/ - ./.kitchen/logs:/logs #Port 5044 accessible on the host ports: ['5044:5044'] networks: ['stack'] depends\_on: elasticsearch: { condition: service\_healthy } healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9600"] retries: 10 interval: 10s #Filebeat container filebeat: container\_name: filebeat hostname: filebeat user: root image: "docker.elastic.co/beats/filebeat:{ELASTIC\_VERSION}"  
volumes:  
#Mount the filebeast configuration so users can make edit  
- ./config/beats/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml  
#Mount the prospectors directory. Users can in turn add propspectors to this directory and they will be dynamically loaded  
#- ./config/beats/filebeat/prospectors.d/:/usr/share/filebeat/prospectors.d/  
#Mount the test volume  
- ./.kitchen/logs/:/usr/share/filebeat/logs/  
#Named volume fsdata. This is used to persist the registry file between restarts, so to avoid data duplication  
- fbdata:/usr/share/filebeat/data/  
networks: ['stack']  
command: filebeat -e -strict.perms=false  
restart: on-failure  
depends\_on:  
#wait for the these services to come up. This ensures the logs are available and logstash exists for indexing  
logstash: { condition: service\_healthy }

## #Configure Stack container. This short lived container configures the stack once Elasticsearch is available. #More specifically, using a script it sets passwords and sets a default index pattern. configure\_stack: container\_name: configure\_stack image: [docker.elastic.co/beats/metricbeat:](http://docker.elastic.co/beats/metricbeat:){ELASTIC\_VERSION} volumes: ['./init/configure-stack.sh:/usr/local/bin/configure-stack.sh:ro'] command: ['/bin/bash', '-c', 'cat /usr/local/bin/configure-stack.sh | tr -d "\r" | bash'] networks: ['stack'] environment: ['ELASTIC\_VERSION={ELASTIC\_VERSION}','ES\_PASSWORD={ES\_PASSWORD}','DEFAULT\_INDEX\_PATTERN={DEFAULT\_INDEX\_PATTERN}'] depends\_on: ['elasticsearch','kibana'] volumes: #Es data esdata: driver: local #Filebeat data i.e. registry file fbdata: driver: local networks: {stack: {}}

and this is the pipeline (conf file) I'm trying to run with docker logstash :

* * *

input{  
beats{  
port =\> "5044"  
}  
}

filter{  
grok  
{  
match =\> {"message" =\> "%{TIMESTAMP\_ISO8601:logdate1} %{LOGLEVEL} %{DATA:thread\_name} %{SPACE} %{DATA:class\_name}:%{GREEDYDATA}"}

```
	}
	if ("_grokparsefailure" in [tags]) {

		drop {}
    	}		
	else{
	
		ruby {
				code => "
				event.set('logdate2', event.get('logdate1'))"
		}

		date {
				match => ["logdate2", "ISO8601"]
				target => "logdate2"
		}

		date_formatter {
				source => "logdate2"
				target => "formatted_date1"
				pattern => "yyyy-MM-dd"
    		}

		date {   		
				match => ["logdate1" , "ISO8601", "YYYY-MM-dd HH:mm:ss,SSS Z"]
				target => "@timestamp"
		}
	 }
	 
	 

    grok {
	
		match => ["[log][file][path]", "%{GREEDYDATA:path}/%{GREEDYDATA:metadata}"]
	}
	if ("_grokparsefailure" in [tags]) {

		mutate {
			add_tag => ["error_linux"]
			remove_tag => ["_grokparsefailure"]  
		}

    }	
	else{
	
    }

    
    grok {
	
		match => ["[log][file][path]", "%{GREEDYDATA:path}\\%{GREEDYDATA:metadata}"]
	}
	if ("_grokparsefailure" in [tags]) {

		mutate {
            add_tag => ["error_windows"]
			remove_tag => ["_grokparsefailure"]  
	    }
    }	
	else{
		    
    }

    mutate { 
			    remove_field => "[log][file][path]"  
    }

	grok 
	{
		match => {"metadata" => "%{GREEDYDATA:process_name}-%{NONNEGINT:run_ID}-%{WORD:output_type}(?<message_metadata>%{NUMBER}-%{GREEDYDATA})"}
	}
	if ("_grokparsefailure" in [tags]) {   
		
		mutate {
			#add_tag => ["error3"]
			remove_tag => ["_grokparsefailure"]  
		}
	}
   	else{
		
	}

	grok {
		
		match => {"metadata" => "%{GREEDYDATA:process_name}-%{NONNEGINT:run_ID}-%{WORD:output_type} %{SPACE}-%{GREEDYDATA}"}		
	}
	if ("_grokparsefailure" in [tags]){
				
		mutate {						
			
			#add_tag => ["error4"]
			remove_tag => ["_grokparsefailure"]
		}	
	}
	else{
		mutate {	
			#remove_tag =>["error3"]				
		}
	}

	mutate { 
		
		remove_field => ["[agent][type]","[agent][version]","[ecs][version]","[host][architecture]","[host][containerized]","[host][hostname]","[host][id]","[host][name]","[host][os][codename]","[host][os][family]","[host][os][kernel]","[host][os][name]","[host][os][platform]","[host][os][version]","[input][type]","[log][offset]","[agent][ephemeral_id]","[agent][hostname]","[agent][id]"]  
	}

```

}

output {

```
elasticsearch {
    	hosts => ['elasticsearch']
	user => 'elastic'
		password => 'changeme'
	index => ["hs_index_full"]
	document_type => "default"
    }
    #pipeline{send_to => ["part2"]}
#pipeline{send_to => ["part3"]}

```

## }

So, my question are the followings:

1. How I could integrate multipipelines with in ELk with docker within the docker-compose file. Basically, how they works the mounted paths in the case of multipipelines, and what considerations I should have in order to do it well? (I have been trying this during all the week).

2. Why the pipeline I mentioned before does not run well with docker ELK if it worked perfectly in the normal ELK suit. I'm figuring out maybe a plugin is stopping it to work properly. If so, how i could add a plugin of logstash or kibanna when i'm using Docoker.

(Please dont direct my to the documentation of the elasticsearch official page, because I having trying with it and I think is not really clear). But please help me! I having trying a lot in this aspect.

Thanks so much,

JUAN DAVID BRICENO GUERRERO  
MASTER STUDENT IN SUSTAINABLE INDUSTRIAL ENGINEERING

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [August 21, 2020, 5:05pm UTC](https://discuss.elastic.co/t/docker-elk-multipipelines-and-pipeline-error/245936/2 "2020-08-21T17:05:25Z")

</div>

I have been trying to even startup my own image for single logstash.

I doubt it that there are many people in this group uses docker.

I have no idea how docker/stack/compose etc.. works togather.

but this is what I did and my complex pipeline worked.

downloaded dockerfile form web  
created my own dirs (config, pipeline, bin)  
/root/docker\_logstash\_test/

dockerfile is pretty much same as found on elk's dockerfile

just added my config stuff in that docker

#Provide a minimal configuration, so that simple invocations will provide  
#Add all which you need to run this pipeline  
#logstash.yaml, pipeline.yaml, java.options  
#all config files, all sql file  
#a good experience.  
ADD config/jvm.options config/jvm.options  
ADD config/pipelines.yml config/pipelines.yml  
ADD config/logstash.yml config/logstash.yml  
ADD config/log4j2.properties config/

#Adding config, sql, jar file to make sure this runs ok  
ADD pipeline/sachin\_quick\_test.conf pipeline/  
ADD logstash-core/lib/jars/ojdbc8.jar logstash-core/lib/jars/  
ADD logstash-core/lib/jars/kafka-standalone-2.1.1.jar logstash-core/lib/jars/

created docker image  
#docker build --tag logstash:7.9.0 --network=host --build-arg https\_proxy=https://xx.xx.xx.xx:443 -f- ./ \< Dockerfile

I know this is not what you looking for but just putting it out incase even one line helps you.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 18, 2020, 5:05pm UTC](https://discuss.elastic.co/t/docker-elk-multipipelines-and-pipeline-error/245936/3 "2020-09-18T17:05:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.