# Logstash forwader and indexer

**URL:** <https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844>\
**Category:** Logstash\
**Created:** [March 14, 2018, 6:40am UTC](https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844 "2018-03-14T06:40:09Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gaurav\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/g/9fc29f/32.png) [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\
**Post date:** [March 14, 2018, 6:40am UTC](https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844/1 "2018-03-14T06:40:09Z")

</div>

Hi  
Wanted to know if within the logstash output if I want to put another filter is it possible.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 14, 2018, 6:51am UTC](https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844/2 "2018-03-14T06:51:55Z")

</div>

The output section can only contain outputs. Filters go in the filter section.

If you ask the real, underlying question instead you'll get more relevant help.

---

<div class="post-metadata">

**Author:** ![Gaurav\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/g/9fc29f/32.png) [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\
**Post date:** [March 14, 2018, 7:07am UTC](https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844/3 "2018-03-14T07:07:45Z")

</div>

actually I have a logstash forwader and a logstash indexer at seperate locations or say vm but now I am trying to combine both the forwaders and indexer as one so that the flow of my code be like

input{  
path}  
filter{  
match (path)  
}  
output  
{  
filter(match for indexer)  
}  
elasticsearch output}}

is this possible

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 14, 2018, 9:43am UTC](https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844/4 "2018-03-14T09:43:04Z")

</div>

If you want to apply different filters for different events you can use conditionals.

[https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html)

---

<div class="post-metadata">

**Author:** ![Gaurav\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/g/9fc29f/32.png) [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\
**Post date:** [March 14, 2018, 10:05am UTC](https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844/5 "2018-03-14T10:05:01Z")

</div>

I will elaborate my question a bit  
what I wanted is to combine my forwarder and indexer as one i.e as of now my forwarder runs on a different vm and sends it data to redis now from redis queue my indexer pulls the data.  
But now I want to combine the indexer and forwarder and I do not want redis in between that is what my question is. So now my config file will look like that its input plugin will have the input of the forwader the filter plugin will contain the filter of the forwarder and the output of the config file will have the filter and output of the indexer. So is it possible to do this.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 14, 2018, 11:56am UTC](https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844/6 "2018-03-14T11:56:42Z")

</div>

Yes. In fact, in sounds like a very simple Logstash setup with an input that injects events that are processed by filters and eventually sent to an output. I don't see the complication.

---

<div class="post-metadata">

**Author:** ![Gaurav\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/g/9fc29f/32.png) [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\
**Post date:** [March 14, 2018, 12:12pm UTC](https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844/7 "2018-03-14T12:12:59Z")

</div>

currently we have four forwarders for four different types of logs namely one for app logs one access logs one for others and one for status logs and similarly there are indexers,  
There are multiple types of apps and I want to feed in the data from all in one so the path extension becomes something like this pah =\> "remote/user/gaurav/cnt/app\_\*/.log to pull all the logs of the apps then have a filter running on the path and according to the he filter we put another filter to match the tags that is what I think of  
But it is not working for me

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 14, 2018, 1:45pm UTC](https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844/8 "2018-03-14T13:45:01Z")

</div>

So what does your configuration look like so far? What does the input look like? What does Logstash's output look like?

---

<div class="post-metadata">

**Author:** ![Gaurav\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/g/9fc29f/32.png) [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\
**Post date:** [March 15, 2018, 4:22am UTC](https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844/9 "2018-03-15T04:22:08Z")

</div>

input {  
file {  
path =\> "/..../..../logs/gaurav/03a/aa.log"

```
	exclude => [
		"access*.log*",
		"fe*.log*",
		"stat*.log*",
		"gc*.log*",
		"*dump*.log*"
	]
	sincedb_path => "/dev/null"
	type => "app"

```

codec =\> multiline {  
pattern =\> "^%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}"  
negate =\> true  
what =\> "previous"  
}  
}

}  
this is how input is

---

<div class="post-metadata">

**Author:** ![Gaurav\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/g/9fc29f/32.png) [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\
**Post date:** [March 15, 2018, 6:57am UTC](https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844/10 "2018-03-15T06:57:54Z")

</div>

[gagarwal3@nceapttls13]~/tmp% docker run -h logstash --name logstash --link elasticsearch:elasticsearch -it --rm -v "$PWD":/remote/backup/logs/APT/CINT logstash -f /remote/backup/logs/APT/CINT/new1.conf  
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties  
06:55:33.180 [main] INFO logstash.modules.scaffold - Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"}  
06:55:33.185 [main] INFO logstash.modules.scaffold - Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"}  
06:55:33.191 [main] INFO logstash.setting.writabledirectory - Creating directory {:setting=\>"path.queue", :path=\>"/var/lib/logstash/queue"}  
06:55:33.193 [main] INFO logstash.setting.writabledirectory - Creating directory {:setting=\>"path.dead\_letter\_queue", :path=\>"/var/lib/logstash/dead\_letter\_queue"}  
06:55:33.217 [LogStash::Runner] INFO logstash.agent - No persistent UUID file found. Generating new UUID {:uuid=\>"9d38957d-f52c-47af-814f-7f124621df51", :path=\>"/var/lib/logstash/uuid"}  
06:55:33.468 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=\>"Expected one of #, {, } at line 77, column 33 (byte 2110) after output {\r\nstdout{ codec =\> rubydebug}\r\n\tif [app] == "fom" or [app] == "sepl" {\r\n\t\telasticsearch {\r\n\t\t\thosts =\> "nceapttlnet:9200"\r\n\t\t\tindex =\> "apt-%{+YYYY.MM.dd}""}  
[gagarwal3@nceapttls13]~/tmp% docker run -h logstash --name logstash --link elasticsearch:elasticsearch -it --rm -v "$PWD":/remote/backup/logs/APT/CINT logstash -f /remote/backup/logs/APT/CINT/new1.conf  
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties  
06:56:31.309 [main] INFO logstash.modules.scaffold - Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"}  
06:56:31.314 [main] INFO logstash.modules.scaffold - Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"}  
06:56:31.321 [main] INFO logstash.setting.writabledirectory - Creating directory {:setting=\>"path.queue", :path=\>"/var/lib/logstash/queue"}  
06:56:31.322 [main] INFO logstash.setting.writabledirectory - Creating directory {:setting=\>"path.dead\_letter\_queue", :path=\>"/var/lib/logstash/dead\_letter\_queue"}  
06:56:31.352 [LogStash::Runner] INFO logstash.agent - No persistent UUID file found. Generating new UUID {:uuid=\>"426f7c96-9e43-48f9-828c-b3f898e09ddf", :path=\>"/var/lib/logstash/uuid"}  
06:56:32.511 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>[[http://nceanet:9200/](http://nceanet:9200/)]}}  
06:56:32.512 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://nceapttnet:9200/](http://nceapttnet:9200/), :path=\>"/"}  
06:56:32.619 [[main]-pipeline-manager] WARN logstash.outputs.elasticsearch - Restored connection to ES instance {:url=\>"[http://ncea.net:9200/](http://ncea.net:9200/)"}  
06:56:32.938 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Using mapping template from {:path=\>nil}  
06:56:32.942 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>50001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"\_all"=\>{"enabled"=\>true, "norms"=\>false}, "dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date", "include\_in\_all"=\>false}, "@version"=\>{"type"=\>"keyword", "include\_in\_all"=\>false}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}}  
06:56:32.961 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//nce.net:9200](https://nce.net:9200)"]}  
06:56:32.982 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>[[http://nceanet:9200/](http://nceanet:9200/)]}}  
06:56:32.983 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://nce.net:9200/](http://nce.net:9200/), :path=\>"/"}  
06:56:33.000 [[main]-pipeline-manager] WARN logstash.outputs.elasticsearch - Restored connection to ES instance {:url=\>"[http://nce.net:9200/](http://nce.net:9200/)"}  
06:56:33.008 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Using mapping template from {:path=\>nil}  
06:56:33.010 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>50001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"\_all"=\>{"enabled"=\>true, "norms"=\>false}, "dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date", "include\_in\_all"=\>false}, "@version"=\>{"type"=\>"keyword", "include\_in\_all"=\>false}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}}  
06:56:33.023 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//nce.net:9200](https://nce.net:9200)"]}  
06:56:33.157 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>5, "pipeline.max\_inflight"=\>500}  
06:56:33.466 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started  
06:56:33.575 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=\>9600}

can u let me know what is the error

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 15, 2018, 9:52am UTC](https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844/11 "2018-03-15T09:52:55Z")

</div>

> Cannot create pipeline {:reason=\>"Expected one of #, {, } at line 77, column 33 (byte 2110) after output {\r\nstdout{ codec =\> rubydebug}\r\n\tif [app] == "fom" or [app] == "sepl" {\r\n\t\telasticsearch {\r\n\t\t\thosts =\> "nceapttlnet:9200"\r\n\t\t\tindex =\> "apt-%{+YYYY.MM.dd}""}

You haven't showed us that part of the configuration so we can't help.

---

<div class="post-metadata">

**Author:** ![Gaurav\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/g/9fc29f/32.png) [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\
**Post date:** [March 15, 2018, 9:55am UTC](https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844/12 "2018-03-15T09:55:20Z")

</div>

input {  
file {  
path =\> "/remote/backup/logs/APT/CINT/httpd/\*"

```
	exclude => [
		"access*.log*",
		"fe*.log*",
		"stat*.log*",
		"gc*.log*",
		"*dump*.log*"
	]
	sincedb_path => "/dev/null"
	type => "app"

```

codec =\> multiline {  
pattern =\> "^%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}"  
negate =\> true  
what =\> "previous"  
}  
}

}  
filter {

```
grok {
	match => {
		"path" => "/logs/(?<env>[^/]+)/(?<instance>[^/]+)/[^/]+"
	}
	overwrite => ["host"]
}

```

if [type] == "web\_access" {  
grok {  
match =\> {  
"message" =\> "%{APT\_APACHE\_ACCESS:logline}"  
}  
named\_captures\_only =\> false  
patterns\_dir =\> "/remote/backup/logs/APT/CINT/pattern"  
break\_on\_match =\> true  
remove\_field =\> ["message", "HOUR", "MINUTE", "MONTH", "MONTHDAY", "SECOND", "TIME", "YEAR"]  
overwrite =\> ["port"]  
}  
}

```
if [type] == "app" {
	grok {
		match => {
			# "message" => "%{APT_TIMESTAMP:timestamp} \| %{HOSTNAME:hostname} \| %{DATA:application} \| %{APT_LOGLEVEL:loglevel} *\| 1-%{DATA:thread:int} \| %{DATA:class} *\| %{DATA:correlationId} \| %{GREEDYDATA:msg}"
			"message" => "%{APT_TIMESTAMP:timestamp} \| \S+ \| \S+ \| %{APT_LOGLEVEL:loglevel} *\| 1-%{DATA:thread:int} \| %{DATA:class} *\| %{DATA:correlationId} \| %{GREEDYDATA:msg}"
		}
		patterns_dir => "/remote/backup/logs/APT/CINT/pattern"
		remove_field => ["message"]
	}
}

if "_grokparsefailure" not in [tags] {
    date {
        match => ["timestamp", "YYYY/MM/dd HH:mm:ss.SSS", "YYYY/MM/dd HH:mm:ss,SSS", "dd/MMM/YYYY:HH:mm:ss +0000", "EEE MMM dd HH:mm:ss YYYY"]
        timezone => "UTC"
	}
	if [type] == "web_access" or [type] == "web_error" {
        mutate {
	        gsub => [
                "referrer","\"","",
                "agent","\"","",
                "JSESSIONID","\"","",
                "APT_SESSIONID","\"","",
                "correlationId","\"","",
                "transactionOriginator","\"","",
                "customerId","\"",""
            ]
	        remove_field => ["logline", "timestamp", "BASE10NUM", "INT", "HOSTNAME", "IPV4", "day", "month", "monthday", "time", "year"]
		}
	}

}

```

}

output {  
stdout{ codec =\> rubydebug}  
if [app] == "fom" or [app] == "sepl" {  
elasticsearch {  
hosts =\> "nce13"  
index =\> "apt-%{+YYYY.MM.dd}"

```
	}
}
else {
	elasticsearch {
		hosts => "nce13"
		index => "apt-%{+YYYY.MM.dd}"
        
}

	
        

}

```

}

This is how the whole config file looks like

---

<div class="post-metadata">

**Author:** ![Gaurav\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/g/9fc29f/32.png) [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\
**Post date:** [March 15, 2018, 9:58am UTC](https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844/13 "2018-03-15T09:58:33Z")

</div>

[gagarwal3@nceapttls13]~/tmp% docker run -h logstash --name logstash --link elasticsearch:elasticsearch -it --rm -v "$PWD":/remote/backup/logs/APT/CINT logstash -f /remote/backup/logs/APT/CINT/new1.conf  
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties  
06:56:31.309 [main] INFO logstash.modules.scaffold - Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"}  
06:56:31.314 [main] INFO logstash.modules.scaffold - Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"}  
06:56:31.321 [main] INFO logstash.setting.writabledirectory - Creating directory {:setting=\>"path.queue", :path=\>"/var/lib/logstash/queue"}  
06:56:31.322 [main] INFO logstash.setting.writabledirectory - Creating directory {:setting=\>"path.dead\_letter\_queue", :path=\>"/var/lib/logstash/dead\_letter\_queue"}  
06:56:31.352 [LogStash::Runner] INFO logstash.agent - No persistent UUID file found. Generating new UUID {:uuid=\>"426f7c96-9e43-48f9-828c-b3f898e09ddf", :path=\>"/var/lib/logstash/uuid"}  
06:56:32.511 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>[[http://nceanet:9200/](http://nceanet:9200/)]}}  
06:56:32.512 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://nceapttnet:9200/](http://nceapttnet:9200/), :path=\>"/"}  
06:56:32.619 [[main]-pipeline-manager] WARN logstash.outputs.elasticsearch - Restored connection to ES instance {:url=\>"[http://ncea.net:9200/](http://ncea.net:9200/)"}  
06:56:32.938 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Using mapping template from {:path=\>nil}  
06:56:32.942 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Attempting to install template {:manage\_template=\>{"template"=\>"logstash-", "version"=\>50001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"\_all"=\>{"enabled"=\>true, "norms"=\>false}, "dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date", "include\_in\_all"=\>false}, "@version"=\>{"type"=\>"keyword", "include\_in\_all"=\>false}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}}  
06:56:32.961 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//nce.net:9200](https://nce.net:9200)"]}  
06:56:32.982 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>[[http://nceanet:9200/](http://nceanet:9200/)]}}  
06:56:32.983 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://nce.net:9200/](http://nce.net:9200/), :path=\>"/"}  
06:56:33.000 [[main]-pipeline-manager] WARN logstash.outputs.elasticsearch - Restored connection to ES instance {:url=\>"[http://nce.net:9200/](http://nce.net:9200/)"}  
06:56:33.008 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Using mapping template from {:path=\>nil}  
06:56:33.010 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Attempting to install template {:manage\_template=\>{"template"=\>"logstash-", "version"=\>50001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"\_all"=\>{"enabled"=\>true, "norms"=\>false}, "dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date", "include\_in\_all"=\>false}, "@version"=\>{"type"=\>"keyword", "include\_in\_all"=\>false}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}}  
06:56:33.023 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//nce.net:9200](https://nce.net:9200)"]}  
06:56:33.157 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>5, "pipeline.max\_inflight"=\>500}  
06:56:33.466 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started  
06:56:33.575 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=\>9600}

can u let me know what is the error

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 15, 2018, 10:29am UTC](https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844/14 "2018-03-15T10:29:37Z")

</div>

Okay, now the error message is gone. So Logstash still isn't working as expected? In what way?

---

<div class="post-metadata">

**Author:** ![Gaurav\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/g/9fc29f/32.png) [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\
**Post date:** [March 15, 2018, 11:31am UTC](https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844/15 "2018-03-15T11:31:07Z")

</div>

it is not pulling any data as expected I guess

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 15, 2018, 2:01pm UTC](https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844/16 "2018-03-15T14:01:40Z")

</div>

Typo in the filename pattern in the file input? Or does the user Logstash runs as lack permissions to access the files?

If you increase the log level you can see what files (if any) the filename pattern expands to.

---

<div class="post-metadata">

**Author:** ![Gaurav\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/g/9fc29f/32.png) [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\
**Post date:** [March 15, 2018, 2:17pm UTC](https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844/17 "2018-03-15T14:17:53Z")

</div>

I hosted it on docker and made a compose file with few changes it is working fine

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 12, 2018, 2:18pm UTC](https://discuss.elastic.co/t/logstash-forwader-and-indexer/123844/18 "2018-04-12T14:18:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.