# Can someone please help me with nginx logstash filter

**URL:** <https://discuss.elastic.co/t/can-someone-please-help-me-with-nginx-logstash-filter/98377>\
**Category:** Logstash\
**Created:** [August 25, 2017, 1:56pm UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-nginx-logstash-filter/98377 "2017-08-25T13:56:18Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [August 25, 2017, 1:56pm UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-nginx-logstash-filter/98377/1 "2017-08-25T13:56:18Z")

</div>

Hi Guys,

Can someone please help me to identify the issue with this filter, I am trying to parse the nginx logs with file input and getting below error.

can somone please help me to rectify the error?

file {  
{  
type =\> nginx\_web  
path =\> ["/var/nginx/_"]  
exclude =\> ["_.gz"]  
}  
}

filter {  
grok {  
match =\> ["message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra\_fields}"]  
overwrite =\> ["message"]  
}

mutate {  
convert =\> ["response", "integer"]  
convert =\> ["bytes", "integer"]  
convert =\> ["responsetime", "float"]  
}

geoip {  
source =\> "clientip"  
target =\> "geoip"  
add\_tag =\> ["nginx-geoip"]  
}

date {  
match =\> ["timestamp" , "dd/MMM/YYYY:HH:mm:ss Z"]  
remove\_field =\> ["timestamp"]  
}

useragent {  
source =\> "agent"  
}  
}

output {  
elasticsearch {  
manage\_template =\> false  
hosts =\> "192.168.5.15:9200"  
index =\> "nginx-%{+YYYY.MM.dd}"

}  
stdout {  
codec =\> "rubydebug"  
}

}

And here is the error

[root@elk5 /etc/logstash/conf.d-ELK5.x]# /usr/share/logstash/bin/logstash -f newnginx.conf  
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.  
WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults  
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console  
19:05:30.001 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=\>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after "}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 25, 2017, 2:07pm UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-nginx-logstash-filter/98377/2 "2017-08-25T14:07:26Z")

</div>

Your file input needs to be wrapped in `input { ... }`.

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [August 25, 2017, 2:47pm UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-nginx-logstash-filter/98377/3 "2017-08-25T14:47:39Z")

</div>

hmm...any idea how to decode this one ☹

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.  
WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults  
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console  
20:13:09.859 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=\>"Expected one of #, =\> at line 11, column 10 (byte 154) after filter {\n grok {\n match "}

input {  
file {  
path =\> "/var/log/\*.log"  
start\_position =\> "beginning"  
}

}

filter {  
grok {  
match =\> ["message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra\_fields}"]  
overwrite =\> ["message"]  
}

mutate {  
convert =\> ["response", "integer"]  
convert =\> ["bytes", "integer"]  
convert =\> ["responsetime", "float"]  
}

geoip {  
source =\> "clientip"  
target =\> "geoip"  
add\_tag =\> ["nginx-geoip"]  
}

date {  
match =\> ["timestamp" , "dd/MMM/YYYY:HH:mm:ss Z"]  
remove\_field =\> ["timestamp"]  
}

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [August 25, 2017, 2:56pm UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-nginx-logstash-filter/98377/4 "2017-08-25T14:56:07Z")

</div>

Seems I got the issue and fixed; I am processing the logs...

Lets see how the stuff goes 🙂

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [August 30, 2017, 6:04pm UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-nginx-logstash-filter/98377/5 "2017-08-30T18:04:03Z")

</div>

Hi Guys,

I am really struggling with Nginx logs and after being spent almost more than 2 weeks I am running out of ideas.

Can someone plssss help indexing nginx logs! Dang stuff is not working out for me ☹

I would really appreciate if someone can provide me the ideal nginx logstash config file..plsss

---

<div class="post-metadata">

**Author:** ![pjanzen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pjanzen/32/13756_2.png) [@pjanzen](https://discuss.elastic.co/u/pjanzen)\
**Post date:** [August 30, 2017, 6:53pm UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-nginx-logstash-filter/98377/6 "2017-08-30T18:53:24Z")

</div>

I cleaned up / corrected you conf. There where some { misplaced.

```
input {
	file {
	
		type => nginx_web
		path => ["/var/nginx/"]
		exclude => [".gz"]
	}
}

filter {
	grok {
		match => [“message” , “%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}”]
		overwrite => [“message”]
	}

	mutate {
		convert => [“response”, “integer”]
		convert => [“bytes”, “integer”]
		convert => [“responsetime”, “float”]
	}

	geoip {
		source => "clientip"
		target => "geoip"
		add_tag => [“nginx-geoip”]
	}

	date {
		match => [“timestamp” , “dd/MMM/YYYY:HH:mm:ss Z”]
		remove_field => [“timestamp”]
	}	

	useragent {
		source => “agent”
	}
}

output {
	elasticsearch {
		manage_template => false
		hosts => "192.168.5.15:9200"
		index => “nginx-%{+YYYY.MM.dd}”
	}
	stdout {
		codec => “rubydebug”
	}
}
```

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [August 31, 2017, 4:24am UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-nginx-logstash-filter/98377/7 "2017-08-31T04:24:18Z")

</div>

Great thanks let me run it and let you know my results 🙂

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [August 31, 2017, 4:41am UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-nginx-logstash-filter/98377/8 "2017-08-31T04:41:04Z")

</div>

> [@pjanzen](#):
>
> input {  
> file {
> 
> ```
> type =&gt; nginx_web
> path =&gt; ["/var/nginx/"]
> exclude =&gt; [".gz"]
> }
> 
> ```
> 
> }
> 
> filter {  
> grok {  
> match =\> [“message” , “%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra\_fields}”]  
> overwrite =\> [“message”]  
> }
> 
> ```
> mutate {
> convert =&gt; [“response”, “integer”]
> convert =&gt; [“bytes”, “integer”]
> convert =&gt; [“responsetime”, “float”]
> }
> 
> geoip {
> source =&gt; "clientip"
> target =&gt; "geoip"
> add_tag =&gt; [“nginx-geoip”]
> }
> 
> date {
> match =&gt; [“timestamp” , “dd/MMM/YYYY:HH:mm:ss Z”]
> remove_field =&gt; [“timestamp”]
> }	
> 
> useragent {
> source =&gt; “agent”
> }
> 
> ```
> 
> }
> 
> output {  
> elasticsearch {  
> manage\_template =\> false  
> hosts =\> "192.168.5.15:9200"  
> index =\> “nginx-%{+YYYY.MM.dd}”  
> }  
> stdout {  
> codec =\> “rubydebug”  
> }  
> }

I am accepting files using filebeat hope this is ok? That is filebeat is installed on nginx server

---

<div class="post-metadata">

**Author:** ![pjanzen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pjanzen/32/13756_2.png) [@pjanzen](https://discuss.elastic.co/u/pjanzen)\
**Post date:** [August 31, 2017, 1:07pm UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-nginx-logstash-filter/98377/9 "2017-08-31T13:07:02Z")

</div>

I am unsure what you're asking here. I do not know what config filebeat accepts but this config is for logstash.

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [September 1, 2017, 2:45am UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-nginx-logstash-filter/98377/10 "2017-09-01T02:45:07Z")

</div>

Well I am confused about accepting messages whether that should be through filebeat or rsyslog directly sending to logstash.

Which is the most feasible way per you?

---

<div class="post-metadata">

**Author:** ![pjanzen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pjanzen/32/13756_2.png) [@pjanzen](https://discuss.elastic.co/u/pjanzen)\
**Post date:** [September 2, 2017, 1:00pm UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-nginx-logstash-filter/98377/11 "2017-09-02T13:00:54Z")

</div>

When I can I use rsyslog / syslog-ng to send the data over to logstash, if an application is not able to use a syslog variant I use filebeat to parse the logs and send it to logstash.

Does that help?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 30, 2017, 1:01pm UTC](https://discuss.elastic.co/t/can-someone-please-help-me-with-nginx-logstash-filter/98377/12 "2017-09-30T13:01:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.