runtman
June 28, 2017, 1:02pm
1
Hello, so I want to visualise some of the Nginx logs we have with the geoip source.
So I have the following in Filebeat configurations to push the logs
- input_type: log
paths:
- /var/log/nginx/dev/access.log
document_type: nginx_dev_access
Then in my nginx filter, I have the following GROK and filters:
filter {
if [type] == "nginx_dev_access" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}
}
}
If I go to the "Discover" home screen and do a search, I can see logs with these filters (image attached)
I see the available fields, they appear to be mark as unknown (the small ?)
So if I save this search and go to plot a map in visulations, I only have these options.
How do I get my custom filters to appear in that list?
It may be worth mentioning, when this ELK stack was built the default filebeats dashboards were adding thinking they would work, they do not as it goes through Logstash first.
runtman
June 28, 2017, 1:32pm
2
Following other threads, I have reduced the filter to just this:
geoip {
source => "clientip"
}
I can again see data under geoip.ip but my drop down only has the 4 options, I'm assuming loading some template has meddled with that drop down box?
I have noticed the mapping is set to "keyword"
root@ip-172-31-25-192:/var/log# curl -u elastic:changeme http://172.31.18.116:9200/filebeat*/_mapping/nginx_dev_access/field/geoip.ip?pretty
{
"filebeat-2017.06.28" : {
"mappings" : {
"nginx_dev_access" : {
"geoip.ip" : {
"full_name" : "geoip.ip",
"mapping" : {
"ip" : {
"type" : "keyword",
"ignore_above" : 1024
}
}
}
}
}
}
}
This needs to be geo_point I am guessing? Even if I empty logstash and elastic search ,load the default filebeat template. This data comes back as "keyword" is there anyway I can inforce it always be geo_point
warkolm
June 29, 2017, 10:15am
4
What's the mapping for the geoip.location field, not the geoip.ip field.
runtman
June 30, 2017, 9:53am
5
Thanks for the reply, interestingly location is empty.
root@ip-172-31-18-116:~# curl -u elastic:changeme http://172.31.18.116:9200/filebeat-2017.06.30/_mapping/nginx.access/field/geoip.location?pretty
{ }
root@ip-172-31-18-116:~# curl -u elastic:changeme http://172.31.18.116:9200/filebeat-2017.06.30/_mapping/nginx.access/field/geoip.ip?pretty
{
"filebeat-2017.06.30" : {
"mappings" : {
"nginx.access" : {
"geoip.ip" : {
"full_name" : "geoip.ip",
"mapping" : {
"ip" : {
"type" : "keyword",
"ignore_above" : 1024
}
}
}
}
}
}
}
However, these exist:
root@ip-172-31-18-116:~# curl -u elastic:changeme http://172.31.18.116:9200/filebeat-2017.06.30/_mapping/nginx.access/field/geoip.location.lon?pretty
{
"filebeat-2017.06.30" : {
"mappings" : {
"nginx.access" : {
"geoip.location.lon" : {
"full_name" : "geoip.location.lon",
"mapping" : {
"lon" : {
"type" : "float"
}
}
}
}
}
}
}
root@ip-172-31-18-116:~# curl -u elastic:changeme http://172.31.18.116:9200/filebeat-2017.06.30/_mapping/nginx.access/field/geoip.location.lat?pretty
{
"filebeat-2017.06.30" : {
"mappings" : {
"nginx.access" : {
"geoip.location.lat" : {
"full_name" : "geoip.location.lat",
"mapping" : {
"lat" : {
"type" : "float"
}
}
}
}
}
}
}
Note the _mapping change is on purpose, we were tidying up some of the log inputs.
warkolm
June 30, 2017, 10:03am
6
What's your Logstash config look like, and your Elasticsearch mapping, for the entire index.
runtman
June 30, 2017, 10:21am
7
Here is the logstash config:
filter {
if [type] == "nginx.access" {
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
}
geoip {
source => "clientip"
}
}
if [type] == "nginx.error" {
grok {
match => { "message" => "%{HTTPD20_ERRORLOG}" }
}
geoip {
source => "clientip"
}
}
if [type] == "nginx" {
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
}
geoip {
source => "clientip"
}
}
}
Ref the elastic search mapping, how should I send this wouldn't it be too much to paste?
I put it in pastebin, I hope that is ok:
https://pastebin.com/uz6VL2bw
warkolm
June 30, 2017, 10:39am
8
Line 56-58 looks right. What does that look like in the index pattern settings in Kibana?
runtman
June 30, 2017, 10:42am
9
Is this what you are after?
warkolm
June 30, 2017, 10:45am
10
Yep, but is there something for geoip.location?
runtman
June 30, 2017, 10:46am
11
Here is the search for that.
warkolm
June 30, 2017, 10:47am
12
Can you do a field refresh?geoip.location, but it's not there which is odd.
warkolm
June 30, 2017, 10:58am
14
Then there must be something else in your config working against this?
runtman
June 30, 2017, 11:02am
15
Which configs?
Here is output of all logstash outputs
filter {
if [type] == "syslog" {
if [message] =~ /last message repeated [0-9]+ times/ {
drop { }
}
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
filter {
if [type] == "nginx.access" {
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
}
geoip {
source => "clientip"
}
}
if [type] == "nginx.error" {
grok {
match => { "message" => "%{HTTPD20_ERRORLOG}" }
}
geoip {
source => "clientip"
}
}
if [type] == "nginx" {
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
}
geoip {
source => "clientip"
}
}
}
filter {
if [type] == "apache" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
}
filter {
if [type] == "drupal" {
grok {
match => ["message", "%{SYSLOGTIMESTAMP} %{HOSTNAME} %{WORD}: %{URI:drupal_vhost}\|%{NUMBER:drupal_timestamp}\|(?<drupal_action>[^\|]*)\|%{IP:drupal_ip}\|(?<drupal_request_uri>[^\|]*)\|(?<drupal_referer>[^\|]*)\|(?<drupal_uid>[^\|]*)\|(?<drupal_link>[^\|]*)\|%{GREEDYDATA:drupal_message}" ]
}
}
}
output {
elasticsearch {
hosts => ["172.31.18.116:9200", "172.31.28.55:9200", "172.31.23.120:9200", "172.31.27.44:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
user => elastic
password => changeme
}
}
runtman
July 3, 2017, 7:15am
16
I just deleted packet/metric indices, ran the templates to ES and started the indexing again and the map doesn't work for either of those two either. I installed the GEO plugin to ES only have I missed a step somewhere perhaps?
I have just done a fresh install of the entire stack, without packet/metric. Ensuring ES has the filebeat template before it has any data. Removing all filters other than ES output and nginx input and I still don't have any plots on the map.
I'm out of ideas
runtman
July 5, 2017, 7:31am
17
Can anyone assist with this, perhaps an Elastic member?
warkolm
July 15, 2017, 10:29pm
18
Is this still not working for you?
runtman
July 16, 2017, 1:22pm
19
Nope, I have not been actively trouble shooting but this isn't working still.
I am having the same issues however with iis logs. I have tried numerous ways to get my logs to show with geoip.location so I can visualize it. I am using version 5.5.
I have had the best luck using
geoip {
source=>"iis_client_ip" add_tag=>["geoip"]
}
for my iis logging. But I am not able to visualize anything even though I can see location information.
