Geoip no longer working

Elastic Stack Logstash
1

Hi, all.

Reecently, I moved from logstash-forwarder to filebeat on my linux nodes. Prior to the change, I had geoip working well. This is the stanza in logstash-forwarder.conf which, along with the input and filter configs on my logstash server made geoip work:

{
  "paths": [ "/var/log/httpd/access_log" ],
  "fields": { "type": "apache-access" }
}

Once I moved to filebeat, I added all of the required information in filebeat.yml. However, I see no equivalent to the above logstash-forwarder stanza. And, at any rate, geoip doesn't work. Can someone help me get back to good?

Thanks.

2

What does your Filebeat configuration look like? Make sure you format it as code with the </> button on the toolbar. What do the resulting Logstash events look like?

3

Hi, Magnus.

The filebeat config file is located here: http://pastebin.com/D0kgZMAZ. Sorry it's so long; I've left in all of the comments from the original.

As to current output, here's what I get from kibana (what other way is their to get output?):

February 22nd 2016, 13:48:38.761 message:Feb 22 13:48:35 satest02 sudo: dyioulos : TTY=pts/1 ; PWD=/home/dyioulos ; USER=root ; COMMAND=/bin/bash beat.hostname:satest02.netatlantic.com beat.name:satest02.netatlantic.com host:satest02.netatlantic.com @version:1 @timestamp:February 22nd 2016, 13:48:38.761 count:1 fields: - input_type:log offset:4125 source:/var/log/secure type:syslog tags:beats_input_codec_plain_applied, syslog _id:AVMKTr1fqUdC8LLBFXTb _type:syslog _index:logstash-2016.02.22 _score:

Finally, what </> button? I'm accessing the data via Kibana.

Diggy

4

The filebeat config file is located here: http://pastebin.com/D0kgZMAZ. Sorry it's so long; I've left in all of the comments from the original.

Nowhere in this file are you setting the type field to "apache-access" like you've been doing previously in logstash-forwarder. Look at the document_type option.

As to current output, here's what I get from kibana (what other way is their to get output?):

A stdout { codec => rubydebug } output on the Logstash side is a good way of dumping exactly what Logstash sees and avoiding the complexity of ES and Kibana.

Finally, what </> button? I'm accessing the data via Kibana.

The toolbar in the discuss.elastic.co interface.

5

OK. First, because my config file is based on the stock/default one, I'm probably better off using just the declarations I need. My paths are listed thusly:

  paths:
    - /var/log/secure
    - /var/log/messages
    - /var/log/httpd/access_log
    - /var/log/yum.log

and my document_type is specified further down as simply:

  document_type: syslog

when what I should be doing is:

  paths:
    - /var/log/secure
   document_type:  syslog
    - /var/log/messages
   document_type:  syslog
    - /var/log/httpd/access_log
   document_type:  apache_access
    - /var/log/yum.log
   document_type:  NOT SURE ABOUT THIS ONE   <--- Do you know, or can you suggest, type for yum?

Is that correct?

And, "doh" on the </> button!

Diggy

6

You need to use different prospectors. See Configure inputs | Filebeat Reference [8.11] | Elastic.

prospectors:
  - paths:
      - /var/log/secure
      - /var/log/messages
    document_type: syslog
  - paths:
      - /var/log/httpd/access_log
    document_type: apache_access
  ...

Also, the indentation is off in your example. YAML is sensitive to indentation. When in doubt, use a YAML validator.

Do you know, or can you suggest, type for yum?

You can pick any string you want as long as it matches the Logstash filters that you use to parse those events.

7

OK. I believe I've now set up my filebeat.conf file properly. However, it doesn't look like the output has changed much:

February 22nd 2016, 15:43:05.396 beat.hostname:satest02.netatlantic.com beat.name:satest02.netatlantic.com host:satest02.netatlantic.com message:173.9.34.198 - - [22/Feb/2016:15:42:58 -0500] "GET / HTTP/1.1" 403 4961 "-" "Mozilla/5.0 (Android 5.0; Mobile; rv:44.0) Gecko/44.0 Firefox/44.0" @version:1 @timestamp:February 22nd 2016, 15:43:05.396 count:1 fields: - input_type:log offset:1401 source:/var/log/httpd/access_log type:access_log tags:beats_input_codec_plain_applied _id:AVMKt5o5qUdC8LLBFXZM _type:access_log _index:logstash-2016.02.22 _score:

Whereas, this is what I got when using logstash-forwarder:

January 24th 2016, 16:34:18.000 host:satest02.netatlantic.com type:apache-access file:/var/log/httpd/access_log offset:2456 line:141.212.122.112 - - [24/Jan/2016:16:34:18 -0500] "GET /x HTTP/1.1" 400 316 "-" "Telesphoreo" @version:1 @timestamp:January 24th 2016, 16:34:18.000 clientip:141.212.122.112 ident:- auth:- timestamp:24/Jan/2016:16:34:18 -0500 verb:GET request:/x httpversion:1.1 response:400 bytes:316 referrer:"-" agent:"Telesphoreo" geoip.ip:141.212.122.112 geoip.country_code2:US geoip.country_code3:USA geoip.country_name:United States geoip.continent_code:NA geoip.region_name:MI geoip.city_name:Ann Arbor geoip.postal_code:48109 geoip.latitude:42.292 geoip.longitude:-83.714 geoip.dma_code:505 geoip.area_code:734 geoip.timezone:America/New_York geoip.real_region_name:Michigan geoip.location:-83.7145, 42.29230000000001 tags:geoip _id:AVJ1kT2zkzWl-HxHjshu _type:apache_access

So, I'm still doing something wrong.

8

The type field is wrong; "access_log" instead of "apache_access".

9

The change in type didn't work. Remembering that I've made no changes to my logstash configs, when it worked with the logstash-forwarder, here's what the stanza in logstash-forwarder.conf looked like:

{
  "paths": [ "/var/log/httpd/access_log" ],
  "fields": { "type": "apache-access" }
}

And, you can see that type specified in the Kibana output:

January 24th 2016, 18:05:24.000 host:satest02.netatlantic.com type:apache-access file:/var/log/httpd/access_log offset:2651 line:213.136.93.123 - - [24/Jan/2016:18:05:24 -0500] "GET / HTTP/1.1" 403 4961 "-" "-" @version:1 @timestamp:January 24th 2016, 18:05:24.000 clientip:213.136.93.123 ident:- auth:- timestamp:24/Jan/2016:18:05:24 -0500 verb:GET request:/ httpversion:1.1 response:403 bytes:4961 referrer:"-" agent:"-" geoip.ip:213.136.93.123 geoip.country_code2:DE geoip.country_code3:DEU geoip.country_name:Germany geoip.continent_code:EU geoip.latitude:51 geoip.longitude:9 geoip.timezone:Europe/Berlin geoip.location:9, 51 tags:geoip _id:AVJ15KAikzWl-HxHjwqr _type:apache-access _index:logstash-2016.01.24 _score:

Anything else I can try?

10

That event looks like it's being properly parsed. What's the problem?

11

Oh, stupid me!!! I had the type as "apache_access", when it should have been "apache-access". All good, now. Magnus, thanks for pointing me in the right direction. I get the prospectors piece in filebeat.conf now.