Need help with adding geoip to syslog and then how to visualize... -

Cdnvballer · November 22, 2019, 6:47pm

Hi,

i'm receiving network device syslog via filebeats - works fine, using default syslog filter

grok {
match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:[%{
POSINT:[system][syslog][pid]}])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }
pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }

I try to add the following:

geoip{

if [system][syslog][hostname] =~ /^10\.101\./  {
   mutate { add_field    => { "[geoip][location]"      => "37.388340" } }

mutate { add_field => { "[geoip][location]" => "-121.888420" } }
mutate { convert => [ "[geoip][location]", "float" ] }
mutate { replace => [ "[geoip][latitude]", 37.388340 ] }
mutate { convert => [ "[geoip][latitude]", "float" ] }
mutate { replace => [ "[geoip][longitude]", -121.888420 ] }
mutate { convert => [ "[geoip][longitude]", "float" ] }
}

!and a few other if statements that check the first two octets of [system][syslog][hostname] and assign latitude and longitude...

}

is this valid?
and if so , what do I need to do in order to access [geoip][location] in order to see these on a map?

Badger · November 22, 2019, 6:58pm

[geoip][location] is a geo_point. There are several options for feeding elasticsearch geo_points. I believe this is one of them...

mutate {
    add_field    => {
        "[geoip][location][lat]" => "37.388340"
        "[geoip][location][lon]" => "-121.888420"
    }
}

Cdnvballer · November 22, 2019, 7:11pm

ok- so that seems to reduce the # of statements required to add the coordinates.
but do I need to do something to get this into my indexing?
from the filebeats side...?
sorry , have a bare-bones setup and still trying to learn this on the fly...

Badger · November 22, 2019, 7:38pm

You would still need to add latitude, longitude, and the ip to geoip, but you can use a single mutate+add_field to add them, and I do not think the conversion to float is required. elasticsearch will convert them as it maps them.

Cdnvballer · November 22, 2019, 8:51pm

(Note, I'm using logstash 6.8.4)
I tried adding the following to the conf file

  geoip{

    if [system][syslog][hostname] =~ /^10\.101\./  {
       mutate { 
            add_field    => { 
                    "[geoip][location][lat]" => "37.388340"
                    "[geoip][location][lon]" => "-121.888420" 
                    } 
            }
    }

and get the following error:

[2019-11-22T12:47:03,843][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 42, column 5 (byte 3197) after filter {\n if [fileset][module] == "system" {\n if [fileset][name] == "auth" {\n grok {\n match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",\n "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",\n "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",\n "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s*%{DATA:[system][auth][user]} %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",\n "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",\n "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$",\n "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}"] }\n pattern_definitions => {\n "GREEDYMULTILINE"=> "(.|\n)"\n }\n remove_field => "message"\n }\n date {\n match => [ "[system][auth][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]\n }\n geoip {\n source => "[system][auth][ssh][ip]"\n target => "[system][auth][ssh][geoip]"\n }\n }\n else if [fileset][name] == "syslog" {\n grok {\n match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }\n pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)" }\n remove_field => "message"\n }\n date {\n match => [ "[system][syslog][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]\n }\n geoip{\n\n\tif ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2577:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:22:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:43:in block in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in block in exclusive'", "org/jruby/ext/thread/Mutex.java:165:in synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in exclusive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:39:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:334:in block in converge_state'"]}

Badger · November 22, 2019, 10:24pm

You cannot put the if statement inside the geoip filter. Put the filter inside the if statement.

Cdnvballer · November 25, 2019, 5:04pm

Sorry I think you've lost me -- so it would be like this?

if [system][syslog][hostname] =~ /^10\.101\./  {
   geoip { 
	add_field    => { 
		"[geoip][location][lat]"      => "37.388340"
		"[geoip][location][lon]"      => "-121.888420" 
		} 
	}
}

Badger · November 25, 2019, 6:27pm

Like this:

filter {
    if [system][syslog][hostname] =~ /^10\.101\./  {
        mutate { 
            add_field    => { 
                "[geoip][location][lat]" => "37.388340"
                "[geoip][location][lon]" => "-121.888420" 
                } 
        }
    }

Cdnvballer · November 25, 2019, 7:37pm

thanks again - I made that change and added the other if statements to cover all other site cases...
after restarting logstash, I now see geoip.location.lat and geoip.location.lon as available fields.
however I'm not sure how to get those into the map visualization
Visualize --> New Visualization --> Coordinate map , using filebeat-* as my index
geo_point is greyed out...

Cdnvballer · November 25, 2019, 7:40pm

sorry - I don't call geoip{} anywhere after the IF statement, am I missing this?

Badger · November 25, 2019, 8:48pm

No, in the case of a private network, where you are adding the fields of geoip using mutate, you do not need to use a geoip filter.

To answer your previous post ... You need to use an index template. It could look very much like the default template for logstash-*, which defines geoip.location as a geo_point. Just change the index_patterns field.

Cdnvballer · November 25, 2019, 9:16pm

you lost me at using index template... =\ sorry, trying to lookg into this now...
I'm seeing weird issue:
messages with system.syslog.hostname = 10.101.x.x or 10.20.x.x get geoip.location.lat and geoip.location.lon values added
messages with 10.111.x.x , 10.24.x.x , 10.45.x.x do not.

my IF block within the filter looks like this:

   if [system][syslog][hostname] =~ /(10\.101.*)/  {
       mutate {
            add_field    => {
                    "[geoip][location][lat]"      => "37.388340"
                    "[geoip][location][lon]"      => "-121.888420"
                    }
            }
    }

    else if   [system][syslog][hostname] =~ /(10\.24.*)/ {
       mutate {
            add_field    => {
                    "[geoip][location][lat]"      => "22.306110"
                    "[geoip][location][lon]"      => "114.187480"
                    }
            }
    }

    else if   [system][syslog][hostname] =~ /(10\.110.*)/  {
       mutate {
            add_field    => {
                    "[geoip][location][lat]"      => "35.654902"
                    "[geoip][location][lon]"      => "139.774605"
                    }
            }
    }

...

    else if [system][syslog][hostname] =~ /(10\.20.*)/ or  [system][syslog][hostname] =~ /(10\.21.*)/ or  [system][syslog][hostname] =~ /(10\.5.*)/ or [system][syslog][hostname] =~ /(10\.2.*)/ {
       mutate {
            add_field    => {
                    "[geoip][location][lat]"      => "37.391640"
                    "[geoip][location][lon]"      => "-121.891630"
                    }
            }
    }

Cdnvballer · November 27, 2019, 1:43am

ok so I ran this to get my index:
curl -XGET http://127.0.0.1:9200/filebeat-6.8.4-2019.11.27/_mapping?pretty > my_mapping.json

and I can find section that has [system][syslog][hostname] and [system][syslog][timestamp]:

"syslog" : {
"properties" : {
"hostname" : {
"type" : "keyword",
"ignore_above" : 1024
},
"message" : {
"type" : "text",
"norms" : false
},
"pid" : {
"type" : "keyword",
"ignore_above" : 1024
},
"program" : {
"type" : "keyword",
"ignore_above" : 1024
},
"timestamp" : {
"type" : "keyword",
"ignore_above" : 1024
}
}
}

would it be just a matter of adding this? -->

  "geoip"  : {
    "dynamic": true,
    "properties" : {
      "ip": { "type": "ip" },
      "location" : { "type" : "geo_point" },
      "latitude" : { "type" : "half_float" },
      "longitude" : { "type" : "half_float" }
    }
  }

system · December 25, 2019, 1:43am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
GEO POINT Problem Beats filebeat	2	413	September 3, 2018
Geo_point is not getting mapped correctly Logstash	6	1726	September 4, 2017
Adding parameters from grok to syslog output Logstash	1	291	October 18, 2018
Completely lost Logstash	6	1141	July 6, 2017
How to setup logstash with geoip Logstash	14	3647	February 21, 2017

Need help with adding geoip to syslog and then how to visualize... -

Related topics