Need help with adding geoip to syslog and then how to visualize... -

Cdnvballer · November 22, 2019, 6:47pm

Hi,

i'm receiving network device syslog via filebeats - works fine, using default syslog filter

grok {
match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:[%{
POSINT:[system][syslog][pid]}])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }
pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }

I try to add the following:

geoip{

if [system][syslog][hostname] =~ /^10\.101\./  {
   mutate { add_field    => { "[geoip][location]"      => "37.388340" } }

mutate { add_field => { "[geoip][location]" => "-121.888420" } }
mutate { convert => [ "[geoip][location]", "float" ] }
mutate { replace => [ "[geoip][latitude]", 37.388340 ] }
mutate { convert => [ "[geoip][latitude]", "float" ] }
mutate { replace => [ "[geoip][longitude]", -121.888420 ] }
mutate { convert => [ "[geoip][longitude]", "float" ] }
}

!and a few other if statements that check the first two octets of [system][syslog][hostname] and assign latitude and longitude...

}

is this valid?
and if so , what do I need to do in order to access [geoip][location] in order to see these on a map?

Badger · November 22, 2019, 6:58pm

[geoip][location] is a geo_point. There are several options for feeding elasticsearch geo_points. I believe this is one of them...

mutate {
    add_field    => {
        "[geoip][location][lat]" => "37.388340"
        "[geoip][location][lon]" => "-121.888420"
    }
}

Cdnvballer · November 22, 2019, 7:11pm

ok- so that seems to reduce the # of statements required to add the coordinates.
but do I need to do something to get this into my indexing?
from the filebeats side...?
sorry , have a bare-bones setup and still trying to learn this on the fly...

Badger · November 22, 2019, 7:38pm

You would still need to add latitude, longitude, and the ip to geoip, but you can use a single mutate+add_field to add them, and I do not think the conversion to float is required. elasticsearch will convert them as it maps them.

Cdnvballer · November 22, 2019, 8:51pm

(Note, I'm using logstash 6.8.4)
I tried adding the following to the conf file

  geoip{

    if [system][syslog][hostname] =~ /^10\.101\./  {
       mutate { 
            add_field    => { 
                    "[geoip][location][lat]" => "37.388340"
                    "[geoip][location][lon]" => "-121.888420" 
                    } 
            }
    }

and get the following error:

[2019-11-22T12:47:03,843][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 42, column 5 (byte 3197) after filter {\n if [fileset][module] == "system" {\n if [fileset][name] == "auth" {\n grok {\n match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",\n "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",\n "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",\n "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s*%{DATA:[system][auth][user]} %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",\n "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",\n "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$",\n "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}"] }\n pattern_definitions => {\n "GREEDYMULTILINE"=> "(.|\n)"\n }\n remove_field => "message"\n }\n date {\n match => [ "[system][auth][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]\n }\n geoip {\n source => "[system][auth][ssh][ip]"\n target => "[system][auth][ssh][geoip]"\n }\n }\n else if [fileset][name] == "syslog" {\n grok {\n match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }\n pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)" }\n remove_field => "message"\n }\n date {\n match => [ "[system][syslog][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]\n }\n geoip{\n\n\tif ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2577:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:22:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:43:in block in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in block in exclusive'", "org/jruby/ext/thread/Mutex.java:165:in synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in exclusive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:39:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:334:in block in converge_state'"]}

Badger · November 22, 2019, 10:24pm

You cannot put the if statement inside the geoip filter. Put the filter inside the if statement.

Cdnvballer · November 25, 2019, 5:04pm

Sorry I think you've lost me -- so it would be like this?

if [system][syslog][hostname] =~ /^10\.101\./  {
   geoip { 
	add_field    => { 
		"[geoip][location][lat]"      => "37.388340"
		"[geoip][location][lon]"      => "-121.888420" 
		} 
	}
}

Badger · November 25, 2019, 6:27pm

Like this:

filter {
    if [system][syslog][hostname] =~ /^10\.101\./  {
        mutate { 
            add_field    => { 
                "[geoip][location][lat]" => "37.388340"
                "[geoip][location][lon]" => "-121.888420" 
                } 
        }
    }

Cdnvballer · November 25, 2019, 7:37pm

thanks again - I made that change and added the other if statements to cover all other site cases...
after restarting logstash, I now see geoip.location.lat and geoip.location.lon as available fields.
however I'm not sure how to get those into the map visualization
Visualize --> New Visualization --> Coordinate map , using filebeat-* as my index
geo_point is greyed out...

Cdnvballer · November 25, 2019, 7:40pm

sorry - I don't call geoip{} anywhere after the IF statement, am I missing this?

Badger · November 25, 2019, 8:48pm

No, in the case of a private network, where you are adding the fields of geoip using mutate, you do not need to use a geoip filter.

To answer your previous post ... You need to use an index template. It could look very much like the default template for logstash-*, which defines geoip.location as a geo_point. Just change the index_patterns field.

Cdnvballer · November 25, 2019, 9:16pm

you lost me at using index template... =\ sorry, trying to lookg into this now...
I'm seeing weird issue:
messages with system.syslog.hostname = 10.101.x.x or 10.20.x.x get geoip.location.lat and geoip.location.lon values added
messages with 10.111.x.x , 10.24.x.x , 10.45.x.x do not.

my IF block within the filter looks like this:

   if [system][syslog][hostname] =~ /(10\.101.*)/  {
       mutate {
            add_field    => {
                    "[geoip][location][lat]"      => "37.388340"
                    "[geoip][location][lon]"      => "-121.888420"
                    }
            }
    }

    else if   [system][syslog][hostname] =~ /(10\.24.*)/ {
       mutate {
            add_field    => {
                    "[geoip][location][lat]"      => "22.306110"
                    "[geoip][location][lon]"      => "114.187480"
                    }
            }
    }

    else if   [system][syslog][hostname] =~ /(10\.110.*)/  {
       mutate {
            add_field    => {
                    "[geoip][location][lat]"      => "35.654902"
                    "[geoip][location][lon]"      => "139.774605"
                    }
            }
    }

...

    else if [system][syslog][hostname] =~ /(10\.20.*)/ or  [system][syslog][hostname] =~ /(10\.21.*)/ or  [system][syslog][hostname] =~ /(10\.5.*)/ or [system][syslog][hostname] =~ /(10\.2.*)/ {
       mutate {
            add_field    => {
                    "[geoip][location][lat]"      => "37.391640"
                    "[geoip][location][lon]"      => "-121.891630"
                    }
            }
    }

Cdnvballer · November 27, 2019, 1:43am

ok so I ran this to get my index:
curl -XGET http://127.0.0.1:9200/filebeat-6.8.4-2019.11.27/_mapping?pretty > my_mapping.json

and I can find section that has [system][syslog][hostname] and [system][syslog][timestamp]:

"syslog" : {
"properties" : {
"hostname" : {
"type" : "keyword",
"ignore_above" : 1024
},
"message" : {
"type" : "text",
"norms" : false
},
"pid" : {
"type" : "keyword",
"ignore_above" : 1024
},
"program" : {
"type" : "keyword",
"ignore_above" : 1024
},
"timestamp" : {
"type" : "keyword",
"ignore_above" : 1024
}
}
}

would it be just a matter of adding this? -->

  "geoip"  : {
    "dynamic": true,
    "properties" : {
      "ip": { "type": "ip" },
      "location" : { "type" : "geo_point" },
      "latitude" : { "type" : "half_float" },
      "longitude" : { "type" : "half_float" }
    }
  }

system · December 25, 2019, 1:43am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Trouble with GeoIP mapping form logstash to elastic search Logstash	8	3403	March 24, 2017
Help with "_grokparsefailure" and "_geoip_lookup_failure" Logstash	6	297	August 1, 2022
Syslog filter for IP Logstash	4	830	March 7, 2021
Geoip property into filebeat index template - help Elasticsearch	1	461	January 6, 2020
_geoip_lookup_failure Error Logstash	3	2018	September 27, 2018

Need help with adding geoip to syslog and then how to visualize... -

Related Topics