# Need help with adding geoip to syslog and then how to visualize... - **URL:** \ **Category:** Logstash\ **Created:** [November 22, 2019, 6:47pm UTC](https://discuss.elastic.co/t/need-help-with-adding-geoip-to-syslog-and-then-how-to-visualize/209113 "2019-11-22T18:47:25Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Cdnvballer](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Cdnvballer](https://discuss.elastic.co/u/Cdnvballer)\ **Post date:** [November 22, 2019, 6:47pm UTC](https://discuss.elastic.co/t/need-help-with-adding-geoip-to-syslog-and-then-how-to-visualize/209113/1 "2019-11-22T18:47:25Z")
Hi, i'm receiving network device syslog via filebeats - works fine, using default syslog filter grok { match =\> { "message" =\> ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:[%{ POSINT:[system][syslog][pid]}])?: %{GREEDYMULTILINE:[system][syslog][message]}"] } pattern\_definitions =\> { "GREEDYMULTILINE" =\> "(.|\n)\*" } I try to add the following: geoip{ ``` if [system][syslog][hostname] =~ /^10\.101\./ { mutate { add_field => { "[geoip][location]" => "37.388340" } } ``` mutate { add\_field =\> { "[geoip][location]" =\> "-121.888420" } } mutate { convert =\> ["[geoip][location]", "float" ] } mutate { replace =\> ["[geoip][latitude]", 37.388340 ] } mutate { convert =\> ["[geoip][latitude]", "float" ] } mutate { replace =\> ["[geoip][longitude]", -121.888420 ] } mutate { convert =\> ["[geoip][longitude]", "float" ] } } !and a few other if statements that check the first two octets of [system][syslog][hostname] and assign latitude and longitude... } is this valid? and if so , what do I need to do in order to access [geoip][location] in order to see these on a map? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 22, 2019, 6:58pm UTC](https://discuss.elastic.co/t/need-help-with-adding-geoip-to-syslog-and-then-how-to-visualize/209113/2 "2019-11-22T18:58:57Z")
[geoip][location] is a geo\_point. There are several [options](https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html) for feeding elasticsearch geo\_points. I believe this is one of them... ``` mutate { add_field => { "[geoip][location][lat]" => "37.388340" "[geoip][location][lon]" => "-121.888420" } } ``` ---
**Author:** ![Cdnvballer](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Cdnvballer](https://discuss.elastic.co/u/Cdnvballer)\ **Post date:** [November 22, 2019, 7:11pm UTC](https://discuss.elastic.co/t/need-help-with-adding-geoip-to-syslog-and-then-how-to-visualize/209113/3 "2019-11-22T19:11:41Z")
ok- so that seems to reduce the # of statements required to add the coordinates. but do I need to do something to get this into my indexing? from the filebeats side...? sorry , have a bare-bones setup and still trying to learn this on the fly... ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 22, 2019, 7:38pm UTC](https://discuss.elastic.co/t/need-help-with-adding-geoip-to-syslog-and-then-how-to-visualize/209113/4 "2019-11-22T19:38:02Z")
You would still need to add latitude, longitude, and the ip to geoip, but you can use a single mutate+add\_field to add them, and I do not think the conversion to float is required. elasticsearch will convert them as it maps them. ---
**Author:** ![Cdnvballer](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Cdnvballer](https://discuss.elastic.co/u/Cdnvballer)\ **Post date:** [November 22, 2019, 8:51pm UTC](https://discuss.elastic.co/t/need-help-with-adding-geoip-to-syslog-and-then-how-to-visualize/209113/5 "2019-11-22T20:51:12Z")
(Note, I'm using logstash 6.8.4) I tried adding the following to the conf file ``` geoip{ if [system][syslog][hostname] =~ /^10\.101\./ { mutate { add_field => { "[geoip][location][lat]" => "37.388340" "[geoip][location][lon]" => "-121.888420" } } } ``` and get the following error: [2019-11-22T12:47:03,843][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, =\> at line 42, column 5 (byte 3197) after filter {\n if [fileset][module] == "system" {\n if [fileset][name] == "auth" {\n grok {\n match =\> { "message" =\> ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",\n "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",\n "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped\_ip]}",\n "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s\*%{DATA:[system][auth][user]} ☹ %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",\n "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",\n "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$",\n "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}"] }\n pattern\_definitions =\> {\n "GREEDYMULTILINE"=\> "(.|\n)_"\n }\n remove\_field =\> "message"\n }\n date {\n match =\> ["[system][auth][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]\n }\n geoip {\n source =\> "[system][auth][ssh][ip]"\n target =\> "[system][auth][ssh][geoip]"\n }\n }\n else if [fileset][name] == "syslog" {\n grok {\n match =\> { "message" =\> ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }\n pattern\_definitions =\> { "GREEDYMULTILINE" =\> "(.|\n)_" }\n remove\_field =\> "message"\n }\n date {\n match =\> ["[system][syslog][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]\n }\n geoip{\n\n\tif ", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile\_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:22:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:43:in `block in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in `block in exclusive'", "org/jruby/ext/thread/Mutex.java:165:in `synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in `exclusive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:39:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:334:in `block in converge\_state'"]} ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 22, 2019, 10:24pm UTC](https://discuss.elastic.co/t/need-help-with-adding-geoip-to-syslog-and-then-how-to-visualize/209113/6 "2019-11-22T22:24:43Z")
You cannot put the if statement inside the geoip filter. Put the filter inside the if statement. ---
**Author:** ![Cdnvballer](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Cdnvballer](https://discuss.elastic.co/u/Cdnvballer)\ **Post date:** [November 25, 2019, 5:04pm UTC](https://discuss.elastic.co/t/need-help-with-adding-geoip-to-syslog-and-then-how-to-visualize/209113/7 "2019-11-25T17:04:34Z")
Sorry I think you've lost me -- so it would be like this? ``` if [system][syslog][hostname] =~ /^10\.101\./ { geoip { add_field => { "[geoip][location][lat]" => "37.388340" "[geoip][location][lon]" => "-121.888420" } } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 25, 2019, 6:27pm UTC](https://discuss.elastic.co/t/need-help-with-adding-geoip-to-syslog-and-then-how-to-visualize/209113/8 "2019-11-25T18:27:57Z")
Like this: ``` filter { if [system][syslog][hostname] =~ /^10\.101\./ { mutate { add_field => { "[geoip][location][lat]" => "37.388340" "[geoip][location][lon]" => "-121.888420" } } } ``` ---
**Author:** ![Cdnvballer](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Cdnvballer](https://discuss.elastic.co/u/Cdnvballer)\ **Post date:** [November 25, 2019, 7:37pm UTC](https://discuss.elastic.co/t/need-help-with-adding-geoip-to-syslog-and-then-how-to-visualize/209113/9 "2019-11-25T19:37:00Z")
thanks again - I made that change and added the other if statements to cover all other site cases... after restarting logstash, I now see geoip.location.lat and geoip.location.lon as available fields. however I'm not sure how to get those into the map visualization Visualize --\> New Visualization --\> Coordinate map , using filebeat-\* as my index geo\_point is greyed out... ---
**Author:** ![Cdnvballer](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Cdnvballer](https://discuss.elastic.co/u/Cdnvballer)\ **Post date:** [November 25, 2019, 7:40pm UTC](https://discuss.elastic.co/t/need-help-with-adding-geoip-to-syslog-and-then-how-to-visualize/209113/10 "2019-11-25T19:40:40Z")
sorry - I don't call geoip{} anywhere after the IF statement, am I missing this? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 25, 2019, 8:48pm UTC](https://discuss.elastic.co/t/need-help-with-adding-geoip-to-syslog-and-then-how-to-visualize/209113/11 "2019-11-25T20:48:41Z")
No, in the case of a private network, where you are adding the fields of geoip using mutate, you do not need to use a geoip filter. To answer your previous post ... You need to use an index template. It could look very much like the [default template for logstash-\*](https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/elasticsearch-template-es7x.json), which defines geoip.location as a geo\_point. Just change the index\_patterns field. ---
**Author:** ![Cdnvballer](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Cdnvballer](https://discuss.elastic.co/u/Cdnvballer)\ **Post date:** [November 25, 2019, 9:16pm UTC](https://discuss.elastic.co/t/need-help-with-adding-geoip-to-syslog-and-then-how-to-visualize/209113/12 "2019-11-25T21:16:44Z")
1. you lost me at using index template... =\ sorry, trying to lookg into this now... 2. I'm seeing weird issue: messages with system.syslog.hostname = 10.101.x.x or 10.20.x.x get geoip.location.lat and geoip.location.lon values added messages with 10.111.x.x , 10.24.x.x , 10.45.x.x do not. my IF block within the filter looks like this: ``` if [system][syslog][hostname] =~ /(10\.101.*)/ { mutate { add_field => { "[geoip][location][lat]" => "37.388340" "[geoip][location][lon]" => "-121.888420" } } } else if [system][syslog][hostname] =~ /(10\.24.*)/ { mutate { add_field => { "[geoip][location][lat]" => "22.306110" "[geoip][location][lon]" => "114.187480" } } } else if [system][syslog][hostname] =~ /(10\.110.*)/ { mutate { add_field => { "[geoip][location][lat]" => "35.654902" "[geoip][location][lon]" => "139.774605" } } } ``` ... ``` else if [system][syslog][hostname] =~ /(10\.20.*)/ or [system][syslog][hostname] =~ /(10\.21.*)/ or [system][syslog][hostname] =~ /(10\.5.*)/ or [system][syslog][hostname] =~ /(10\.2.*)/ { mutate { add_field => { "[geoip][location][lat]" => "37.391640" "[geoip][location][lon]" => "-121.891630" } } } ``` ---
**Author:** ![Cdnvballer](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Cdnvballer](https://discuss.elastic.co/u/Cdnvballer)\ **Post date:** [November 27, 2019, 1:43am UTC](https://discuss.elastic.co/t/need-help-with-adding-geoip-to-syslog-and-then-how-to-visualize/209113/13 "2019-11-27T01:43:47Z")
ok so I ran this to get my index: curl -XGET [http://127.0.0.1:9200/filebeat-6.8.4-2019.11.27/\_mapping?pretty](http://127.0.0.1:9200/filebeat-6.8.4-2019.11.27/_mapping?pretty) \> my\_mapping.json and I can find section that has [system][syslog][hostname] and [system][syslog][timestamp]: "syslog" : { "properties" : { "hostname" : { "type" : "keyword", "ignore\_above" : 1024 }, "message" : { "type" : "text", "norms" : false }, "pid" : { "type" : "keyword", "ignore\_above" : 1024 }, "program" : { "type" : "keyword", "ignore\_above" : 1024 }, "timestamp" : { "type" : "keyword", "ignore\_above" : 1024 } } } would it be just a matter of adding this? --\> ``` "geoip" : { "dynamic": true, "properties" : { "ip": { "type": "ip" }, "location" : { "type" : "geo_point" }, "latitude" : { "type" : "half_float" }, "longitude" : { "type" : "half_float" } } } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 25, 2019, 1:43am UTC](https://discuss.elastic.co/t/need-help-with-adding-geoip-to-syslog-and-then-how-to-visualize/209113/14 "2019-12-25T01:43:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.