# Grokparsefailure, Geoip lookup failure **URL:** \ **Category:** Logstash\ **Created:** [July 25, 2018, 3:54am UTC](https://discuss.elastic.co/t/grokparsefailure-geoip-lookup-failure/141477 "2018-07-25T03:54:28Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Adah](https://avatars.discourse-cdn.com/v4/letter/a/a183cd/32.png) [@Adah](https://discuss.elastic.co/u/Adah)\ **Post date:** [July 25, 2018, 3:54am UTC](https://discuss.elastic.co/t/grokparsefailure-geoip-lookup-failure/141477/1 "2018-07-25T03:54:28Z")
Hi! I am using ELK 6.2.4 on Ubuntu Centos7. I'm a student doing a project and i am having trouble in getting the geoip ☹ ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/a/4a4f9ac213ba99b658e5eb47be8c5e69079a56b8.png) logstash conf file: ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/1/712ae646d12965fbc9a90f0499f9b3b9caf8ad14.png) i run /usr/share/logstash/bin$ sudo ./logstash -f /etc/logstash/conf.d/alert.conf the output: ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/2/c2f37366da6d064ae531a6bec45d4643aa211549.png) I'm using filebeat to logstash . Can you help me to solve it for my school project. deadlin submission is soon ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 25, 2018, 4:03am UTC](https://discuss.elastic.co/t/grokparsefailure-geoip-lookup-failure/141477/2 "2018-07-25T04:03:13Z")
Please don't post pictures of text, they are difficult to read and some people may not be even able to see them 🙂 ---
**Author:** ![Adah](https://avatars.discourse-cdn.com/v4/letter/a/a183cd/32.png) [@Adah](https://discuss.elastic.co/u/Adah)\ **Post date:** [July 25, 2018, 4:46am UTC](https://discuss.elastic.co/t/grokparsefailure-geoip-lookup-failure/141477/3 "2018-07-25T04:46:21Z")
Sorry about that! **logstash conf file:** input { beats { port =\> 5044 } } filter { grok { match =\> { "message" =\> ["%{COMBINEDAPACHELOG}" , "%{COMMONAPACHELOG}"]} } date { match =\> ["timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"] } geoip { source =\> "clientip" } mutate { convert =\> ["[geoip][coordinates]", "float"] } csv { columns =\> ["datetime","msg","src\_ip","src\_port","src\_mac-addr","dst\_ip","dst\_port","dst\_mac\_addr","category","proto","priority"] separator =\> "," } } output { elasticsearch { hosts =\> ["localhost:9200"] } stdout { codec =\> "rubydebug"} } Then i run, /usr/share/logstash/bin$ sudo ./logstash -f /etc/logstash/conf.d/alert.conf the output: WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [INFO] 2018-07-25 12:44:28.037 [main] scaffold - Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"} [INFO] 2018-07-25 12:44:28.052 [main] scaffold - Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"} [WARN] 2018-07-25 12:44:28.849 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO] 2018-07-25 12:44:29.269 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=\>"6.2.4"} [INFO] 2018-07-25 12:44:29.698 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=\>9600} [INFO] 2018-07-25 12:44:33.786 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50} [INFO] 2018-07-25 12:44:34.917 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>[[http://localhost:9200/](http://localhost:9200/)]}} [INFO] 2018-07-25 12:44:34.957 [[main]-pipeline-manager] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://localhost:9200/](http://localhost:9200/), :path=\>"/"} [WARN] 2018-07-25 12:44:35.381 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=\>"[http://localhost:9200/](http://localhost:9200/)"} [INFO] 2018-07-25 12:44:36.433 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es\_version=\>6} [WARN] 2018-07-25 12:44:36.434 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>6} [INFO] 2018-07-25 12:44:36.475 [[main]-pipeline-manager] elasticsearch - Using mapping template from {:path=\>nil} [INFO] 2018-07-25 12:44:36.507 [[main]-pipeline-manager] elasticsearch - Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>60001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date"}, "@version"=\>{"type"=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}} [INFO] 2018-07-25 12:44:36.552 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//localhost:9200](https://localhost:9200)"]} [INFO] 2018-07-25 12:44:36.981 [[main]-pipeline-manager] geoip - Using geoip database {:path=\>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"} [INFO] 2018-07-25 12:44:38.490 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=\>"0.0.0.0:5044"} [INFO] 2018-07-25 12:44:38.838 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started successfully {:pipeline\_id=\>"main", :thread=\>"#\"} [INFO] 2018-07-25 12:44:38.938 [[main]\1, :pipelines=\>["main"]} Here you go! ---
**Author:** ![Krunal\_kalaria](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krunal_kalaria/32/23862_2.png) [@Krunal\_kalaria](https://discuss.elastic.co/u/Krunal_kalaria)\ **Post date:** [July 25, 2018, 8:01am UTC](https://discuss.elastic.co/t/grokparsefailure-geoip-lookup-failure/141477/4 "2018-07-25T08:01:52Z")
Hello @Adah, Change thee field geoip { source =\> "clientip" } to geoip { source =\> "src\_ip" } and add create this template template.json and save it where your logstash file is their. { "template" : "new-_", "version" : 50001, "settings" : { "index.refresh\_interval" : "5s" }, "mappings" : { "default" : { "\_all" : {"enabled" : true, "omit\_norms" : false}, "dynamic\_templates" : [ { "message\_field" : { "path\_match" : "message", "match\_mapping\_type" : "string", "mapping" : { "type" : "text", "omit\_norms" : false } } }, { "string\_fields" : { "match" : "_", "match\_mapping\_type" : "string", "mapping" : { "type" : "text", "omit\_norms" : false, "fields" : { "keyword" : { "type": "keyword", "ignore\_above": 256 } } } } } ], "properties" : { "@timestamp": { "type": "date", "include\_in\_all": false }, "@version": { "type": "keyword", "include\_in\_all": false }, "geoip" : { "dynamic": true, "properties" : { "ip": { "type": "ip" }, "location" : { "type" : "geo\_point" }, "latitude" : { "type" : "half\_float" }, "longitude" : { "type" : "half\_float" } } }, "location": { "type": "geo\_point" } } } } } and add this two line in your configuration file of last in output part. template =\> "/usr/share/logstash/bin/template.json" template\_name =\> "new-\*" I hope this information may be worked for you and its sufficient. Thanks & Regards, Krunal. ---
**Author:** ![Adah](https://avatars.discourse-cdn.com/v4/letter/a/a183cd/32.png) [@Adah](https://discuss.elastic.co/u/Adah)\ **Post date:** [July 27, 2018, 1:24am UTC](https://discuss.elastic.co/t/grokparsefailure-geoip-lookup-failure/141477/5 "2018-07-27T01:24:55Z")
Hi! So i've change to { source =\> "src\_ip" } and create template.json and put exactly the same as yours. **logstash conf file:** input { beats { port =\> 5044 } } filter { grok { match =\> { "message" =\> "%{COMBINEDAPACHELOG}" } } date { match =\> ["timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"] } geoip { source =\> "src\_ip" target =\> "geoip" } mutate { convert =\> ["[geoip][coordinates]", "float"] } csv { columns =\> ["datetime","msg","src\_ip","src\_port","src\_mac-addr","dst\_ip","dst\_port","dst\_mac\_addr","category","proto","priority"] separator =\> "," } } output { elasticsearch { hosts =\> ["localhost:9200"] } template =\> "/usr/share/logstash/bin/template.json" template\_name =\> "new-\*" } * * * **then i run : /usr/share/logstash/bin$ sudo ./logstash -f /etc/logstash/conf.d/alert.conf** Output: WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [INFO] 2018-07-27 09:11:24.018 [main] scaffold - Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"} [INFO] 2018-07-27 09:11:24.034 [main] scaffold - Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"} [WARN] 2018-07-27 09:11:24.585 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO] 2018-07-27 09:11:24.954 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=\>"6.2.4"} [INFO] 2018-07-27 09:11:25.297 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=\>9600} [ERROR] 2018-07-27 09:11:25.488 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, { at line 42, column 10 (byte 514) after output {\n \nelasticsearch {\nhosts =\> ["localhost:9200"]\n}\n\ntemplate ", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in`compile\_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in`map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:in`initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in`execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in `block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in`with\_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in`each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in `converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in`block in converge\_state\_and\_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in`converge\_state\_and\_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:in`block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]} it has an error. do you know how to solve the error? Thank you 🙂 ---
**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\ **Post date:** [July 27, 2018, 5:06am UTC](https://discuss.elastic.co/t/grokparsefailure-geoip-lookup-failure/141477/6 "2018-07-27T05:06:57Z")
> [@Adah](#): > > template =\> "/usr/share/logstash/bin/template.json" > template\_name =\> "new-\*" This should be inside the brackets to work. They are options specific to the elasticsearch output plugin and therefore must be included within the plugin declaration. ---
**Author:** ![Adah](https://avatars.discourse-cdn.com/v4/letter/a/a183cd/32.png) [@Adah](https://discuss.elastic.co/u/Adah)\ **Post date:** [July 27, 2018, 5:59am UTC](https://discuss.elastic.co/t/grokparsefailure-geoip-lookup-failure/141477/7 "2018-07-27T05:59:28Z")
Alright! So i've put in this way output { elasticsearch { hosts =\> ["localhost:9200"] template =\> "/usr/share/logstash/bin/template.json" template\_name =\> "new-\*" } } and the kibana discover still show beats\_input\_codec\_plain\_applied, \_grokparsefailure, \_geoip\_lookup\_failure ---
**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\ **Post date:** [July 27, 2018, 6:04am UTC](https://discuss.elastic.co/t/grokparsefailure-geoip-lookup-failure/141477/8 "2018-07-27T06:04:44Z")
Could you post some sample logs that you are trying to parse? ---
**Author:** ![Adah](https://avatars.discourse-cdn.com/v4/letter/a/a183cd/32.png) [@Adah](https://discuss.elastic.co/u/Adah)\ **Post date:** [July 27, 2018, 7:02am UTC](https://discuss.elastic.co/t/grokparsefailure-geoip-lookup-failure/141477/9 "2018-07-27T07:02:58Z")
here are the sample log (.csv) which i tried to parse into kibana: 05/23-01:43:41.181405 ,"Failed Login Attempt",192.168.199.129,21,00:0C:29:25:FD:06,192.168.199.128,59852,00:0C:29:32:1C:75,TCP,,,10000003 05/23-01:44:14.242689 ,"Failed Login Attempt",192.168.199.129,21,00:0C:29:25:FD:06,192.168.199.128,59854,00:0C:29:32:1C:75,TCP,,,10000003 05/23-01:44:46.848152 ,"Failed Login Attempt",192.168.199.129,21,00:0C:29:25:FD:06,192.168.199.128,59856,00:0C:29:32:1C:75,TCP,,,10000003 Output in kibana: ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/6/26ea5213aa8c3a89405f737d726e5cdba9a86534.png) ---
**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\ **Post date:** [July 27, 2018, 7:12am UTC](https://discuss.elastic.co/t/grokparsefailure-geoip-lookup-failure/141477/10 "2018-07-27T07:12:05Z")
Hi Adah, If you are trying to parse a csv, do you really need a grok filter? Would something similar to this work? ```auto filter { csv { columns => ["datetime","msg","src_ip","src_port","src_mac-addr","dst_ip","dst_port","dst_mac_addr","category","proto","priority"] separator => "," } date { match => ["datetime" , "dd/MMM/yyyy:HH:mm:ss Z"] } geoip { source => "src_ip" target => "geoip" } mutate { convert => ["[geoip][coordinates]", "float"] } } ``` ---
**Author:** ![Krunal\_kalaria](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krunal_kalaria/32/23862_2.png) [@Krunal\_kalaria](https://discuss.elastic.co/u/Krunal_kalaria)\ **Post date:** [July 27, 2018, 7:18am UTC](https://discuss.elastic.co/t/grokparsefailure-geoip-lookup-failure/141477/11 "2018-07-27T07:18:16Z")
try with this config file: input { beats { port =\> 5044 } } filter { grok { match =\> { "message" =\> "%{COMBINEDAPACHELOG}" } } mutate { add\_field =\> { "generated\_time" =\> "%{year}-%{monthnum}-%{daynum}T%{time\_1}.000+05:30" } remove\_field =\> ["year", "month", "daynum", "time", "monthnum", "time\_1"] } date { match =\> ["generated\_time", "yyyy-MM-dd'T'HH:mm:ss.SSSZ"] timezone =\> "America/New\_York" #change with your timezone target =\> "@timestamp" } geoip { source =\> "src\_ip" target =\> "geoip" add\_field =\> ["[geoip][coordinates]","%{[geoip][longitude]}" ] add\_field =\> ["[geoip][coordinates]","%{[geoip][latitude]}" ] } mutate { convert =\> ["[geoip][coordinates]", "float"] } csv { separator =\> "," columns =\> ["datetime","msg","src\_ip","src\_port","src\_mac-addr","dst\_ip","dst\_port","dst\_mac\_addr","category","proto","priority"] } } output { stdout { codec =\> rubydebug } elasticsearch { hosts =\> ["localhost:9200"] template =\> "/usr/share/logstash/bin/template.json" template\_name =\> "new-\*" index =\> "test-%{+YYYY.MM.dd}" } } Are you not using X-pack right ? Thanks & Regards, Krunal. ---
**Author:** ![Adah](https://avatars.discourse-cdn.com/v4/letter/a/a183cd/32.png) [@Adah](https://discuss.elastic.co/u/Adah)\ **Post date:** [July 27, 2018, 11:38pm UTC](https://discuss.elastic.co/t/grokparsefailure-geoip-lookup-failure/141477/12 "2018-07-27T23:38:38Z")
Thank you! will try out! nope, i'm not using x-pack ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 24, 2018, 11:38pm UTC](https://discuss.elastic.co/t/grokparsefailure-geoip-lookup-failure/141477/13 "2018-08-24T23:38:41Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.