Kibana +geoip from ipv4 addr in a varx field

[bruno-d (bruno.dev)]

Hi,

i parsed syslog_message to extract ip address from my ADSL modem to check who's ringing and store it into a new text field Varx (of my logstash index). Varx is a IPV4 text value computed every hours by an update_by_query POST in a cron job. The query looks into the syslog_message string to set Varx. I developed the query in Kibana console and use the cUrl in a shell.

I would like later to use a service that fill geoip structure (using free maxmind DB) by using the IPv4 stored in Varx. Last step would be to display a MAP into kibana ?

My question is : I did not yet find a way to to that. Parsing all non empty VarX and set geoip.yyy to the appropriate value return by function(Varx).

Any accurate tips ? thanks for the help

Please note : i am completely new running ELK 5.6.4 on RPI3

[warkolm (Mark Walkom)]

FYI we’ve renamed ELK to the Elastic Stack, otherwise Beats and APM feel left out!

Can you show us examples of the varx field?

[bruno-d (bruno.dev)]

Hi, Varx are string IPV4 WAN address as 'text' like xxx.xxx.xxx.xxx
Mostly it's bot who perform port scanning. One value could be Varx = 222.186.129.68 (this one is in china )

As explained, once Varx has been computed (by the update_by_query running each hour) i would like to compute the geoip fields from a DB or services using Varx as parameter to display a map with stats (count).

thanks for any ideas.

Regards

[warkolm (Mark Walkom)]

You will need to reprocess the document through the geoip filter, either in Logstash or an Ingest pipeline.

[bruno-d (bruno.dev)]

Thanks for your answer. Could you point out the way to do it accurately ?
thanks

[bruno-d (bruno.dev)]

hi guys, does someone can help on this product !

[bruno-d (bruno.dev)]

put this thread on top of stack:joy:
Still not solve

[warkolm (Mark Walkom)]

https://www.elastic.co/guide/en/logstash/current/plugins-filters-geoip.html explains how to do this.

[Christian_Dahlqvist]

You can do this through the update by query API if you direct the data to update through an ingest node pipeline that uses the ingest geoip processor plugin (need to be installed separately).

[bruno-d (bruno.dev)]

Sorry, to bother but if the answer was easy in the doc we might not ask. From my perspective doc and info are really confusing not representative of the possibilities of the solution.
Anyway, we will try to make it run.
thanks i did a filter source => "Varx" and got a geoip failure parsing output how ever Varx is not filled at that time (empty string) . you mention that i need to reprocess once Varx is Filled but i don't know how.

[bruno-d (bruno.dev)]

Hi Thanks Christian, I read your answer again and installed the pipeline geoip. Perfect (need to increase heap space to avoid crash on the PI). Then it's complicated. I tried as in the doc this

PUT _ingest/pipeline/geoip
{
"description" : "Add geoip info",
"processors" : [
{
"geoip" : {
"field" : "Varx"
}
}
]
}

As Varx is my text field containing IP. Kibana acknoledge TRUE.
{
"acknowledged": true
}

my geoIP is still empty ? Could you help again ? Thanks and apologize for bothering

[Christian_Dahlqvist]

So you have stored the new pipeline. How did you use it to update documents?

[bruno-d (bruno.dev)]

good question

I was able to get google IP using this syntax
PUT logstash-2017.12.24/syslog/_all/?pipeline=geoip
{
"Varx": "8.8.8.8"
}
and GET logstash-2017.12.24/syslog/_all returned

{
"_index": "logstash-2017.12.24",
"_type": "syslog",
"_id": "_all",
"_version": 2,
"found": true,
"_source": {
"Varx": "8.8.8.8",
"geoip": {
"continent_name": "North America",
"city_name": "Mountain View",
"country_iso_code": "US",
"region_name": "California",
"location": {
"lon": -122.0838,
"lat": 37.386
}
}
}
}

But i don't know to process _all my elements of syslog that have VarX affected Value ? Do you know how please ?

[Christian_Dahlqvist]

Have you looked at the examples in the update-by-query link I provided?

[bruno-d (bruno.dev)]

Hey M8 thanks to answer.

I ran POST /%3Clogstash-%7Bnow%2Fd%7D%3E/syslog/_update_by_query?pipeline=geoip

It's start to sounds good.. but is GeoIP case sensitive '???' what should be VarX format ?
Does the space at the end is an issue ?

"failures": [
{
"index": "logstash-2017.12.24",
"type": "syslog",
"id": "AWCG2d79MywR9Y3-E7vT",
"cause": {
"type": "exception",
"reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: '191.101.167.235 ' is not an IP string literal.",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "java.lang.IllegalArgumentException: '191.101.167.235 ' is not an IP string literal.",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "'191.101.167.235 ' is not an IP string literal."
}
},

PS: when default value is '0' the filter crash. so i set the default to 8.8.8.8 mostly when the syslog message is not related to intrusion i.e Varx has no real value.

regards

[bruno-d (bruno.dev)]

looks better

[bruno-d (bruno.dev)]

Thanks work like a charm. need to trim the IP field to remove space... now it's perfect . nice job.