Geoip help

Elastic Stack Logstash
1

Hello ,

I am trying to implement ELK for my customized logs. Almost done with my limited knowledge.

Now logs are successfully parsing , storing in elasticsearch and showing in kibana.

But like to include geo location (tilemap) in kibana. Could you please suggest best options for doing this?

Logs format:
[Key]value, [Key2]value2, [key3]value3
[Key]value, [Key2]value2, [key3]value3

logstash configuration:-
input {
file{
path => "*.log"
start_position => "beginning"
}
}

filter
{
kv {
source => "message"
value_split => "]"
field_split => ","
trimkey => "["
}
output {
elasticsearch { hosts => ["127.0.0.1:9200"] }
stdout { codec => rubydebug }
}

2

Please show an example message so we can see what data is at our disposal.

3

Thanks you Magnus for help.

Sample logs:-

[HOST]THOST, [LOCATION]001, [TIME_STAMP]2016-05-18 08:52:37, [REG]4001, [OPTION]1754, [LAYER]GSDIM, [TYPE]REQ, [METHOD]SESSION, [ACTION]START
[HOST]THOST, [LOCATION]001, [TIME_STAMP]2016-05-18 08:52:38, [REG]4001, [OPTION]175, [LAYER]GSDIM, [TYPE]REP, [TIME]1, [METHOD]SESSION, [ACTION]START, [RESPONSE]Session Started
[HOST]THOST, [LOCATION]010, [TIME_STAMP]2016-05-18 08:52:45, [REG]4002, [OPTION]1756, [LAYER]GSDIM, [TYPE]REQ, [METHOD]LINE_ITEM, [ACTION]ADD
[HOST]THOST, [LOCATION]010, [TIME_STAMP]2016-05-18 08:52:46, [REG]4002, [OPTION]1754, [LAYER]GSDIM, [TYPE]REP, [TIME]1, [METHOD]LINE_ITEM, [ACTION]ADD, [RESPONSE]Operation SUCCESSFUL
[HOST]THOST, [LOCATION]010, [TIME_STAMP]2016-05-18 08:53:24, [REG]4002, [OPTION]1756, [LAYER]GSDIM, [TYPE]REQ,[METHOD]DEVICE, [ACTION]CUST_QUESTION

4

And how would you extract geographical data from events like these? The only geo-related field I can find is the location field but that's just a single integer.

5

Hi Magnus,

I am looking for an option to to get geo details in my log. Currently log does not have any geo location (ip, longitude and latitude etc ..) . Could you please suggest best practice her?

Will something application can include in my log file ? or Logstash can append here?

Thank your for prompt response....

6

Well, whatever thing that knows anything about the geographical location obviously needs to supply that information in the log. I have no idea what's producing your log and whether there even is a relevant geographical context so I can't make specific suggestions. Once you have some kind of useful data we can talk about how to make use of it in Logstash.

7

Hi Magnus,
Just tried to add latitude and longitude in logs for logstash.

In logstash configuration:-

included under filter:-

mutate
{
add_field =>["[geoip][location]","%{[latitude]}"]
add_field =>["[geoip][location]","%{[longitude]}"]
}

mutate
{
convert =>["[geoip][location]","float"]

}
Log file:-

[LAYER]GSDIM, [TYPE]REQ, [METHOD]SESSION, [ACTION]START, [latitude]47.60621,[longitude]-122.33207
[LAYER]GSDIM, [TYPE]REP, [TIME]1, [METHOD]SESSION, [ACTION]START, [RESPONSE]Session Started,[latitude]47.60621,[longitude]-122.33207

But still getting exception in elastic search

Caused by: java.lang.IllegalArgumentException: illegal latitude value [-122.33207] for geoip.location
at org.elasticsearch.index.mapper.geo.GeoPointFieldMapper.parse(GeoPointFieldMapper.java:123)
at org.elasticsearch.index.mapper.geo.BaseGeoPointFieldMapper.parse(BaseGeoPointFieldMapper.java:469)
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:309)
at org.elasticsearch.index.mapper.DocumentParser.parseArray(DocumentParser.java:371)
at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:254)
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:306)
at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:326)
at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:252)
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:122)

8

You've probably mixed up the order of latitude and longitude. The former are always in the range [-90, 90] while the latter are in the [-180, 180] range.

9

Thank you , really helped.

I have switched latt and long from - http://mondeca.com/index.php/en/any-place-en