Mapping Issues With GeoIP

Elastic Stack Logstash
1

I'm at a complete loss. I'm using 5.6 and literally nothing I do changes how coordinates get mapped. They seem to be mapping automatically to whatever elastic thinks it is while entirely ignoring my mapping.

My mapping is too long to show in its entirety, but keep in mind I have been keeping the template name matching with what I index. I'm aware of common issues arising from that. I made sure to create the index AND store the mapping before actually shipping any data to the index. This still somehow didn't solve my issue. If I populate the location field with float values the resulting type shown in kibana is number. If a string it shows as a string. All while the mapping is of type geo_point.

Here is the relevant portion of my filter as of now:

filter {

mutate {
add_field => {"[geoip][city_name]" => "Seoul"}
add_field => {"Point of Contact" => "Jim : 940-255-1234"}
}

ruby {

init => "
require 'csv'
"
code => "
csv_text = File.read('/simplemaps-worldcities-basic.csv')
csv = CSV.parse(csv_text, :headers => true)
found_row = csv.find{|row| row['city'] == event.get('[geoip][city_name]')}

event.set('[client_geoip][location]', [found_row['lat'].to_f, found_row['lng'].to_f])
"
}
2

So [client_geoip][location] is the field you want to have mapped as geo_point? What are the current mappings of that field? What does your template look like? Feel free to omit other fields for brevity.

3

Was trying to hold off getting back to you until I solved my issue and I could get back on topic, but I had to pretty much roll back to my original mappings to get everything up and running again. Nothing drastic changed, but something odd is happening now. Nothing is showing in my discovery tab.

yellow open .kibana             lZ9rkfLeR8yi2gETZe-I5A 1 1   3 0 36.2kb 36.2kb
yellow open gee222              Pu49Z_2DTRiE1cMKYcnoNg 5 1 989 0  4.3mb  4.3mb
yellow open filebeat-2018.01.30 4MVGETiNSxi7ql0921rqmA 5 1 989 0  2.6mb  2.6mb

That is what I get when I do a cat indices. Both are obviously being updated as I can watch them grow in size.

{
  "took": 13,
  "timed_out": false,
  "_shards": {
   "total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
  },
   "hits": {
   "total": 989,
"max_score": 1,
    "hits": [
  {
    "_index": "gee222",
    "_type": "log",
    "_id": "AWFHcDG3pIRzKq69BX-t",
    "_score": 1,
    "_source": {
      "audit_suid": "0",
      "audit_terminal": "?",
      "process_id": "5081",
      "audit_fp": "SHA256:88:22:c3:d0:eb:df:35:3c:a6:dd:10:54:b8:06:80:dc:93:02:fe:d7:b8:da:47:fa:5e:45:97:76:35:09:b7:fc",
      "source": "/var/log/audit/audit.log",
      "type": "log",
      "audit_addr": "?",
      "result": "success",
      "audit_direction": "?",
      "@version": "1",
      "beat": {
        "name": "client1",
        "hostname": "client1",
        "version": "5.6.3"
      },
      "host": "client1",
      "client_geoip": {
        "location": "37.5663491,126.999731"
      },
      "audit_msg": [
        "audit(1517321527.470:163504):",
        """op=destroy kind=server fp=SHA256:88:22:c3:d0:eb:df:35:3c:a6:dd:10:54:b8:06:80:dc:93:02:fe:d7:b8:da:47:fa:5e:45:97:76:35:09:b7:fc direction=? spid=5081 suid=0  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success"""
      ],
      "geoip": {
        "city_name": "Seoul"
      },
      "offset": 2955560,
      "Point of Contact": "Tom Perry:940-255-1234",
      "audit_kind": "server",
      "input_type": "log",
      "audit_op": "destroy",
      "session_id": "4294967295",
      "message": """type=CRYPTO_KEY_USER msg=audit(1517321527.470:163504): pid=5081 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:88:22:c3:d0:eb:df:35:3c:a6:dd:10:54:b8:06:80:dc:93:02:fe:d7:b8:da:47:fa:5e:45:97:76:35:09:b7:fc direction=? spid=5081 suid=0  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'""",
      "syslog_message": """type=CRYPTO_KEY_USER msg=audit(1517321527.470:163504): pid=5081 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:88:22:c3:d0:eb:df:35:3c:a6:dd:10:54:b8:06:80:dc:93:02:fe:d7:b8:da:47:fa:5e:45:97:76:35:09:b7:fc direction=? spid=5081 suid=0  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'""",
      "audit_exe": "/usr/sbin/sshd",
      "tags": [
        "beats_input_codec_plain_applied"
      ],
      "audit_subj": "system_u:system_r:sshd_t:s0-s0:c0.c1023",
      "audit_uid": "4294967295",
          "@timestamp": "2018-01-30T14:12:08.604Z",

Above is what I get from performing a _search on the gee222 index. I cut it off intentionally at the timestamp as I think that might be the issue and the rest isn't really relevant. Clearly it's being stored. What is throwing me off is the timestamp is now somehow in the future by a few hours, but even setting the absolute range from five years ago to two months in the future doesn't give results on either filebeat or gee222.

4

What timezone are you in? Keep in mind that @timestamp is UTC.

5

ET. I've known about the time difference. I'm confused as to how the timestamp randomly jumped to 2PM when it was 9:30AM ET at the time. Regardless of that, setting my time range to:
From: 2017-12-01 00:00:00.000 To: 2019-04-06 23:59:59.999
Still didn't display anything for either index.

6

Metrics are 100% working with the same time range as the discovery tab. Discovery tab still isn't showing anything.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.