Dec 15th, 2021: [en] UFOs in Kibana Maps

[bhavyarm]

When the call came for doing a post for the advent calendar, I signed up to write about our amazing maps capabilities in Kibana again. But, in my end-of-year escapism from real and scary things like covid and climate emergency, I decided to look for quirky data to ingest and play with in our Maps app. And of course, internet never disappoints. I found the UFOs dataset . My thanks to for @planetsig for the work on data collection and the clean up.

Once I found the dataset - I needed to ingest it into Elasticsearch. I decided to use the file upload in add integrations in Kibana home. What is nice about this is that this uses our machine learning end point to analyse data we are uploading and gives us insights. The result of this exercise is UFO Index in my local Elasticsearch with 80,260 documents in it.

Here is the mapping and ingest pipeline settings for this dataset:

Mapping:
{
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "date": {
      "type": "date",
      "format": "M/dd/yyyy||MM/dd/yyyy||MM/d/yyyy||M/d/yyyy||MM/dd/yy||MM-dd-yy||MM/dd-yy||MM-dd-yyyy||MM/d/yy||MM-d-yy||MM.dd.yy||dd/MM/yyyy||M/d/yyyy"
    },
    "field": {
      "type": "long"
    },
    "field2": {
      "type": "long"
    },
    "field3": {
      "type": "long"
    },
    "field4": {
      "type": "double"
    },
    "field5": {
      "type": "double"
    },
    "message": {
      "type": "text"
    },
    "location": {
      "type": "geo_point"
    }
  }
}

Ingest pipeline:
{
  "description": "Ingest pipeline created by text structure finder",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{DATE:timestamp} %{INT:field}:%{INT:field2},.*?%{INT:field3}.*?,.*?%{DATE:date2}.*?%{NUMBER:field4}.*?%{NUMBER:field5}.*"
        ]
      }
    },
    {
      "date": {
        "field": "timestamp",
        "timezone": "{{ event.timezone }}",
        "formats": [
          "dd/MM/yyyy",
          "dd/M/yyyy",
          "MM.dd.yy",
          "M/dd/yyyy",
          "MM/dd/yyyy",
          "M.d.yyyy",
          "M/d/yyyy"
        ]
      }
    },
    {
      "convert": {
        "field": "field",
        "type": "long",
        "ignore_missing": true
      }
    },
    {
      "convert": {
        "field": "field2",
        "type": "long",
        "ignore_missing": true
      }
    },
    {
      "convert": {
        "field": "field3",
        "type": "long",
        "ignore_missing": true
      }
    },
    {
      "convert": {
        "field": "field4",
        "type": "double",
        "ignore_missing": true
      }
    },
    {
      "convert": {
        "field": "field5",
        "type": "double",
        "ignore_missing": true
      }
    },
    {
      "remove": {
        "field": "timestamp"
      }
    },
    {
      "set": {
        "field": "location",
        "value": "{{field4}},{{field5}}"
      }
    }
  ]
}

Now let's see what we can do with this index in Kibana. This is what I see in my discover. The oldest sighting in my data set is on Nov 11, 1906. Wow. This is exciting. You can obviously do a lot in Discover in Kibana to explore and ask questions to your dataset.

From discover, what I am interested in is to explore this data which has geo locations in maps app. And I can navigate to maps app in multiple ways. Here I clicked the location field in the side bar of discover. And because this is a geospatial field with values - Voila, Kibana opens my data in a new map. Awesome, isn't it. Gives us a quick view of how our data's geo distribution looks.

Now, let's build a map with different layers in maps app. My base layer is going to have some administrative boundaries from Elastic Maps Service. On top of it I am going to layer my documents layer. This will show all the UFO documents individual location in different countries.

Here is something very cool. We have a new feature - time slider - which will let us see UFOs popping up all over the map depending on the recorded date of their sightings!!!

If I want to see density of my data, I can use the heatmap layer. We can see from the map that US has the highest number of UFO sightings.

I also changed the zoom level in my document layer so that its visible for zoom levels starting 6->24. This way I can see the density first and then zoom in to look at individual documents.

I can also use aggregations to plot big data and not overwhelm my network or browser layers. Here is a quick example using clusters and grids to layer UFOs data with document layer. I am using grids and set the zoom level of this layer to 0->5. And changed the zoom setting of document layer to 5->24. For the grids - I can see calculated values of each group of documents and then when I zoom in - grids layer disappears and I can see individual documents.

Oh, here is a question. What happened to my UFO heatmap with celestial sparkly colors? Hint: its in the tooltip in this screenshot.

Now I can to search across my layers to focus on my data. I want to find out the colors of these UFOs, shapes, was there fire?, did anybody see ? or did somebody get abducted? I can ask these questions in the search bar as free text.

• Orange color - 8857 hits
  There is always some light, sometimes like a fireball - in red, orange or bluish colors.
  

  

• Alien - 138 hits.
  An alien like ship or actual alien. Insert lots of ET references.
  

  

• Abduct - 4 hits
  Aww poor Fido. Or may be he/she is out having an adventure
  

  

This is it. Many thanks to @thomasneirynck for the review and answers and @Jeffvestal for all the help with data ingestion. Now I am off to think of Fido having a case of zoomies while lording over an alien spaceship.