Adding added field to index

Elastic Stack Elasticsearch
1

i use the docker elk stack on macos with the logstash http pipeline to accept our apps http posts as input

it created an index (I didn't before)

reviewing the content, there are two fields that combined could make a geopoint, so I added that field thru the web UI. and I can use the visualizer to see the map of the points..

now I want my external app to use the qeopoint filter query on that field

 result = await client.search({
    index: 'logs-*', 
    "query": {
      "bool": {
        "must": {
          "match_all": {}
        },
          "filter" : {
              "geo_bounding_box" : {
                  "navigate_location" : {  // <-  the manually created field
                      "top_left" : {
                          "lat" : 34.73,
                          "lon" : -98.00
                      },
                      "bottom_right" : {
                          "lat" : 30.717,
                          "lon" : -97.030
                      }
                  }
              }
          }
      }
  }

when I query on other fields, I get the data record

2023 @ 11: 18:28.383event.original{
  "devices": [],
  "end": 1685981908383,
  "retrycount": 0,
  "type": 1,
  "status": "canceled",
  "appVersion": "1.0",
  "location": {
    "coords": {
      "altitude": 235.09720251895487,
      "longitude": -97.628558119899495,   // part of constructed point
      "altitudeAccuracy": 1.9524757439985525,
      "speed": 0.021378767720873718,
      "latitude": 30.463607267669115,      / part of constructed point
      "heading": -1,
      "accuracy": 7.068291962625123
    },
    "timestamp": 1685981907000.032
  },
  "logtype": "navigate",
  "phoneinfo": {
    "manufacturer": "Apple",
    "platform": "iOS",
    "version": "16.5",
    "model": "iPhone13,2"
  },
  "start": 1685981884953
}

but not the constructed field ..
when I try to add it to the index thru the developers console I get an error, already exists
when I dump out the mappings I see

        "navigate_location": {
          "type": "geo_point"
        },

but I cant get it by name

get logs-generic-default/_mapping/navigate_location

yields

{
  "error": "no handler found for uri [/logs-generic-default/_mapping/navigate_location?pretty=true] and method [GET]"
}

altho it just showed be the datatype mapping in that index...

how do I resolve this..

2

if I do this with a more specific index name

result = await client.search({
      index: 'logs-generic-default', 
    "query": {
      "bool": {
        "must": {
          // "match_all": { }
          "term": { "logtype": "navigate" }
        },
          "filter" : {
              "geo_bounding_box" : {
                  "navigate_location" : {
                      "top_left" : {
                      "lat": 30.6627347,
                      "lon": -97.922001
                      },
                      "bottom_right" : {
                          "lat" : 30.415,
                        "lon": -97.0185702
                      }
                  }
              }
          }
      }

i get a successful search, but no hits

{
  took: 7,
  timed_out: false,
  _shards: { total: 1, successful: 1, skipped: 0, failed: 0 },
  hits: { total: { value: 0, relation: 'eq' }, max_score: null, hits: [] }

at least its not an error on the field...

BUT the data says there should be 100's of hits.. with this quadrant

3

a little more data

the field has a script (because the actual data isn't an object with lat/lon)

if(doc['location.coords.longitude'].size()!= 0){
    emit ( doc['location.coords.latitude'].value ,doc['location.coords.longitude'].value)
}

on the previous field, i forced the datatype via a put to the mappings, and destroyed the script.. and now no data..

BUT no error..

I just created another field with the script (via the UI).. and have data, but the new field is NOT in the mappings, so the search fails

"failed to find geo field [fribble]"
4

You can't get a specific field for mappings, only the entire thing.

It'd help if you shared the mapping and a sample document you are using here, as you're referencing things we can't guess at.

5

thanks.. using the UI, discover tab, here is a document record in the selected timeframe window, with the original content field as well..

{
  "@timestamp": [
    "2023-06-05T16:18:28.765Z"
  ],
  "@version": [
    "1"
  ],
  "appVersion": [
    "1.0"
  ],
  "data_stream.dataset": [
    "generic"
  ],
  "data_stream.namespace": [
    "default"
  ],
  "data_stream.type": [
    "logs"
  ],
  "database_post_delay": [
    2850303289
  ],
  "devices.address": [
    "97CAB758-A70B-9867-D2D5-4C678EF5BD1D"
  ],
  "devices.sequence": [
    5
  ],
  "devices.subtype": [
    "10"
  ],
  "devices.td": [
    2367
  ],
  "devices.type": [
    1
  ],
  "end": [
    1683131605476
  ],
  "end_date": [
    "2023-05-03T16:33:25.476Z"
  ],
  "event.original": [
    "{\"start\":1683131605305,\"logtype\":\"scanresult\",\"end\":1683131605476,\"devices\":[{\"address\":\"97CAB758-A70B-9867-D2D5-4C678EF5BD1D\",\"sequence\":5,\"subtype\":\"10\",\"type\":1,\"td\":2367}],\"phoneinfo\":{\"manufacturer\":\"Apple\",\"version\":\"16.4.1\",\"model\":\"iPhone13,2\",\"platform\":\"iOS\"},\"location\":{\"coords\":{\"altitude\":234.23440742492676,\"heading\":-1,\"latitude\":30.463546480177587,\"longitude\":-97.628650949809725,\"altitudeAccuracy\":10.748244285583496,\"speed\":-1,\"accuracy\":35},\"timestamp\":1683131599474.551},\"type\":1,\"status\":\"success\",\"appVersion\":\"1.0\",\"retrycount\":0}"
  ],
  "fribble": [
    {
      "coordinates": [
        -97.62865447998047,
        30.463546752929688
      ],
      "type": "Point"
    }
  ],
  "host.ip": [
    "172.22.0.1"
  ],
  "http.method": [
    "POST"
  ],
  "http.request.body.bytes": [
    "640"
  ],
  "http.request.mime_type": [
    "application/json"
  ],
  "http.version": [
    "HTTP/1.1"
  ],
  "location.coords.accuracy": [
    35
  ],
  "location.coords.altitude": [
    234.2344
  ],
  "location.coords.altitudeAccuracy": [
    10.748244
  ],
  "location.coords.heading": [
    -1
  ],
  "location.coords.latitude": [
    30.463547
  ],
  "location.coords.longitude": [
    -97.628654
  ],
  "location.coords.speed": [
    -1
  ],
  "location.timestamp": [
    1683131600000
  ],
  "logtype": [
    "scanresult"
  ],
  "navigate_elapsed": [
    171
  ],
  "phoneinfo.manufacturer": [
    "Apple"
  ],
  "phoneinfo.model": [
    "iPhone13,2"
  ],
  "phoneinfo.platform": [
    "iOS"
  ],
  "phoneinfo.version": [
    "16.4.1"
  ],
  "retrycount": [
    0
  ],
  "start": [
    1683131605305
  ],
  "start_date": [
    "2023-05-03T16:33:25.305Z"
  ],
  "status": [
    "success"
  ],
  "type": [
    1
  ],
  "url.path": [
    "/"
  ],
  "url.port": [
    3000
  ],
  "user_agent.original": [
    "App/1 CFNetwork/1408.0.4 Darwin/22.5.0"
  ],
  "_id": "__BajIgBAgeSes04TwTH",
  "_index": ".ds-logs-generic-default-2023.04.30-000001",
  "_score": null
}

note the variable fribble..

i created the variable in the discover tab using this

Screenshot 2023-06-09 at 8.22.17 AM
Screenshot 2023-06-09 at 8.22.17 AM1057×772 104 KB

and using discover I can see the data
---- sorry, new user cannot insert more than one media item .... see next post

so, now I want to use that field in the geo_bounding_box filter from an external app

"filter" : {
              "geo_bounding_box" : {
                  "fribble" : {  // <-  the manually created field
                      "top_left" : {

the document record indicates it is in index .ds.logs-generic-default-
when I dump the mappings for the old navigate_location in that index I see

        "navigate_location": {
          "type": "geo_point"
        },

but I don't know if I'm supposed to see the script in this output

(sorry, trying lots of different things before coming here )
(force insert mapping, .... however fribble is NOT shown in the mappings output of this index

the execution of the query returns (for either field)

error={"name":"ResponseError","meta":{"body":{"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:43] [bool] unknown field [navigate_location]"}],"type":"x_content_parse_exception","reason":"[1:43] [bool] unknown field [navigate_location]"},"status":400},"statusCode":400,"headers":{"x-elastic-product":"Elasticsearch"
...
6

here is the other media item using the discover to view the variable data, so the script is successful
Screenshot 2023-06-09 at 8.24.21 AM

7

here is the node js test app

'use strict'

const { Client } = require('@elastic/elasticsearch')
const client = new Client({
  node: "http://localhost:9200",
  auth: {
    username:   use as appropriate
    password: 
  },
})

async function run () {

  // Let's search!
let result
  try{
    result = await client.search({
    index: 'logs-generic-default', 
    "query": {
      "bool": {
        "must": {
           "match_all": { }
        },
        "filter": {
          "geo_bounding_box": {
            "navigate_location": {  // or "fribble"    fribble gets variable not found, navigate returns no hits
              "top": 30.6627347,
              "bottom": 30.415,
              "left": -97.922001,
              "right": -97.0185702
            }
          }
        }  
      }
  }


  })
 } catch(error){
    console.log("error="+JSON.stringify(error));
 }
 finally {
  console.log(result)
    const data = result.rows.map(row => {
    const obj = {}
    for (let i = 0; i < row.length; i++) {
       obj[result.columns[i].name] = row[i]
    }
    return obj
  })
 }
}

run().catch(console.log)
8

after looking at some more examples while trying to solve another problem, I came to this, which does exactly what I want, and I don't have to fiddle with the index..

'use strict'

const { Client } = require('@elastic/elasticsearch')
const client = new Client({
  node: "http://localhost:9200",
    auth: {
        username:   as appropriate 
        password: 
    }
})

async function run () {

  // Let's search!
let result
  try{
    result = await client.search({
        index: 'logs-generic-default', 
        "runtime_mappings": {
          "navigate_location": {
            "type": "geo_point",
            "script": {
              "source":
                "if(doc['location.coords.longitude'].size()!= 0){emit(doc['location.coords.latitude'].value, doc['location.coords.longitude'].value)}"
            }
          }
        },
        "query": {
          "bool": {
            "must": {
              "term": { "logtype": "navigate" }
            },
            "filter": {
              "geo_bounding_box": {
                "navigate_location": {  
                  "top": 30.6627347,
                  "bottom": 30.415,
                  "left": -97.922001,
                  "right": -97.0185702
                }
              }
            }
          }
        },
        "fields": ["navigate_location"]
      }
  )
 } catch(error){
    console.log("error="+JSON.stringify(error));
 }
 finally {
      console.log(result)
      let i = 1;
      const data = result.hits.hits.forEach(row => {
        const obj = {}
        console.log("result document "+i++)
        Object.keys(row).forEach(field => {
        
            console.log("field=" + field + " value=" + JSON.stringify(row[field]))
        })
        console.log("\n")
    })
}

run().catch(console.log)
9

Oh, is the fribble a runtime field?

10

yes, just trying to make another field...certain to not be in the index

11

Runtime fields defined in Kibana like this do not show in the mappings as it's applied at the data view level. After you define the runtime field you should be able to see a little info icon and text popup when you open the data view in Stack Management, like this one;
SCR-20230614-khku

You can also check this with GET kbn:/api/data_views/data_view/<view-id>.

If you want this to be available for all requests to the index in Elasticsearch then you need to update the mapping for the index, which can be done in Dev Tools.

12

I've made a PR to make this clearer in the docs Update manage-data-views.asciidoc by markwalkom · Pull Request #159636 · elastic/kibana · GitHub

13

thank you for that

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.