Nested mapping issues

Elastic Stack Elasticsearch
1

Hi ,

Below are my mappings

PUT _template/prod 
    {
      "index_patterns": ["*"],
      "settings":{
        "codec": "best_compression",
        "number_of_shards": 1,
        "number_of_replicas": 1
      },
      "mappings": {
        "doc": {
          "_size": {
            "enabled": true
          },
          "properties": {
            "dst_ip": {
              "type": "ip"
            },
            "src_ip": {
              "type": "ip"
            },
            "notification": {
              "properties": {
                "threatInfo": {
                  "properties": {
                    "indicators": {
                      "type": "nested",
                      "properties": {
                        "indicatorName": {
                          "type": "text"
                        },
                        "sha256Hash": {
                          "type": "text"
                        },
                        "applicationName": {
                          "type": "text"
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }

After indexing the document, i dont see its properly nested.

22%20AM
Screen Shot 2019-05-01 at 11.46.22 AM.png956×461 56.5 KB

2

Kindly let me know, whats the mistake i m doing.

3

I do not see what is wrong. Could you please elaborate a bit more on what you are expecting? These fields look like keywords, so are you sure you want to map them as text?

1 Like
4

@Christian_Dahlqvist . those are keywords. I m expecting the below results.

"indicators": [
{
"indicatorName": ["CODE_DROP", "POLICY_DENY"],
"applicationName": ["cbe.exe", "pcdr.exe"],
"sha256hash": ["gfyefhjefhjfb", "hhjbdshjbdshcbds"]
}
]

So to get the above result, do i need to change the mapping of field from text to keyword.

5

The mappings will not change the structure of the document, just determine how it is indexed. Specifying a nested structure means that each item in the list will be indexed as a separate document behind the scenes, meaning that you can search for combinations of fields within list items. If the mapping was not nested data would be indexed like per field without separating the different list items and a search for indicatorName of CODE_DROP and applicationName of powershell.exe would give a match even though these values are in different list items.

1 Like
6

my intention is to display all the item in the list of each field in the grafana table panel.

7

Here i am not going to search for the specific indicatorName or applicationName. Instead i am going to list down all applicationName / indicatorName.

8

I have no experience with Grafana, so will unfortunately not be able to help you much there.

9

the reason, why i mention grafana is that.. there is screenshot, which shows the field cant display the items. i m trying to display all the items from that field. Unfortunately items cant be display due to nested type. if its properly nested, i can show all items.

10

Below after indexing.

{
        "_index": "test_2019.05",
        "_type": "doc",
        "_id": "puD1cGoBPzIHdg4Oi9G4",
        "_score": 1,
        "_source": {
          "@timestamp": "2019-05-01T01:15:01.194Z",
          "success": true,
          "@version": "1",
          "notifications": [
            {
              "threatInfo": {
                "indicators": [
                  {
                    "indicatorName": "CODE_DROP",
                    "sha256Hash": "d3f8fade829d2b7bd596c4504a6dae5c034e789b6a3defbe013bda7d14466677",
                    "applicationName": "powershell.exe"
                  },
                  {
                    "indicatorName": "POLICY_DENY",
                    "sha256Hash": "d82f393de1fb61b106f379469590e22eae15ae1915991145021b92118967191a",
                    "applicationName": "pcdrwi.exe"
                  },
                  {
                    "indicatorName": "BYPASS_POLICY",
                    "sha256Hash": "d3f8fade829d2b7bd596c4504a6dae5c034e789b6a3defbe013bda7d14466677",
                    "applicationName": "powershell.exe"
                  },
                  {
                    "indicatorName": "RUN_BLACKLIST_APP",
                    "sha256Hash": "d82f393de1fb61b106f379469590e22eae15ae1915991145021b92118967191a",
                    "applicationName": "pcdrwi.exe"
                  }
                ],
                "time": 1556673132944,
                "threatCause": {
                  "actorProcessPPid": "8580-1556619043118-476",
                  "originSourceType": "WEB",
                  "reputation": "ADAPTIVE_WHITE_LIST",
                  "causeEventId": "1c35f3606bae11e9a078dd735ff1db25",
                  "threatCategory": "NON_MALWARE",
                  "reason": "T_RUN_BLACK",
                  "actor": "ccae2d265bfe1f43bf92e71c74740c80a26b6aa6b20e85d97cbd44ccd2127e5d",
                  "actorType": null,
                  "actorName": ""
                },
                "summary": "****",
                "score": 4,
                "incidentId": "O8BGEJIE"
              },
              "type": "THREAT",
              "url": "**********",
              "eventDescription": "******\n",
              "eventTime": 1556673072860,
              "deviceInfo": {
                "deviceId": 482262,
                "groupName": "****",
                "deviceHostName": null,
                "externalIpAddress": "****",
                "deviceName": "Tstjhoa",
                "deviceType": "WINDOWS",
                "internalIpAddress": "*****",
                "deviceVersion": "Windows 10 x64",
                "email": "tstjhoa",
                "targetPriorityType": "MEDIUM",
                "targetPriorityCode": 0
              },
              "ruleName": "Threat priority greater than 1"
            }
          ]
}

my intention is to , while accessing the field "notifications.threatInfo.indicators.indicatorName" must give all items in the list.

11

This sounds like more of client-side problem (specifically, Grafana) and how it handles hierarchical JSON documents. As @Christian_Dahlqvist outlined, mapping options only dictate how we index the content for search and analytics - we never change the JSON source that is presented and preserve the same structure when it is returned in its raw form.

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.