Some Fields of index not listed in kibana - discover- under avilable fields , , eventhough shown in in index using GET command

Elastic Stack Elasticsearch
1

Following log entries are put in index
2021-09-27T03:35:53.263029+05:30 _gateway devname="PPFW01"
2021-09-27T03:42:22.549689+05:30 _gateway devname="PPFW02"
2021-09-27T04:02:36.089881+05:30 _gateway devname="PPFW03"
2021-09-27T04:05:06.749370+05:30 _gateway devname="PPFW04"
2021-09-27T04:10:04.904638+05:30 _gateway devname="PPFW05"
2021-09-27T04:03:18.223319+05:30 _gateway devname="PPFW06" from="aditya@gmail.com" attachment="yes"
2021-09-27T04:10:26.326831+05:30 _gateway devname="PPFW02" from="contact@online.com" attachment="no"

But the fields from (from="aditya@gmail.com")and attachment (attachment="no")are not listed under available fields in when the index is viewed using kibana-> discover. but all fields are shown under the document

While using GET lp_indexttt/_search/?pretty
all fileds are shown , 2 sample entries shown below

{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 7,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [

{
"_index" : "lp_indexttt",
"_type" : "_doc",
"_id" : "g8foa3wBjuQOzRLqEY0I",
"_score" : 1.0,
"_source" : {
"agent" : {
"hostname" : "shi-Latitude-3510",
"name" : "shi-Latitude-3510",
"id" : "058ff2af-04fe-470d-bbe9-9a87caad3718",
"ephemeral_id" : "adcc3879-a556-4585-818d-295e0c2ecbf1",
"type" : "filebeat",
"version" : "7.15.0"
},
"log" : {
"file" : {
"path" : "/home/shi/logfortipartsnew/utm-emailfilshortest.log"
},
"offset" : 296
},
"message" : """2021-09-27T04:10:04.904638+05:30 _gateway devname="PPFW05" """,
"kvmsg" : """devname="PPFW05" """,
"tags" : [
"lp_tag2"
],
"input" : {
"type" : "filestream"
},
"@timestamp" : "2021-09-26T22:40:04.904638000Z",
"ecs" : {
"version" : "1.11.0"
},
"host" : {
"hostname" : "shi-Latitude-3510",
"os" : {
"kernel" : "5.11.0-37-generic",
"codename" : "focal",
"name" : "Ubuntu",
"family" : "debian",
"type" : "linux",
"version" : "20.04.3 LTS (Focal Fossa)",
"platform" : "ubuntu"
},
"containerized" : false,
"ip" : [
"192.168.1.6",
"fe80::cd8f:a19c:a6b2:2627"
],
"name" : "shi-Latitude-3510",
"id" : "3104765478f478eadec013d7ghf2c1c",
"mac" : [
"7c:76:a5:54:06:76",
"90:b6:44:76:e9:b2"
],
"architecture" : "x86_64"
},
"devname" : ""PPFW05"",
"fields" : {
"type" : "ttt"
}
}
},
{
"_index" : "lp_indexttt",
"_type" : "_doc",
"_id" : "hMfoa3wBjuQOzRLqEY0I",
"_score" : 1.0,
"_source" : {
"agent" : {
"hostname" : "shi-Latitude-3510",
"name" : "shi-Latitude-3510",
"id" : "058ff2af-04fe-470d-bbe9-9a87caad3718",
"ephemeral_id" : "adcc3879-a556-4585-818d-295e0c2ecbf1",
"type" : "filebeat",
"version" : "7.15.0"
},
"log" : {
"file" : {
"path" : "/home/shi/logfortipartsnew/utm-emailfilshortest.log"
},
"offset" : 396
},
"message" : "2021-09-27T04:03:18.223319+05:30 _gateway devname="PPFW06" from="aditya@gmail.com" attachment="yes"",
"kvmsg" : "devname="PPFW06" from="aditya@gmail.com" attachment="yes"",
"tags" : [
"lp_tag2"
],
"input" : {
"type" : "filestream"
},
"@timestamp" : "2021-09-26T22:33:18.223319000Z",
"ecs" : {
"version" : "1.11.0"
},
"attachment" : ""yes"",
"host" : {
"hostname" : "shi-Latitude-3510",
"os" : {
"kernel" : "5.11.0-37-generic",
"codename" : "focal",
"name" : "Ubuntu",
"type" : "linux",
"family" : "debian",
"version" : "20.04.3 LTS (Focal Fossa)",
"platform" : "ubuntu"
},
"containerized" : false,
"ip" : [
"192.168.1.6",
"fe80::cd8f:a19c:a6b2:2627"
],
"name" : "shi-Latitude-3510",
"id" : "3104765478f478eadec013d7ghf2c1c",
"mac" : [
"7c:76:a5:54:06:76",
"90:b6:44:76:e9:b2"
],
"architecture" : "x86_64"
},
"devname" : ""PPFW06"",
"from" : ""aditya@gmail.com"",
"fields" : {
"type" : "ttt"
}
}
},

2

Can you please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:

3

Sir,
Actually all fields were viewable, only when i changed the options in the following way

Under Kibana- Discover- after selecting the index pattern, for the field names - Filter by type - The options " Aggregatable = yes" and "Searchable = yes" was selected .
Then the fields from ([from="aditya@gmail.com")and attachment (attachment="no")were NOT listed

Only when the options " Aggregatable = any" and Searchable = any" was selected ,
Then the fields from ([from="aditya@gmail.com)and attachment (attachment="no") got listed

Now the issue is HOW to make these fields come under Aggregatable = yes" and "Searchable = yes" ?

Following are the 2 sample entries from the output of GET lp_indexttt/_search/?prettyProcessing: indexoutputtt...

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 7,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [

  {
    "_index" : "lp_indexttt",
    "_type" : "_doc",
    "_id" : "g8foa3wBjuQOzRLqEY0I",
    "_score" : 1.0,
    "_source" : {
      "agent" : {
        "hostname" : "shi-Latitude-3510",
        "name" : "shi-Latitude-3510",
        "id" : "058ff2af-04fe-470d-bbe9-9a87caad3718",
        "ephemeral_id" : "adcc3879-a556-4585-818d-295e0c2ecbf1",
        "type" : "filebeat",
        "version" : "7.15.0"
      },
      "log" : {
        "file" : {
          "path" : "/home/shi/logfortipartsnew/utm-emailfilshortest.log"
        },
        "offset" : 296
      },
      "message" : """2021-09-27T04:10:04.904638+05:30 _gateway devname="PPFW05" """,
      "kvmsg" : """devname="PPFW05" """,
      "tags" : [
        "lp_tag2"
      ],
      "input" : {
        "type" : "filestream"
      },
      "@timestamp" : "2021-09-26T22:40:04.904638000Z",
      "ecs" : {
        "version" : "1.11.0"
      },
      "host" : {
        "hostname" : "shi-Latitude-3510",
        "os" : {
          "kernel" : "5.11.0-37-generic",
          "codename" : "focal",
          "name" : "Ubuntu",
          "family" : "debian",
          "type" : "linux",
          "version" : "20.04.3 LTS (Focal Fossa)",
          "platform" : "ubuntu"
        },
        "containerized" : false,
        "ip" : [
          "192.168.1.6",
          "fe80::cd8f:a19c:a6b2:2627"
        ],
        "name" : "shi-Latitude-3510",
        "id" : "3104765478f478eadec013d7ghf2c1c",
        "mac" : [
          "7c:76:a5:54:06:76",
          "90:b6:44:76:e9:b2"
        ],
        "architecture" : "x86_64"
      },
      "devname" : "\"PPFW05\"",
      "fields" : {
        "type" : "ttt"
      }
    }
  },

  {
    "_index" : "lp_indexttt",
    "_type" : "_doc",
    "_id" : "hMfoa3wBjuQOzRLqEY0I",
    "_score" : 1.0,
    "_source" : {
      "agent" : {
        "hostname" : "shi-Latitude-3510",
        "name" : "shi-Latitude-3510",
        "id" : "058ff2af-04fe-470d-bbe9-9a87caad3718",
        "ephemeral_id" : "adcc3879-a556-4585-818d-295e0c2ecbf1",
        "type" : "filebeat",
        "version" : "7.15.0"
      },
      "log" : {
        "file" : {
          "path" : "/home/shi/logfortipartsnew/utm-emailfilshortest.log"
        },
        "offset" : 396
      },
      "message" : "2021-09-27T04:03:18.223319+05:30 _gateway devname=\"PPFW06\" from=\"aditya@gmail.com\" attachment=\"yes\"",
      "kvmsg" : "devname=\"PPFW06\" from=\"aditya@gmail.com\" attachment=\"yes\"",
      "tags" : [
        "lp_tag2"
      ],
      "input" : {
        "type" : "filestream"
      },
      "@timestamp" : "2021-09-26T22:33:18.223319000Z",
      "ecs" : {
        "version" : "1.11.0"
      },
      "attachment" : "\"yes\"",
      "host" : {
        "hostname" : "shi-Latitude-3510",
        "os" : {
          "kernel" : "5.11.0-37-generic",
          "codename" : "focal",
          "name" : "Ubuntu",
          "type" : "linux",
          "family" : "debian",
          "version" : "20.04.3 LTS (Focal Fossa)",
          "platform" : "ubuntu"
        },
        "containerized" : false,
        "ip" : [
          "192.168.1.6",
          "fe80::cd8f:a19c:a6b2:2627"
        ],
        "name" : "shi-Latitude-3510",
        "id" : "3104765478f478eadec013d7ghf2c1c",
        "mac" : [
          "7c:76:a5:54:06:76",
          "90:b6:44:76:e9:b2"
        ],
        "architecture" : "x86_64"
      },
      "devname" : "\"PPFW06\"",
      "from" : "\"aditya@gmail.com\"",
      "fields" : {
        "type" : "ttt"
      }
    }
  },

thanks and regards
shini
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.