Unable to see many custom index fields in Kibana Discover

[ozonshak]

Hello. I'm attempting to load a new index into Elastic and Kibana. I have loaded it into ElasticSearch and I can see my fields just fine. Here is an example where I run a query against Dev Tools:

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "wcs_prod_ihs-2020.06",
        "_type" : "Prod",
        "_id" : "t8yjwnIBufjMecmPqK-B",
        "_score" : 1.0,
        "_source" : {
          "agent" : {
            "version" : "7.7.1",
            "ephemeral_id" : "d159fc96-asdf-asdf-asdf-03ff4eca3610",
            "hostname" : "SERVER00",
            "id" : vafdvasda9df-413d-832a-aasdvasdvas",
            "type" : "filebeat"
          },
          "TrueClientIP" : "###.##.##.##",
          "input" : {
            "type" : "log"
          },
          "@version" : "1",
          "message" : "###masked###",
          "ecs" : {
            "version" : "1.5.0"
          },
          "httpRequest" : "GET /request/@self HTTP/1.1",
          "httpTime" : "2020-06-14-00.06.53",
          "httpRequestMethod" : "GET",
          "tags" : [
            "beats_input_codec_plain_applied"
          ],
          "type" : "apache_access",
          "@timestamp" : "2020-06-14T04:06:53.000Z",
          "httpReferer" : "https://www.google.com",
          "host" : {
            "architecture" : "x86_64",
            "hostname" : "SERVER00",
          },
          "httpResponseTime" : 16546,
          "httpUserAgent" : "Mozilla/5.0 (iPhone; CPU iPhone OS 13_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Mobile/15E148 Safari/604.1",
          "geoip" : {
            "region_code" : "FL",
            "continent_code" : "NA",
            "ip" : "###.##.##.##",
            "dma_code" : 528,
            "postal_code" : "33055",
            "region_name" : "Florida",
            "country_code3" : "US",
            "latitude" : 25.9487,
            "longitude" : -80.2819,
            "country_code2" : "US",
            "city_name" : "Opa-locka",
            "location" : {
              "lon" : -80.2819,
              "lat" : 25.9487
            },
            "country_name" : "United States",
            "timezone" : "America/New_York"
          },
          "httpResponseCode" : 200,
          "log" : {
            "offset" : 1345522,
            "file" : {
              "path" : "/access_log.202006140000"
            }
          },
          "httpResponseSize" : 221,
          "httpHostIP" : "###.##.##.##",
          "httpXForwarded" : "###.##.##.##6",
          "httpJSessionID" : "asdfasdf",
          "httpWCSMemberID" : "-",
          "fields" : {
            "log_type" : "weblogs"
          }
        }
      },

However, after I create the index pattern from the Elastic Index and choose a time field (only providing me with the @timestamp option), I then go up to "Discover" and just look at everything in the index. When I do this, I'm not seeing most of the fields above that I can see via Dev Tools. For example, I don't see any of the field like httpResponseSize, httpTime, httpHostIP, TrueClientIP, etc. When I expand the list of available fields, they are there. But, they are not showing up because Kibana doesn't think there is any data in those fields.

Any idea what I'm doing wrong here? Thanks!

[tylersmalley]

Are you sure that those documents contain that data? If you expand the row is the data there? You can take the information from that _id and _index and grab the document directly from Elasticsearch to verify. On discover we're really just making a search and displaying the documents as they are returned.

[ozonshak]

None of my custom fields show up with data. I forced a show of the fields as columns just to make sure. So, Discover doesn’t think there is data. However I see the data when running a query in Dev tools (as shown in my screen shot). Any idea why Discover doesn’t see data in these fields in the same index? Let me know if you need to see other details.

[tylersmalley]

Would you mind expanding one of those rows on Discover which don't contain the data and get the _index and _id field. I would like to verify that that document in particular doesn't have data.

Often times when comparing against raw search requests people see different documents since the time picker isn't applied.

[ozonshak]

Sure, the below screen shot shows one of them expanded. Looks like it is only the standard host and agent fields and such.

So, I'm definitely getting data populated in the index. This is a brand new index I'm populating for the first time with a single web access log file. The timestamp is wrong however. When creating the index pattern, my only option was @timestamp, which seems to default to the time I imported the data, not when the log entries were actually made. Was going to look into that after trying to figure out why my other fields are not being populated.

[tylersmalley]

The option of which property are available for the time series data is based on the mapping of the index pattern. It's giving you any which are a date type. I am not sure how you are creating the mapping and ingesting the data, but I would take a look at that.

If you open up "Dev Tools" then "Console" in Kibana and run:

GET wcs_prod_ihs-2020.06/J9ILw3IBufjMecmPSH1v

Do you see the httpTime or TrueClientIP properties there?

While you're there, you can inspect the mapping with GET wcs_prod_ihs-2020.06/_mapping

[ozonshak]

Ok, so I have a little more information here. After running a few more tests, I can see field mappings in Discover. Looks like all the bad ones (missing all custom fields) are sorted to the top so all I saw initially was bad entries. If I reverse the sort on the time column, I see some data populated:

For the group that doesn't seem to work, I'm seeing this tag: "_grokparsefailure". So, I'm guessing some of these web log entries don't match any of my groc patterns (I'm using 3 patterns). I'm going to have to look into that one.

For the time series issue I mentioned above, I'm ingesting logs sent over by FileBeat. Filebeat is sending them into Logstash. In Logstash, I have a pipeline setup with the following filter:

date {
          locale => "en"
          match => ["httpTime", "yyyy-MM-dd-HH.mm.ss"]
          timezone => "US/Eastern"
       }

Any idea why this isn't showing up for me as an option when creating the index pattern in Kibana? The "httpTime" field is the one I want to use for the time series since this is the actual timestamp of the log entry.