Problem with terms -- they are ALL raw

Elastic Stack Kibana
1

I am trying desperately to understand how the ELK stack works. I spent several weeks setting up a rather complicated set of regex filters so that logstash could pass along useful data to elasticsearch. Now I am in Kibana trying to set up a visualization and none of my data looks anything like any of the examples in the tutorials.

When I go to do a term aggregation, I only have a single date option (@timestamp) and the rest are all raw strings in spite of the fact that many of the logstash filters I applied are built in. Here is the json of one of my messages to perhaps shed some light on what I'm doing wrong. If you look at the fields, some of them are very obviously dates. Others are very obviously numbers. Help!

{
  "_index": "logstash-2017.11.30",
  "_type": "my-type",
  "_id": "AWAO_ARPZ7pGkwr72PRo",
  "_score": null,
  "_source": {
    "body_bytes_sent": "172",
    "remote_address": "111.222.333.444",
    "http_response_metadata": "|502|172|-|-|-|0.002|0.000, 0.001|",
    "message": "Oct  6 04:51:44 serverName nginx-access: 111.222.333.444|-|-    |06/Oct/2017:04:51:43 +0000|GET /theApp/api/|502|172|-|-|-|0.002|0.000, 0.001|",
    "type": "my-type",
    "http_verb": "GET",
    "http_user_agent": "-",
    "forwarded_for": "-",
    "path": "/var/local/test-logs/alb/nginx-access.log",
    "log_type": "nginx-access:",
    "@timestamp": "2017-11-30T22:11:25.323Z",
    "request_time": "0.002",
    "local_time": "Oct  6 04:51:44",
    "http_referer": "-",
    "@version": "1",
    "host": "localhost.localdomain",
    "request_path": "/theApp/api/",
    "http_request": "GET /theApp/api/",
    "upstream_response_time": "0.000",
    "host_name": "serverName",
    "status": "502"
  },
  "fields": {
    "@timestamp": [
      1512079885323
    ]
  },
  "sort": [
    1512079885323
  ]
}
2

Hi Brandon,

Now that you're getting the data you want through logstash, the next step is to make sure the mapping of that data to specific data types is correct. Elasticsearch will apply some default data types but we often have to help by setting fields to specific types.

To see how your fields are currently mapped (and to see what you might have to define) in Kibana go to Dev Tools > Console, and do a request like GET logstash-*/_mapping

Then I'll refer you to this blog about mappings; https://www.elastic.co/blog/logstash_lesson_elasticsearch_mapping

And if you Google logstash set mapping you'll find a lot of information and tutorials. But if you get stuck let me know and I'll help.

Regards,
Lee

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.