Cannot import json template for cloudfront logs to cluster

Elastic Stack Logstash
1

Hi,
I'm attempting to get my cloudfront logs into a local elasticsearch cluster (7.8.0) and am having a bit of trouble.
The template I'm attempting to import is from here
https://aws.amazon.com/premiumsupport/knowledge-center/cloudfront-logs-elasticsearch/

#cloudfront.template.json
    {
      "template": "cloudfront-logs-*",
      "mappings": {
        "logs": {
          "_source": {
            "enabled": false
          },
          "_all": {
            "enabled": false
          },
          "dynamic_templates": [
            {
              "string_fields": {
                "mapping": {
                  "index": "not_analyzed",
                  "type": "string"
                },
                "match_mapping_type": "string",
                "match": "*"
              }
            }
          ],
          "properties": {
            "geoip": {
              "dynamic": true,
              "properties": {
                "ip": {
                  "type": "ip"
                },
                "location": {
                  "type": "geo_point"
                },
                "latitude": {
                  "type": "float"
                },
                "longitude": {
                  "type": "float"
                }
              }
            }
          }
        }
      }
    }

so I've created an index:
curl -XPUT "http://elasticsearch:9200/cloudfront-logs"
and attempting to add the template:
curl -XPUT "http://elasticsearch:9200/_template/cloudfront-logs/" -H "Content-Type: application/json" -d "@cloudfront.template.json" | jq .
gives me the following

      "error": {
        "root_cause": [
          {
            "type": "mapper_parsing_exception",
            "reason": "Root mapping definition has unsupported parameters:  [logs : {_source={enabled=false}, dynamic_templates=[{string_fields={mapping={index=not_analyzed, type=string}, match_mapping_type=string, match=*}}], _all={enabled=false}, properties={geoip={dynamic=true, properties={ip={type=ip}, latitude={type=float}, location={type=geo_point}, longitude={type=float}}}}}]"
          }
        ],
        "type": "mapper_parsing_exception",
        "reason": "Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [logs : {_source={enabled=false}, dynamic_templates=[{string_fields={mapping={index=not_analyzed, type=string}, match_mapping_type=string, match=*}}], _all={enabled=false}, properties={geoip={dynamic=true, properties={ip={type=ip}, latitude={type=float}, location={type=geo_point}, longitude={type=float}}}}}]",
        "caused_by": {
          "type": "mapper_parsing_exception",
          "reason": "Root mapping definition has unsupported parameters:  [logs : {_source={enabled=false}, dynamic_templates=[{string_fields={mapping={index=not_analyzed, type=string}, match_mapping_type=string, match=*}}], _all={enabled=false}, properties={geoip={dynamic=true, properties={ip={type=ip}, latitude={type=float}, location={type=geo_point}, longitude={type=float}}}}}]"
        }
      },
      "status": 400
    }

Which I'm guessing is something in the template it doesn't like.
Can anyone please tell me what it is?..
Most of my elastic experience has been with maintaining the indexes for bitbucket (since v5 with ES2.4), so I've got the basics. Now I'm attempting to get all the systems logging into the cluster via logstash.

Thanks,
Sean

2

Document types are no longer supported, so remove the "logs" { } from the template. It should be

  "mappings": {
      "_source": {
        "enabled": false
      },

etc.

3

Aha. Thanks for that.
The following was accepted:

{
  "template": "cloudfront-logs-*",
  "mappings": {
      "dynamic_templates": [
        { 
          "string_fields": {
            "mapping": {
              "index": "not_analyzed",
              "type": "string"
            },
            "match_mapping_type": "string",
            "match": "*"
          }
        }
      ],
      "properties": {
        "geoip": {
          "dynamic": true,
          "properties": {
            "ip": {
              "type": "ip"
            },
            "location": {
              "type": "geo_point"
            },
            "latitude": {
              "type": "float"
            },
            "longitude": {
              "type": "float"
            }
          }
        }
      }
    }
}

I took out the _all section otherwise it complained

    "type": "mapper_parsing_exception",
    "reason": "Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [_all : {enabled=false}]",

The awkward thing about the elastic stack is that anything that isn't the official documentation tends to get out of date rather quickly...

4

Right, so with a bit of fiddling with the cloudfront.conf from the AWS page, I've for the thing to start properly, but it's giving me the following, which looks like something is missing from the fields configuration?.. (probably the formentioned _doc type)

[2020-07-13T18:45:15,857][WARN ][logstash.outputs.elasticsearch][main][cefe51442f6e53d4f864dc28b757357504201f5497fb99198a7aeae7093b2e6c] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"cloudfront-logs-2020.07", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x2f6de1cc>], :response=>{"index"=>{"_index"=>"cloudfront-logs-2020.07", "_type"=>"_doc", "_id"=>"vXt-SXMB-vn8vFtw1BH3", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to find type parsed [string] for [path]"}}}}

I'm happy to figure this out myself if someone could point me in the right direction.

Thanks,
Sean

5

Wow, that template must be old. The string type was removed and replaced by text and keyword in 5.0. The blog post explaining that is here.

6

The AWS page says "Last updated: 2019-05-22", but it is also talking about logstash 5.5., so... :wink:
Do I (just) need to remove and re-add the updated index/template with the updated parameters?..

7

I think you can just overwrite the template with an updated version.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.