ES gives me 400 for my index template but its seems to be validate


(red der) #1

Logstash can connect to the cluster fine and I can query http://mycluster/_template/ so it looks like the 400 is just about something in the template it doesn't like.

I get this error:
2018-02-17T00:35:11,571][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/iis-request-log
[2018-02-17T00:35:11,703][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'http://mycluster:80/_template/iis-request-log'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", ....

Im running ES 6.0

Here is my template, its valid json, not sure how to debug, ES gives me no useful info on what it doesn't like about it:
{
"template": "logstash-iis*",
"settings": {
"number_of_shards": 5,
"number_of_replicas": 1
},
"aliases": {
"iis-request-logs": {}
},
"order": 12,
"mappings": {
"default": {
"dynamic": false,
"_all": {
"enabled": false
},
"properties": {
"@timestamp": {
"type": "date",
"include_in_all": false
},
"@version": {
"type": "keyword",
"include_in_all": false
},
"bytesReceived": {
"type": "long"
},
"bytesSent": {
"type": "long"
},
"clientIP": {
"type": "ip"
},
"cookie": {
"type": "text",
"norms": false
},
"userAgent_device": {
"type": "keyword"
},
"userAgent_name": {
"type": "keyword"
},
"userAgent_os": {
"type": "keyword"
},
"userAgent_os_name": {
"type": "keyword"
},
"geoip": {
"dynamic": "true",
"properties": {
"city_name": {
"type": "keyword"
},
"continent_code": {
"type": "keyword"
},
"country_code2": {
"type": "keyword"
},
"country_code3": {
"type": "keyword"
},
"country_name": {
"type": "keyword"
},
"dma_code": {
"type": "long"
},
"ip": {
"type": "ip"
},
"latitude": {
"type": "half_float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "half_float"
},
"postal_code": {
"type": "keyword"
},
"region_code": {
"type": "keyword"
},
"region_name": {
"type": "keyword"
},
"timezone": {
"type": "keyword"
}
}
},
"host": {
"type": "ip"
},
"log_timestamp": {
"type": "keyword"
},
"message": {
"type": "text",
"norms": false
},
"method": {
"type": "keyword"
},
"port": {
"type": "long"
},
"protocolVersion": {
"type": "keyword"
},
"referer": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword"
}
}
},
"requestHost": {
"type": "keyword"
},
"response": {
"type": "keyword"
},
"serverIP": {
"type": "ip"
},
"serverName": {
"type": "keyword"
},
"serviceName": {
"type": "keyword"
},
"subresponse": {
"type": "keyword"
},
"tags": {
"type": "text",
"norms": false
},
"timetaken": {
"type": "long"
},
"type": {
"type": "keyword"
},
"uriQuery": {
"type": "text",
"norms": false
},
"uriStem": {
"type": "keyword"
},
"userAgent": {
"type": "keyword"
},
"username": {
"type": "keyword"
},
"win32response": {
"type": "keyword"
},
"xForwardedFor": {
"type": "text",
"norms": false
},
"xForwardedForProto": {
"type": "text",
"norms": false
},
"xForwardedForSrc": {
"type": "ip",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
}
}


(Ranjith M) #2

If I am not wrong in es 6 , this is not template any more but pattern.


(Robert Cowart) #3

Remove the definition to the _all field and all include_in_all attributes.

This should work...

{
  "template": "logstash-iis*",
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1
  },
  "aliases": {
    "iis-request-logs": {}
  },
  "order": 12,
  "mappings": {
    "doc": {
      "dynamic": false,
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "@version": {
          "type": "keyword"
        },
        "bytesReceived": {
          "type": "long"
        },
        "bytesSent": {
          "type": "long"
        },
        "clientIP": {
          "type": "ip"
        },
        "cookie": {
          "type": "text",
          "norms": false
        },
        "userAgent_device": {
          "type": "keyword"
        },
        "userAgent_name": {
          "type": "keyword"
        },
        "userAgent_os": {
          "type": "keyword"
        },
        "userAgent_os_name": {
          "type": "keyword"
        },
        "geoip": {
          "dynamic": "true",
          "properties": {
            "city_name": {
              "type": "keyword"
            },
            "continent_code": {
              "type": "keyword"
            },
            "country_code2": {
              "type": "keyword"
            },
            "country_code3": {
              "type": "keyword"
            },
            "country_name": {
              "type": "keyword"
            },
            "dma_code": {
              "type": "long"
            },
            "ip": {
              "type": "ip"
            },
            "latitude": {
              "type": "half_float"
            },
            "location": {
              "type": "geo_point"
            },
            "longitude": {
              "type": "half_float"
            },
            "postal_code": {
              "type": "keyword"
            },
            "region_code": {
              "type": "keyword"
            },
            "region_name": {
              "type": "keyword"
            },
            "timezone": {
              "type": "keyword"
            }
          }
        },
        "host": {
          "type": "ip"
        },
        "log_timestamp": {
          "type": "keyword"
        },
        "message": {
          "type": "text",
          "norms": false
        },
        "method": {
          "type": "keyword"
        },
        "port": {
          "type": "long"
        },
        "protocolVersion": {
          "type": "keyword"
        },
        "referer": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          }
        },
        "requestHost": {
          "type": "keyword"
        },
        "response": {
          "type": "keyword"
        },
        "serverIP": {
          "type": "ip"
        },
        "serverName": {
          "type": "keyword"
        },
        "serviceName": {
          "type": "keyword"
        },
        "subresponse": {
          "type": "keyword"
        },
        "tags": {
          "type": "text",
          "norms": false
        },
        "timetaken": {
          "type": "long"
        },
        "type": {
          "type": "keyword"
        },
        "uriQuery": {
          "type": "text",
          "norms": false
        },
        "uriStem": {
          "type": "keyword"
        },
        "userAgent": {
          "type": "keyword"
        },
        "username": {
          "type": "keyword"
        },
        "win32response": {
          "type": "keyword"
        },
        "xForwardedFor": {
          "type": "text",
          "norms": false
        },
        "xForwardedForProto": {
          "type": "text",
          "norms": false
        },
        "xForwardedForSrc": {
          "type": "ip",
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}

Hopefully that helps.

Rob

Robert Cowart (rob@koiossian.com)
www.koiossian.com
True Turnkey SOLUTIONS for the Elastic Stack


(red der) #4

Thank you!!

I should have stated I used this template with 5.4 previously and I am know trying to use it with 6.0.

I did some googling after your post and found those fields are no longer valid. Thanks for looking at this for me!


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.