In my ES 7.17 cluster, I'm using index templates to create a new weekly index where I send log data. The index names are generated by my application, not by ES; the template handles index patterns, field limits, and aliases.
I modified the template to decrease the number of shards I was creating for each index, and I must have unknowingly made some other change to the index template, because new indices are now rejecting documents with this error:
{"took":2,"ingest_took":0,"errors":true,"items":[{"index":{"_index":"nlog-2022.38_le","_type":"logevent","_id":"HbAuTYMBsEj06M5VHS-G","status":400,"error":{"type":"illegal_argument_exception","reason":"Invalid type: expecting [_doc] but got [logevent]"}}},{"index":{"_index":"nlog-2022.38_le","_type":"logevent","_id":"HrAuTYMBsEj06M5VHS-G","status":400,"error":{"type":"illegal_argument_exception","reason":"Invalid type: expecting [_doc] but got [logevent]"}}}]}
The key part apparently being Invalid type: expecting [_doc] but got [logevent]".
The documents look like:
{"index":{"_index":"nlog-2022.38","_type":"logevent","pipeline":"MainIngestionPipeline"}}
{"@timestamp":"2022-09-17T22:06:36.4183000","level":"Info","message":"2022-09-17 15:06:36.4183|INFO|StartupLogger|Initializing User Interface","MachineName":"BSN0000","Application":"MainUI","LogLevel":"INFO","Source":"StartupLogger","Callsite":"StacktraceHere","LogMessage":"Initializing User Interface","ElapsedMs":7707}
I realize that format looks a little weird, but I'm having trouble capturing exactly what NLog, my logging library, is spitting out. This has all been working for a long time: the data being sent and the pipeline (which appends _le to the index name, for logevent) have not changed, and the only change I know I made lately is to the index template. I can send other, simpler documents to the new index without issue.
Additionally, if I send this type of entry to a previous week's index, it still gets indexed correctly, leading me to believe there's something in my new index settings that is wrong. I can read the error and see that it has something to do with being a logevent instead of a _doc, but then why does it still work, in other indices? If I
GET /nlog-2022.38_le/_settings it looks exactly like the previous week's settings.
What other settings should I be looking at to determine what's wrong, or what else could be happening here? The cluster is otherwise healthy. Any help or suggestions are much appreciated.