First Mappings are the "Schema" if you do do create one with a template a default will be created for you. If you are going to do anything at scale you should learn about them.
In your sample ingest if you want to see the mapping
GET myindex/
You are using a default template right now.. .which is OK but super not efficient, we can come back to that later.
So I just ran your complete setup above with this log file
2021/01/31,00:00:02:00,0-0,0,0,0,0,0,0,05338000,78000109,D2880027,4C15262E,00010000,7002C080,9,2,0,0,2.850
2021/01/31,00:00:03:00,0-0,0,0,0,0,0,0,05338001,78000109,D2880027,4C15262E,00010001,7002C080,9,2,0,0,2.850
2021/01/31,00:00:04:00,0-0,0,0,0,0,0,0,05338002,78000109,D2880027,4C15262E,00010002,7002C080,9,2,0,0,2.850
2021/01/31,00:00:05:00,0-0,0,0,0,0,0,0,05338003,78000109,D2880027,4C15262E,00010003,7002C080,9,2,0,0,2.850
2021/01/31,00:00:06:00,0-0,0,0,0,0,0,0,05338004,78000109,D2880027,4C15262E,00010004,7002C080,9,2,0,0,2.850
Everything work great... ingest no errors
GET fb-hist-h00-2021.09.04/_search
{
"took" : 5,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 5,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "fb-hist-h00-2021.09.04",
"_type" : "_doc",
"_id" : "TlQssnsBLtlbFaz6r6fI",
"_score" : 1.0,
"_source" : {
"date" : "2021/01/31",
"nDispatchTaskStartedBy" : "0",
"agent" : {
"hostname" : "ceres",
"name" : "T0031",
"id" : "a19f917f-52b5-4121-b287-5828e596c1b1",
"ephemeral_id" : "f2db4956-8af4-4b9d-9a61-d9499a1f49f8",
"type" : "filebeat",
"version" : "7.14.0"
},
"log" : {
"file" : {
"path" : "/Users/sbrown/workspace/sample-data/discuss/sample-csv-log-mhare.log"
},
"offset" : 0
},
"ModuleBLowDIO" : "7002C080",
"task_id" : "0",
"ModuleA32BitInput" : "05338000",
"task_timer" : "2.850",
"ModuleB32BitInput" : "D2880027",
"RecoveryTriggersActive" : "0",
"datetime" : "2021/01/31 00:00:02:00",
"ModuleB32BitOutput" : "4C15262E",
"ecs" : {
"version" : "1.10.0"
},
"host" : {
"name" : "T0031"
},
"BDispatchAbortReq" : "0",
"nTurboComPhase" : "2",
"nDispatchTaskEndCode" : "0",
"ModuleA32BitOutput" : "78000109",
"task_bool" : "0",
"RecoveryRunning" : "0",
"message" : "2021/01/31,00:00:02:00,0-0,0,0,0,0,0,0,05338000,78000109,D2880027,4C15262E,00010000,7002C080,9,2,0,0,2.850",
"ModuleBHighDIO" : "00010000",
"input" : {
"type" : "log"
},
"@timestamp" : "2021-01-31T00:00:02.000Z",
"nErMonErrorNumber" : "9",
"time" : "00:00:02:00",
"fields" : {
"type" : "H00"
},
"box_running" : "0-0",
"nDispatchState" : "0"
}
},
....
NOW I just see you say your logs are in Unicode heheheh ... are your logs unicode you need to figure out the right encoding
and set in the log input
See the list but I would try this first
encoding : utf-8
I have spent accumulated probably months / years on date time programming