I am trying to get all the hits from a specific index over the past 24 hours. I entered the following command into the kibana dev console, but no hits are being returned.
I know the index is healthy, because its "Docs count" in the Index Management tab is increasing. Also, other GET commands that don't specify a date range are able to return hits, but I suspect that it is older hits being returned. This date range issue only started a few days ago, this index was acting total normal before now. I would also like to note that no recent (past 48 hours) hits are being for the index pattern on the kibana discover tab as well. Does any know how I can fix this issue? I would appreciate any and all help.
hmm, I ran the command and the timestamp of the record is 2021-12-27T14:14:17.292Z (Eastern Standard Time I presume). So I guess my index isn't actually getting anymore hits, which is weird because the index health is showing as "green" and the "Docs count" and "Storage size" is increasing at a steady pace, which I took as meaning that records are being added to the index. Would anyone happen to know the meaning behind this disparity?
I've taken a look at our Logstash logs, and I'm seeing a bunch of these warnings for my index elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"my-index", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0xcfb514b>], :response=>{"index"=>{"_index"=>"my-index", "_type"=>"_doc", "_id"=>"6s4GB34B21stRv-of0ss", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Limit of total fields [1000] has been exceeded"}}}}
So it appears that the record that induced this warning has over 1000 fields. My index should actually have ~300 fields, so I don't know where these extra fields are coming from. Are random fields being added to index hits a known problem?
To see the number of fields that my index has, I ran GET /my-index/_mapping and did a ctrl-f for the word "type", which returned 984 results. There seems to have been a dynamic field blowup for whatever reason. I have now turned off dynamic mapping for my index, but my GET date range queries still aren't returning new records. Should I have does something else after turning off dynamic mapping?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.