As the title suggests, our index mapping has exceeded 1,000. What happens to the logs that introduce new fields? Are they not recorded or are the 1,001+ fields just not usable at this time?
It's looking like we need about 2,500 fields, but we may split up the index into multiple indices instead.
They will be rejected as the mapping can not be updated.
You can also increase this limit but it's better probably to check if you really need so many fields or to split into multiple indices.
But ES has error logs saying failed to execute bulk item..... limit of total fields 1000 in index X has been exceeded.
Yeah I saw that in the documentation. I just wanted clarification that "rejected" mean the data was not stored in ES or if it is stored, but those extra fields are not usable.
Well we increased the field limit to 3,000, but that isn't enough.
These logs are specific to all windows logs: System, Application, and Security. Does anyone know the total number of fields possible for all windows events?
They all look like this for the most part. Some have more fields than others since we have this issue going on, but I imagine there are probably 100k or more total windows event. I need something to pitch to the boss that says we can't log this for various reasons.
Authentication Package Negotiate
t Caller Domain domain
t Caller Logon ID (0x0,0x3E7)
t Caller Process ID 444
t Caller User Name dc$
t Domain domain
t Logon GUID -
t Logon ID (0x0,0x93FD2C08)
t Logon Process Advapi
t Logon Type 3
t Source Network Address 1.1.1.1
t Source Port 37076
t Transited Services -
t User Name cssclient
t Workstation Name dc
@timestamp June 20th 2019, 14:47:35.941
t@version 1
tAccountName test
tAccountType User
tCategory Logon/Logoff
#CategoryNumber 2
tDomain domain
#EventID 540
tEventReceivedTime 2019-06-20 14:47:36
tEventTime 2019-06-20 14:47:35
tEventTimeWritten 2019-06-20 14:47:35
tEventType AUDIT_SUCCESS
tFileName Security
tHostname dc
#RecordNumber -1,452,110,094
tSeverity INFO
#SeverityValue 2
tSourceModuleName in
tSourceModuleType im_mseventlog
tSourceName Security
t_id AWt2Ny8C7-UxPwC7EoPl
t_index logstash-2019.06.20
#_score
t_type corp_windows_events
thost 1.1.1.1
#port 1,070
ttags Low
ttype corp_windows_events
This is a whole log. I'm really just wanting to know if someone knows how many total fields there are for all the windows events.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.