Elasticsearch Index Mapping Exceeded

Hello there,

As the title suggests, our index mapping has exceeded 1,000. What happens to the logs that introduce new fields? Are they not recorded or are the 1,001+ fields just not usable at this time?

It's looking like we need about 2,500 fields, but we may split up the index into multiple indices instead.

They will be rejected as the mapping can not be updated.
You can also increase this limit but it's better probably to check if you really need so many fields or to split into multiple indices.

By rejected, do you mean they will not be stored in Elasticsearch at all? Currently, the index mapping shows there are 2,200 fields.

I figured the fields would stay in a ! state and not usable until the index mapping was fixed.

Which version?

We are on 7.1

I believe that documents should be rejected. That's how I read https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#mapping-limit-settings

@jpountz will know much better than me.

But ES has error logs saying failed to execute bulk item..... limit of total fields 1000 in index X has been exceeded.

Yeah I saw that in the documentation. I just wanted clarification that "rejected" mean the data was not stored in ES or if it is stored, but those extra fields are not usable.

Bulk rejection means that the document is not stored.

Well we increased the field limit to 3,000, but that isn't enough.

These logs are specific to all windows logs: System, Application, and Security. Does anyone know the total number of fields possible for all windows events?

What does a typical document is looking like?

Well we are shipping windows logs directly from the windows servers. We using Nxlog to send all 3 log types and all severities.

Ok. And what does a typical document is looking like?

They all look like this for the most part. Some have more fields than others since we have this issue going on, but I imagine there are probably 100k or more total windows event. I need something to pitch to the boss that says we can't log this for various reasons.

Please don't post images of text as they are hardly readable and not searchable.

Instead paste the text and format it with </> icon. Check the preview window.

Could you share a full document? It can be on gist.github.com if too big for this forum.
Thanks.

Authentication Package	  	Negotiate
t Caller Domain	  	domain
t Caller Logon ID	  	(0x0,0x3E7)
t Caller Process ID	  	444
t Caller User Name	  	dc$
t Domain	  	domain
t Logon GUID	  	-
t Logon ID	  	(0x0,0x93FD2C08)
t Logon Process	  	Advapi  
t Logon Type	  	3
t Source Network Address	  	1.1.1.1
t Source Port	  	37076
t Transited Services	  	-
t User Name	  	cssclient
t Workstation Name	  	dc
@timestamp	  	June 20th 2019, 14:47:35.941
t@version	  	1
tAccountName	  	test
tAccountType	  	User
tCategory	  	Logon/Logoff 
#CategoryNumber	  	2
tDomain	  	domain
#EventID	  	540
tEventReceivedTime	  	2019-06-20 14:47:36
tEventTime	  	2019-06-20 14:47:35
tEventTimeWritten	  	2019-06-20 14:47:35
tEventType	  	AUDIT_SUCCESS
tFileName	  	Security
tHostname	  	dc
#RecordNumber	  	-1,452,110,094
tSeverity	  	INFO
#SeverityValue	  	2
tSourceModuleName	  	in
tSourceModuleType	  	im_mseventlog
tSourceName	  	Security
t_id	  	AWt2Ny8C7-UxPwC7EoPl
t_index	  	logstash-2019.06.20
#_score	  	
t_type	  	corp_windows_events
thost	  	1.1.1.1
#port	  	1,070
ttags	  	Low
ttype	  	corp_windows_events

This is a whole log. I'm really just wanting to know if someone knows how many total fields there are for all the windows events.

Wondering why you didn't show a json document (ie the result of a GET index/_doc/id.

Anyway here you have very few fields so I don't understand where the 3000 fields are coming from.

So what is the current mapping you have?

That's why I am saying that posting a sample log isn't going to help lol

We're are currently logging ALL Windows event types and severity: Application, System, Security with Info, Warming, Verbose, Critical, etc.

I'm just wanting to know if anyone knows how many fields I should be expecting from logging this many events; if someone ran into this similar issue?

Didn't you say that when you increase the limit to 3000 you are seeing rejected documents?

Yeah because the count is still over 3,000 :smiley: probably because of the insane variety of log types.

So what is the mapping?