Need help disabling field type casting for computer forensics logs

Elastic Stack Elasticsearch
1

Hi,

While currently working on developing a tool for computer forensics relying on Elasticsearch, I am facing several issues related to field types.

The tool is parsing data using Vector, and sends it directly to Elasticsearch via API credentials.

Unfortunately, forensics logs sent to Elasticsearch can be very uncertain and tend to generate many type-related errors:

  • Some date fields may be empty, and a quick workaround for this precise type is to drop empty fields in vector before sending it to Elasticsearch to deal with the "date field cannot be empty"-type errors.
  • However, in numerous cases, some fields may have multiple, unpredictable types, such as the "logontype" or the "user" field as illustrated below :

Logontype-related error (field can be text or long):

image
image3794×137 179 KB

User-related error (field may contain {} characters, which from my understanding is the source of the Elasticsearch error):
\"reason\":\"[1:902] failed to parse field [user] of type [text] in document with id 'dmflS5UBruhBQKkAW60B'. Preview of field's value: '{domain=NT AUTHORITY}'

Is there anything you would recommend in the index configuration or Elasticsearch-wide configuration to resolve any type-related issues and make sure all types are accepted by default, even if it means fewer builtin functions may be user on the ingested logs later?
For operational purposes, the usage of vector to send data directly to Elasticsearch is preferable.

Every advice and piece of information on this matter is more than welcome!

Many thanks, any help is appreciated.

2

I'd never heard of Vector, but Vector has its own elasticsearch sink? Are you using it? Could you?

I very much doubt there is a "accept anything we send to it" combination of setting in elasticsearch that can "resolve any type-related issues and make sure all types are accepted by default". In its default config, it uses dynamic mappings anyways, so it will try its best unless thats been changed for your indices.

btw, the image you attached was unreadably tiny for me on my phone, please just cut and paste the text. If you can share examples of data that was sent to elasticsearch then we have better chance to understand why it was not accepted.