i am trying to ingested csv file into elasticsearch via logstash and i have a dedicated schema for it. my data gets ingested for date fields where format is not specified and with format specified , it throws that into DLQ. Could i make these field optional(3 fields are populated based on condition and not simultaneously) if blank and get the rest the fields ingested in document and only throw it in dlq if the format is invalid?
Elasticsearch is a NoSQL database, meaning that you can index data dynamically. You can index several documents in the same index that have different structure while there's no type conflicts between them.
You should avoid to have sparse data for performance and agregation purpose but you can have some field added or missing.
The mapping allow to cast type to fileds but doesn't make them mandatory.
while for any other type its fine , but for date field it doesn't seem so.
Yes, but logstash can put the indexation date by default
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.