How does the concept of a Splunk sourcetype within an index translate to ES?

Elastic Stack Elasticsearch
1

So coming from Splunk, there are various sourcetypes within an index, presumably without the limitations ES has on _type and the fields needing to be uniform. How does the concept of a sourcetype translate to ES if you have several different sources of data you want to be able to search on within an index? I'll give a brief example:

Let's say you have an index called Windows, and within it you have three different kinds of logs- security, system, and application. The logs have some fields in common but also several not in common. ES recommends in the blog to just make a field called "type" (instead of _type) so that it can be filtered on in search. So it would be something like "index = Windows and type=System"

Does that sound like the best way to do it?

2

Yep, that's pretty much the idea!

Design choices as opposed to limitations :slight_smile:
Splunk does index on read, which is great for just throwing data into a system, but not as performant when it comes to wanting to do that reading (ie analysis, alerting, etc) in real time on large datasets.
Putting structure on the data as it goes in makes sure that when you want to access it (again, analysis, alerting etc) it's as realtime as it can be, irrespective of the size. That's what Elasticsearch does when it indexes data in.

Ultimately both systems do the same work, they just position that unit of work at different parts of the pipeline and optimise for different experiences.

1 Like
3

Thanks!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.