My team is new to Elasticsearch and is considering it for log analysis. I need to be able to tell my manager whether the technology can scale to meet our needs. So that's my question: Can Elasticsearch meet our goals at scale?
- Ingest logs from ~150 servers, each generating ~2GB of logs per day
- Events are interlaced in the logs and need to be aggregated (see below example).
- We would like a live view of the events for an operations team to monitor. If only the
startline of an event has come through, in Kibana, we want the event to show a
durationfield that is the difference between now and the start time. As other lines from the event come through, the event should be updated with more detail. Once the
endline has come through, the
durationshould become the difference between the end time and the start time.
Example log format:
sameTaskId, start, username,size sameTaskId,networkstats anotherTaskId, start, username,size sameTaskId, end, authAuditInfo anotherTaskId,networkstats anotherTaskId, end, authAuditInfo
We have considered the following
- Logstash aggregate filter - I don't think this will work because it requires limiting to a single worker thread. This means we can't easily scale. Also, it would hold onto the event until the
endcame through, so we wouldn't get the "live" view we want.
- Logstash Elasticsearch filter - Logstash could look up the
startevent and update it when the
endline comes through. But what if they are received out of order? I assume the lookup would fail. This approach is also very chatty.
- Elasticsearch continuous transforms - This is what I'm leaning toward. It sounds like it doesn't have the same inherent scaling concerns as the other options. The main concern is how "live" the data can be in Kibana.
Primary Question: Can Elasticsearch meet our goals?
- Can continuous transforms keep up with the data and provide the "live" view we want?
- Some events are only milliseconds long. Will transforms handle the case where the
endline arrives before the