Estimation for ELK setup?

[sunilmchaudhari (Sunil Chaudhari)]

Hello,
can some expert help me to estimate in man/days, to setup ELK stack in one evnironment?

Data on-boarding of system matrix from 40 clients.
Writing Groke patterns for 5 different patterns
Configuration of Logstash-Kibana-ES
Installation of x-pack
Creating visualizations: Host overview, System-Overview using Guage/bar charts/Timeseries using Visual builder
and some ad-hoc tasks and unforeseen issues

How much time I need to do this all?
I want expert opinion considering some practical issues they faced in their experience. Please comment.

br,
Sunil.

[warkolm (Mark Walkom)]

Is this for a customer of yours?

[rcowart (Rob Cowart)]

This is almost impossible to answer as it depends a lot on how truly "expert" the person doing the work is, and how much of the solution the expert has "pre-developed".

For example we recently deployed a five node ES cluster, with a 3 node Kafka cluster (incl. 3 node Zookeeper) on the ingest side, and a single node monitoring instance, tapped into a datafeed from Azure IoT Hub and built the first dashboards... all done by one person in a single day. To do so requires a lot of experience with the platforms, and a lot of preconfigured bits to start from.

The same goes for handling data sources. Take something as supposedly simple as syslog. Our parsing of the just the header portion of a syslog message can handle all the most common structures you will see. But it also handles 30+ "wierd ways vendors abuse syslog". So in just a few minutes we can setup syslog handling that will reliably handle a wide variety of messages thrown at it, where others could spend a day or two just figuring out all the variations of timestamp that they will have to deal with.

In case you are wondering, this is the regex for all of the variations of timestamp we have seen...

(?:(?:\b(?:[Jj]an(?:uary|uar)?|[Ff]eb(?:ruary|ruar)?|[Mm](?:a|ä)?r(?:ch|z)?|[Aa]pr(?:il)?|[Mm]a(?:y|i)?|[Jj]un(?:e|i)?|[Jj]ul(?:y)?|[Aa]ug(?:ust)?|[Ss]ep(?:tember)?|[Oo](?:c|k)?t(?:ober)?|[Nn]ov(?:ember)?|[Dd]e(?:c|z)(?:ember)?)\b +(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) (?!<[0-9])(?:2[0123]|[01]?[0-9]):(?:[0-5][0-9])(?::(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?))(?![0-9])( +(?>\d\d){1,2})?|\b(?:[Jj]an(?:uary|uar)?|[Ff]eb(?:ruary|ruar)?|[Mm](?:a|ä)?r(?:ch|z)?|[Aa]pr(?:il)?|[Mm]a(?:y|i)?|[Jj]un(?:e|i)?|[Jj]ul(?:y)?|[Aa]ug(?:ust)?|[Ss]ep(?:tember)?|[Oo](?:c|k)?t(?:ober)?|[Nn]ov(?:ember)?|[Dd]e(?:c|z)(?:ember)?)\b +(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) (?>\d\d){1,2} (?!<[0-9])(?:2[0123]|[01]?[0-9]):(?:[0-5][0-9])(?::(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?))(?![0-9]))|(?>\d\d){1,2}-(?:0?[1-9]|1[0-2])-(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])[T ](?:2[0123]|[01]?[0-9]):?(?:[0-5][0-9])(?::?(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?))?(?:Z|[+-](?:2[0123]|[01]?[0-9])(?::?(?:[0-5][0-9])))?)\s+

So it will depend a lot on the scale required, how truly expert the person doing the work is, and how much of the solution might pre-exist.

[sunilmchaudhari (Sunil Chaudhari)]

This is required for me. I want to give estimation and later need to justify to superiors who don't know the stack.