Just started using Elastic Stack after hearing rave reviews from friends in the security industry. I used the free trial thing just to have a prod around the interface before setting my own instance up.
Got a few questions I'm hoping that someone here can answer:
Is there a way of integrating LogStash into the Kibana GUI? I've looked through documents and videos and I can't find a single thing talking about this anywhere. It's OK if not, I just want to understand so that I can stop looking. The only reason why I ask is because the interface on the 'hosted' platform has a section for Log Stash under Management -> Stack Management. Perhaps I've missed something, but if someone can either confirm/deny and then point me at the correct resources, I'd appreciate it.
Integration of scripting/API collection. Part of my use case is to collect OSINT from forums, twitter, and such. Obviously Twitter has it's own specific integration, but if I wanted to scrape data from a forum via the use of a script - how would I achieve this? The script is being worked on by a colleague, so I don't have it to hand, but it is being written in Python with some Regex. The output of the script will be .JSON - my reading leads me to believe that this is the best format for ingestion into LogStash/Elasticsearch. If this isn't correct, please correct me here also.
Ingesting the output of this data into ElasticSearch is then the next step. I assume that any entry will require a key field of some description - which will usually be a date/timestamp. If we ingest the output of the script mentioned in step 2, say once per day, it will just create a series of new entries and not overwrite the old data? Just thinking for trend analysis.
Thanks for any help.