Hi
First of all I want to say that I am completely new to the ELK stack, but I have a project to test it out. Short about my project is that I am pulling real time data about arrivals and departures from five of the largest airports in Norway. This is dynamic data which I update every five min with updates about delays, new gates etc. I update this from a SQL like this:
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://localhost:3306/avinor"
jdbc_user => "user"
jdbc_password => "pass"
jdbc_driver_library => "/xxx/xxx/xxx/mysql-connector-java-5.1.41/mysql-connector-java-5.1.41-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
statement => "SELECT * FROM temp_BGO"
}
}
output {
stdout { codec => json_lines }
elasticsearch {
"hosts" => "localhost:9200"
"index" => "bgo"
"document_type" => "data"
"document_id" => "%{flight}"
}
}
One of the columns in this database is called "iata", and holds the IATA code of the airport (e.g. LAX for Los Angeles) the flight is either departing for, or arriving from.
I also have a file which contains deeper information about all the airports in the world. It looks like this:
3484,"Los Angeles International Airport","Los Angeles","United States","LAX","KLAX",33.94250107,-118.4079971,125,-8,"A","America/Los_Angeles","airport","OurAirports"
What I want is to enrich my real time data for my five airports with information from this file with full name, Country, coordinates etc etc. I could of course do this in SQL, but I was wondering if there is a smarter way to do this in the ELK stack?
Is there a way to make some sort of pointer in my real time data pointing to the database of airport, or would I have to merge it in to elasticsearch? The database containing information about the airports is static, so no need to update that.
Any input would be very much appreciated.
Thanks!