Dec 12th, 2017: [IT][ElasticStack] Monitorare Bitcoin&co con Elastic Stack


(Antonio Bonuccelli) #1

Bitcoin e le "crypto-valute" in generale continuano a riscuotere molto interesse per gli addetti ai lavori ma non solo.

Oggi vedremo come poter monitorare le performance di queste valute con Elastic Stack e CoinMarketCap API

Vogliamo anche far uso di una nuova fantastica feature di Logstash che ci consente di modificare le nostre pipelines direttamente da Kibana UI senza dover riavviare Logstash dopo ogni modifica :sunglasses::rocket:

Per abilitare remote management e dichiarare una pipeline, aggiungiamo la configurazione seguente in logstash.yml:

#remote management
xpack.management.enabled: true
xpack.management.elasticsearch.url: "https://es-node:9200"

#autenticazione
xpack.management.elasticsearch.username: logstash_admin
xpack.management.elasticsearch.password: password
xpack.management.logstash.poll_interval: 5s 
xpack.management.elasticsearch.ssl.ca: "/path/to/ca/ca.crt"
xpack.management.elasticsearch.ssl.keystore.path: "/path/to/node/cert.p12"
xpack.management.elasticsearch.ssl.keystore.password: "secretpassword"

#la nostra pipeline
xpack.management.pipeline.id: ["coins"]

Usiamo Logstash Input http_poller per accedere ai dati offerti da CoinMarketCap per ottenere i dati sulle crypto valute:

per referenza

input {
        http_poller {
            urls => {
                btc => "https://api.coinmarketcap.com/v1/ticker/bitcoin/"
                ltc => "https://api.coinmarketcap.com/v1/ticker/litecoin/"
                eth => "https://api.coinmarketcap.com/v1/ticker/ethereum/"
                neo => "https://api.coinmarketcap.com/v1/ticker/neo/"
                lisk => "https://api.coinmarketcap.com/v1/ticker/lisk/"
                btcc => "https://api.coinmarketcap.com/v1/ticker/bitcoin-cash/"
                ada => "https://api.coinmarketcap.com/v1/ticker/cardano/"
                ethc => "https://api.coinmarketcap.com/v1/ticker/ethereum-classic/"
                globaldata => "https://api.coinmarketcap.com/v1/global/"
            }
        request_timeout => 60
        schedule => { cron => "*/1 * * * * UTC"}
        codec => "json"
        }
}

filter {
         
}

output {
        stdout{codec=>rubydebug}
        elasticsearch{
                hosts => "https://es-node:9200"
                index => "coin-metrics-%{+YYYY.MM.dd}"
                user => "logstash_writer"
                password => "password"
                ssl => true
                cacert => "/path/to/ca.crt"    
        }
}

In Elasticsearch utilizziamo un dynamic template
per assicurarci che i dati vengano indicizzati con i tipi di dato adatti per poter eseguire tutte le aggregations piΓΊ interessanti:

PUT _template/coin
{
  "order": 0,
  "index_patterns": [
"coin*"
  ],
  "settings": {
"index": {
  "number_of_shards": "1"
}
  },
  "mappings": {
"doc": {
  "dynamic_templates": [
    {
      "usd_as_double": {
        "match_mapping_type": "string",
        "match": "*_usd",
        "mapping": {
          "type": "double"
        }
      }
    },
    {
      "eur_as_double": {
        "match_mapping_type": "string",
        "match": "*_eur",
        "mapping": {
          "type": "double"
        }
      }
    },
    {
      "pct_as_double": {
        "match_mapping_type": "string",
        "match": "percent_change_*",
        "mapping": {
          "type": "double"
        }
      }
    },
    {
      "price_as_double": {
        "match_mapping_type": "string",
        "match": "price_*",
        "mapping": {
          "type": "double"
        }
      }
    },
    {
      "rank_as_long": {
        "match_mapping_type": "string",
        "match": "rank",
        "mapping": {
          "type": "long"
        }
      }
    },
    {
      "supply_as_long": {
        "match_mapping_type": "string",
        "match": "*_supply",
        "mapping": {
          "type": "long"
        }
      }
    }
  ]
}
  }
}

In pochi minuti abbiamo creato una dashboard con diverse visualizzazioni:

i prossimi passi potrebbero consistere nello sfruttare questi dati tramite Timelion , configurare qualche alert per notificarci via slack o eseguire un'altra azione...


(Tat Dat Pham) #2

Hi @nellicus, i have one question with Coin prices over time chart.
What kind of aggregation in Y-Axis?. Thanks!


(Antonio Bonuccelli) #3

Hi @tatdat that's single metric max agg run on price_usd
hth


(Tat Dat Pham) #4

Thanks @nellicus.


(Mark Walkom) closed #5