CloudFlare Log Ingest (Grok?)

[wwalker (Walker)]

Anybody ever setup Logstash to ingest CloudFlare API pulls? I'm not sure how to go about setting up grok patterns to match the output. Additionally, the returned data is in NDJson format, which I think is a double edged sword...or maybe not.

Anyways, is there a good place that I can go that breaks down each grok pattern so I can understand which to use? Below are a couple sample return lines to give you an idea of what's returned.

I have no pipeline config setup yet because there's a couple ingest methods I need to investigate and output will be to elasticsearch so that's pretty straightforward.

{"ClientIP":"2600:6c40:4200:1e:0:baf7:d740:a599","ClientRequestHost":"host.domain.TLD","ClientRequestMethod":"GET","ClientRequestURI":"SomLongURI","EdgeEndTimestamp":1535026490200000000,"EdgeResponseBytes":2589,"EdgeResponseStatus":200,"EdgeStartTimestamp":1535026170210000128,"RayID":"44ed787bdfd69ec3"}
{"ClientIP":"207.160.98.250","ClientRequestHost":"host.domain.TLD","ClientRequestMethod":"PUT","ClientRequestURI":"SomeLongURI","EdgeEndTimestamp":1535026425190000128,"EdgeResponseBytes":14364,"EdgeResponseStatus":200,"EdgeStartTimestamp":1535026228378000128,"RayID":"44ed79e75cd79d97"}

[magnusbaeck (Magnus Bäck)]

This is JSON so you just use a json codec or json filter. Don't use a grok filter for this.

[wwalker (Walker)]

doh....that made this sooo much easier, lol.

[wwalker (Walker)]

In case anyone else runs across wanting to pull CloudFlare Enterprise logs, here's the pipeline I have. I have a PowerShell script running as a scheduled task that does the API call and saves the results to the ingest directory.

input {
  file {
    path => "d:/ElasticStack/Ingest/CloudFlare/*.txt"
    start_position => beginning
  }
}
filter {
  json {
    source => "message"
  }
  mutate {
    id => "Field addition"
    add_field => {
      "Full URL" => "https://%{ClientRequestHost}%{ClientRequestURI}"
    }
  }
}
output {
  elasticsearch {
    id => "Send to Elasticsearch"
    hosts => ["1.2.3.4:9200"]
    index => "cloudflare-%{+YYYY.MM.dd}"
  }
}