Logstash to Splunk HTTP Event Collector

[sm00thindian (Kilynn W)]

I'm trying to use logstash to send data directly to an http event collector (HEC). The HEC collector accepts the following correctly. But I'm trying to translate the to the appropriate HTTP Output config for logstash.

curl -k https://hec.example.com:8088/services/collector/event -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" -d '{"event": "hello world"}'
{"text": "Success", "code": 0}

Has anyone had success with this?

-krw

[sm00thindian (Kilynn W)]

Nothing huh... Was it because I said the S word?

[rcowart (Rob Cowart)]

There may not be a lot of folks here who use Logstash with Splunk. I can't tell you much about the HTTP output. Splunk did recently announce improved support ingesting from Kafka (about time!), so perhaps you could go...

logstash -> kafka -> splunk

Of course many of us here would encourage you to just store the data in Elasticsearch instead of Splunk.

[sm00thindian (Kilynn W)]

I was able to successfully send data directly to the HEC HTTP Event Collector with the following settings. Keys were to use the 'raw' input and to have a valid certificate for the destination. I'm sure a JKS trusted store would work as well. HEC expects JSON, and make sure acknowledgement is off on the HEC side.

filter {
        ruby { code => "event.set('time', event.get('@timestamp').to_f)" }
        mutate {
                rename => { "message" => "event" }
        }
}
output {
            http {
                    format => "json"
                    content_type => "application/json"
                    http_method => "post"
                    url => "https://xxx.org:8088/services/collector/raw"
                    cacert => "/****valid***cert_ca.pem"
                    headers => ['Authorization', 'Splunk 6e2e58ef-fe02-49de-9fd9-xxxxxxxxxxxxxxxxxxx']
            }
    }

The ruby code adds 'time' as epoch in float format and by changing the name of the message to event the receiving collector indexes it automatically.