Hello,
I am trying to send data from Logstash(6.2.3) to influxDB(1.7.5) using Logstash-influxDB plugin.
I tried to send 10 lines of logs where 7 lines went through and 3 lines didn’t get into InfluxDB. I tried to figure out whats the odd part but couldn’t figure out the issue.
LOGS sent:
20190524193000|18|REQUEST|10.194.20.31|cctg-npm-deployer|GET|/api/npm/cctg-npm-npm/iconv-lite|HTTP/1.1|304|0
20190524193000|26|REQUEST|10.194.20.31|cctg-npm-deployer|GET|/api/npm/cctg-npm-npm/grunt-legacy-util|HTTP/1.1|200|0
20190524193000|3|REQUEST|10.194.20.31|cctg-npm-deployer|GET|/api/npm/cctg-npm-npm/colors/-/colors-0.6.2.tgz|HTTP/1.1|200|6289
20190524193000|30|REQUEST|10.194.20.31|cctg-npm-deployer|GET|/api/npm/cctg-npm-npm/js-yaml|HTTP/1.1|304|0
20190524193000|7|REQUEST|10.194.20.31|cctg-npm-deployer|GET|/api/npm/cctg-npm-npm/iconv-lite/-/iconv-lite-0.2.11.tgz|HTTP/1.1|200|235160
20190524193000|0|REQUEST|10.25.79.236|non_authenticated_user|GET|/api/docker/docker-login/v2/|HTTP/1.1|401|0
20190524193000|5|REQUEST|10.25.79.236|anonymous|GET|/api/docker/docker-login/v2/token|HTTP/1.1|200|0
20190524193000|5|REQUEST|173.39.45.237|content-security-builds-deployer|GET|/api/storage/content-security-builds-snapshot/ESA/phoebe-12-1-br/|HTTP/1.1|200|0
20190524193000|4|REQUEST|10.194.20.31|cctg-npm-deployer|GET|/api/npm/cctg-npm-npm/which/-/which-1.0.9.tgz|HTTP/1.1|200|2285
Logs didnot go through (3 logs didnot go through)
20190524193000|4|REQUEST|10.194.20.31|cctg-npm-deployer|GET|/api/npm/cctg-npm-npm/nopt/-/nopt-1.0.10.tgz|HTTP/1.1|200|10240
20190524193000|26|REQUEST|10.194.20.31|cctg-npm-deployer|GET|/api/npm/cctg-npm-npm/grunt-legacy-util|HTTP/1.1|200|0
20190524193000|7|REQUEST|10.194.20.31|cctg-npm-deployer|GET|/api/npm/cctg-npm-npm/iconv-lite/-/iconv-lite-0.2.11.tgz|HTTP/1.1|200|235160
Logstash filter
grok {
# Enable multiple matchers
break_on_match => false
match => { "message" => "%{DATA:timestamp_local}\|%{NUMBER:duration}\|%{WORD:requesttype}\|%{IP:ip}\|%{DATA:username}\|%{WORD:method}\|%{DATA:resource}\|%{DATA:protocol}\|%{NUMBER:statuscode}\|%{NUMBER:bytes}" }
# Extract repo and path
match => { "resource" => "/%{DATA:repo}/%{GREEDYDATA:resource_path}"}
# Extract resource name
match => { "resource_path" => "(?<resource_name>[^/]+$)" }
# Extract file extension
match => { "resource_path" => "(?<resource_type>[^.]+$)" }
}
Logstash outputs
influxdb {
host => "***.*.*.*"
db => "testing"
port => 8086
codec => "json"
measurement => "requests"
send_as_tags => ["ip","repo","site"]
use_event_fields_for_data_points => true
exclude_fields => ["@timestamp","timestamp","timestamp_object","timestamp_local","protocol","message","@version"]
coerce_values => {
'bytes' => 'integer'
'duration' => 'integer'}
}
when i tried to query influxdb
select bytes,ip,resource_path,username from requests
name: requests
time bytes ip resource_path username
---- ----- -------- ------------- --------
1558726546126000000 0 10.194.20.31 npm/cctg-npm-npm/iconv-lite cctg-npm-deployer
1558726546153000000 0 10.194.20.31 npm/cctg-npm-npm/js-yaml cctg-npm-deployer
1558726546160000000 0 10.25.79.236 docker/docker-login/v2/ non_authenticated_user
1558726546160000000 6289 10.194.20.31 npm/cctg-npm-npm/colors/-/colors-0.6.2.tgz cctg-npm-deployer
1558726546161000000 0 10.25.79.236 docker/docker-login/v2/token anonymous
1558726546161000000 0 173.39.45.237 storage/content-security-builds-snapshot/ESA/phoebe-12-1-br/ content-security-builds-deployer
1558726551069000000 2285 10.194.20.31 npm/cctg-npm-npm/which/-/which-1.0.9.tgz cctg-npm-deployer
I tried enabling debug logs in logstash but that didnot help and neither does influxdb throw any errors.
UPDATE 1:
I cleaned out influxDB and tested using the same logs again but this time different set of logs went through
Can anyone see whats missing ? Thanks