Since it's clear you wanted your grok rule to work, I took the liberty of adding type => "apache-access" to your config, so the conditional if statement would have something to match against:
input {
file {
path => "/Users/buh/test_access_log"
start_position => "beginning"
sincedb_path => "/Users/buh/mysincedb"
type => "apache-access"
}
}
Now, I delete the sincedb file:
rm /Users/buh/mysincedb
(I also deleted the index in Elasticsearch so it would be a fresh start)
Now, when I re-run bin/logstash -f test.conf --debug, I see what you were probably expecting:
[2017-04-24T13:41:50,598][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>1000}
[2017-04-24T13:41:50,703][INFO ][logstash.pipeline ] Pipeline main started
[2017-04-24T13:41:50,705][DEBUG][logstash.inputs.file ] _globbed_files: /Users/buh/test_access_log: glob is: ["/Users/buh/test_access_log"]
[2017-04-24T13:41:50,706][DEBUG][logstash.inputs.file ] _discover_file: /Users/buh/test_access_log: new: /Users/buh/test_access_log (exclude is [])
[2017-04-24T13:41:50,707][DEBUG][logstash.inputs.file ] _open_file: /Users/buh/test_access_log: opening
[2017-04-24T13:41:50,707][DEBUG][logstash.inputs.file ] /Users/buh/test_access_log: initial create, no sincedb, seeking to beginning of file
[2017-04-24T13:41:50,707][DEBUG][logstash.inputs.file ] Received line {:path=>"/Users/buh/test_access_log", :text=>"177.140.155.68 - - [24/Apr/2017:13:37:50 +0000] \"GET / HTTP/1.0\" 200 612 \"-\" \"Wget(linux)\""}
[2017-04-24T13:41:50,714][DEBUG][logstash.agent ] Starting puma
[2017-04-24T13:41:50,715][DEBUG][logstash.agent ] Trying to start WebServer {:port=>9600}
[2017-04-24T13:41:50,716][DEBUG][logstash.api.service ] [api-service] start
[2017-04-24T13:41:50,726][DEBUG][logstash.inputs.file ] writing sincedb (delta since last write = 1493062910)
[2017-04-24T13:41:50,732][DEBUG][logstash.pipeline ] filter received {"event"=>{"path"=>"/Users/buh/test_access_log", "@timestamp"=>2017-04-24T19:41:50.724Z, "@version"=>"1", "host"=>"localhost.local", "message"=>"177.140.155.68 - - [24/Apr/2017:13:37:50 +0000] \"GET / HTTP/1.0\" 200 612 \"-\" \"Wget(linux)\"", "type"=>"apache-access"}}
[2017-04-24T13:41:50,733][DEBUG][logstash.filters.grok ] Running grok filter {:event=>2017-04-24T19:41:50.724Z localhost.local 177.140.155.68 - - [24/Apr/2017:13:37:50 +0000] "GET / HTTP/1.0" 200 612 "-" "Wget(linux)"}
[2017-04-24T13:41:50,739][DEBUG][logstash.filters.grok ] Event now: {:event=>2017-04-24T19:41:50.724Z localhost.local 177.140.155.68 - - [24/Apr/2017:13:37:50 +0000] "GET / HTTP/1.0" 200 612 "-" "Wget(linux)"}
[2017-04-24T13:41:50,746][DEBUG][logstash.pipeline ] output received {"event"=>{"request"=>"/", "agent"=>"\"Wget(linux)\"", "auth"=>"-", "ident"=>"-", "verb"=>"GET", "message"=>"177.140.155.68 - - [24/Apr/2017:13:37:50 +0000] \"GET / HTTP/1.0\" 200 612 \"-\" \"Wget(linux)\"", "type"=>"apache-access", "path"=>"/Users/buh/test_access_log", "referrer"=>"\"-\"", "@timestamp"=>2017-04-24T13:37:50.000Z, "response"=>"200", "bytes"=>"612", "clientip"=>"177.140.155.68", "@version"=>"1", "host"=>"localhost.local", "httpversion"=>"1.0", "timestamp"=>"24/Apr/2017:13:37:50 +0000"}}
[2017-04-24T13:41:50,747][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
{
"request" => "/",
"agent" => "\"Wget(linux)\"",
"auth" => "-",
"ident" => "-",
"verb" => "GET",
"message" => "177.140.155.68 - - [24/Apr/2017:13:37:50 +0000] \"GET / HTTP/1.0\" 200 612 \"-\" \"Wget(linux)\"",
"type" => "apache-access",
"path" => "/Users/buh/test_access_log",
"referrer" => "\"-\"",
"@timestamp" => 2017-04-24T13:37:50.000Z,
"response" => "200",
"bytes" => "612",
"clientip" => "177.140.155.68",
"@version" => "1",
"host" => "localhost.local",
"httpversion" => "1.0",
"timestamp" => "24/Apr/2017:13:37:50 +0000"
}
^C[2017-04-24T13:41:53,503][WARN ][logstash.runner ] SIGINT received. Shutting down the agent.
And again, when queried from Elasticsearch:
$ curl -XGET http://localhost:9200/logstash-2017.04.24/_search?pretty -d '
{
"query": {
"match_all": {}
}
}
'
{
"took" : 29,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [
{
"_index" : "logstash-2017.04.24",
"_type" : "apache-access",
"_id" : "AVuhfAPwRx488yJUxuVA",
"_score" : 1.0,
"_source" : {
"request" : "/",
"agent" : "\"Wget(linux)\"",
"auth" : "-",
"ident" : "-",
"verb" : "GET",
"message" : "177.140.155.68 - - [24/Apr/2017:13:37:50 +0000] \"GET / HTTP/1.0\" 200 612 \"-\" \"Wget(linux)\"",
"type" : "apache-access",
"path" : "/Users/buh/test_access_log",
"referrer" : "\"-\"",
"@timestamp" : "2017-04-24T13:37:50.000Z",
"response" : "200",
"bytes" : "612",
"clientip" : "177.140.155.68",
"@version" : "1",
"host" : "localhost.local",
"httpversion" : "1.0",
"timestamp" : "24/Apr/2017:13:37:50 +0000"
}
}
]
}
}
Everything works, and /Users/buh/mysincedb is again updated, per the log file:
$ cat /Users/buh/mysincedb
58694687 1 4 91