Gz file input problem (terminal shows no response)

Elastic Stack Logstash
1

This might be a basic problem. I have the error code of "invalid config" when i try to parse gz file into logstash.

Below is my config:
input {
file {
type => "gzip"
path => "/Users/yangyan/Desktop/log_file/s3_logs/2014/testpurchase_funnel.log.gz"
start_position => "beginning"
codec => "gzip_lines"
}

filter {
grok{
match => { "message" => "%{SYSLOGBASE} %{GREEDYDATA:message}" }
overwrite => [ "message" ]
}

json {
source => "message"
}

mutate{
rename => { "[purchase][source]" => "p_source" }
rename => { "[purchase][type_id]" => "p_typeId" }
rename => { "[purchase][content_id]" => "p_contentId" }
}

date{
match => ["date", "YYYY-MM-dd HH:mm:ss", "ISO8601" ]
timezone => "Europe/Dublin"
target => "date"
}

prune {
whitelist_names => [ "@timestamp", "app_id", "member_id", "locale", "date", "p_source", "p_typeId", "p_contentId" ]
}
}

output {
stdout { codec => rubydebug }
}

AND below is the error:
etched an invalid config {:config=>"input {\n file {\n type => "gzip"\n path => "/Users/yangyan/Desktop/log_file/s3_logs/2014/testpurchase_funnel.log.gz"\n start_position => "beginning"\n codec => "gzip_lines"\n} \n\n filter {\n grok{\n match => { "message" => "%{SYSLOGBASE} %{GREEDYDATA:message}" }\n overwrite => [ "message" ]\n}\n\n json {\n source => "message"\n}\n\n mutate{\n rename => { "[purchase][source]" => "p_source" }\n rename => { "[purchase][type_id]" => "p_typeId" }\n rename => { "[purchase][content_id]" => "p_contentId" } \n}\n\n date{\n match => ["date", "YYYY-MM-dd HH:mm:ss", "ISO8601" ]\n timezone => "Europe/Dublin" \n target => "date"\n}\n\n\n prune {\n whitelist_names => [ "@timestamp", "app_id", "member_id", "locale", "date", "p_source", "p_typeId", "p_contentId" ]\n} \n}\n\noutput {\n stdout { codec => rubydebug }\n}\n\n", :reason=>"Expected one of #, => at line 10, column 9 (byte 207) after input {\n file {\n type => "gzip"\n path => "/Users/yangyan/Desktop/log_file/s3_logs/2014/testpurchase_funnel.log.gz"\n start_position => "beginning"\n codec => "gzip_lines"\n} \n\n filter {\n grok", :level=>:error}

2

Hi @Allie_Yang,

This forum is for the Elastic Cloud service. Could you move your question to the Logstash forum.

Cheers,
Ben

3

You're not closing your input block before you're opening your filter block. If you indent your configuration file consistently things like this becomes very easy to spot.

4

Hey thanks i noticed that too. Now, I don't have any error. It is now showing no response when i begin to parse:

My config:
input {
file {
path => "/Users/yangyan/Desktop/log_file/s3_logs/2014/01/28/event_log/test.log.gz"
start_position => "beginning"
codec => "gzip_lines"
type => "gzip"
ignore_older => 0
sincedb_path => "/dev/null"
}
}

filter {
grok{
match => { "message" => "%{SYSLOGBASE} %{GREEDYDATA:message}" }
overwrite => [ "message" ]
}

json {
source => "message"
}

mutate{
rename => { "[purchase][source]" => "p_source" }
rename => { "[purchase][type_id]" => "p_typeId" }
rename => { "[purchase][content_id]" => "p_contentId" }
}

date{
match => ["date", "YYYY-MM-dd HH:mm:ss", "ISO8601" ]
timezone => "UTC"
target => "date"
}

prune {
whitelist_names => [ "@timestamp", "app_id", "member_id", "locale", "date", "p_source", "p_typeId", "p_contentId" ]
}
}

output {
stdout { codec => rubydebug }
}

My terminal response:
yangyan-osx:2.3.4 yangyan$ bin/logstash -f testgzip.conf
Settings: Default pipeline workers: 8
Pipeline main started

(nothing shows up)