Overwriting the @timestamp work on stdout but don't work on es

Elastic Stack Logstash
1

I want to overwrite the @timestamp with date filter.
The result in the stdout is ok, the @timestamp is written by the log time, but there is no document written into the elasticsearch.
If I remove the date filter, the document is normally written into the elasticsearch.

The pattern
NGINX_ACCESS %{IPORHOST:remote_addr} - %{USERNAME:remote_user} \[%{HTTPDATE:timelocal}\] "%{WORD:request_method} %{DATA:request} %{DATA:http_version}" %{INT:status} %{NUMBER:body_bytes_sent} "%{DATA:http_refer}" "%{DATA:http_user_agent}" "%{NUMBER:request_time}" "%{DATA:ssl_protocol}" "%{DATA:ssl_cipher}" "%{DATA:http_x_forwarded_for}""%{DATA:upstream_addr}" "%{DATA:upstream_status}" "%{DATA:upstream_response_length}" "%{DATA:upstream_response_time}"

Here is the config file of the logstash

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    patterns_dir => "/home/dpc/elk/logstash/patterns"
    match => { "message" => "%{NGINX_ACCESS}" }
  }
  mutate {
	remove_field => [
		"upstream_addr", "ssl_cipher", "ssl_protocol", "host", "ecs", "@version",    
		"input", "http_x_forwarded_for", "http_user_agent", "http_refer", "body_bytes_sent",
		"agent", "remote_user", "input"
	]   
  }
  date {
     match => [ "timelocal", "dd/MMM/yyyy:HH:mm:ss Z" ]
     target => "@timestamp"
  }
}

output {
  elasticsearch {
    hosts => ["http://10.193.161.30:9200"]
    index => "server_log_%{+YYYY.MM.dd}"
    codec => json
  }
  stdout {
    codec => json
  }
}

This is the log
10.192.1.144 - - [14/Jan/2020:20:34:20 +0800] "POST /msxiaobing_callback HTTP/1.1" 403 47 "-" "python-requests/2.18.4" "0.004" "-" "-" "-""10.193.161.5:8080" "403" "47" "0.004"

The result of the stdout

{
    "http_version":"HTTP/1.1",
    "request":"/msxiaobing_callback",
    "request_method":"POST",
    "upstream_response_time":"0.004",
    "@timestamp":"2020-01-14T12:34:20.000Z",
    "upstream_status":"403",
    "message":"10.192.1.144 - - [14/Jan/2020:20:34:20 +0800] "POST /msxiaobing_callback HTTP/1.1" 403 47 "-" "python-requests/2.18.4" "0.004" "-" "-" "-""10.193.161.5:8080" "403" "47" "0.004"",
    "upstream_response_length":"47",
    "timelocal":"14/Jan/2020:20:34:20 +0800",
    "remote_addr":"10.192.1.144",
    "log":{
        "offset":28092,
        "file":{
            "path":"/data/dingpeichang/logs/server_access.log"
        }
    },
    "status":"403",
    "tags":[
        "beats_input_codec_plain_applied"
    ],
    "request_time":"0.004"
}
2

Is there no error in the Logstash logs?

3

There is no any error log in the logstash logs. That's why it's hard for me to solve it.
Thank you for your replying.

4

Just to make sure: Querying your ES index like this
GET server_log_2020.01.14/_search?q=@timestamp:"2020-01-14T12:34:20.000Z"
returns nothing?

1 Like
5

I used the wrong index in the es.
Thank you!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.