# After upgrade 5.7 logstash to 6.0 unable to get some logs

**URL:** <https://discuss.elastic.co/t/after-upgrade-5-7-logstash-to-6-0-unable-to-get-some-logs/209514>\
**Category:** Logstash\
**Created:** [November 26, 2019, 1:00pm UTC](https://discuss.elastic.co/t/after-upgrade-5-7-logstash-to-6-0-unable-to-get-some-logs/209514 "2019-11-26T13:00:50Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![shadrach1985](https://avatars.discourse-cdn.com/v4/letter/s/977dab/32.png) [@shadrach1985](https://discuss.elastic.co/u/shadrach1985)\
**Post date:** [November 26, 2019, 1:00pm UTC](https://discuss.elastic.co/t/after-upgrade-5-7-logstash-to-6-0-unable-to-get-some-logs/209514/1 "2019-11-26T13:00:50Z")

</div>

After upgrade 5.7 logstash to 6.0 unable to get some logs in kibana

I am getting below error in logstash error log

[2019-11-26T12:17:21,720][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"nne-java-prod-logstash-2019.11", :\_type=\>"log", :\_routing=\>nil}, #LogStash::Event:0x2ab724aa], :response=\>{"index"=\>{"\_index"=\>"nne-java-prod-logstash-2019.11", "\_type"=\>"log", "\_id"=\>"t4ilp24BQ\_HGsnDQB4eb", "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse field [timestamp] of type [date] in document with id 't4ilp24BQ\_HGsnDQB4eb'", "caused\_by"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"Invalid format: "Nov 26 11:48:05""}}}}}

my parsing script below

filter {  
if ([fields][wstream] == "rsyslog") {  
grok {  
match =\> {  
"message" =\> "(?m)%{SYSLOGTIMESTAMP:timestamp} %{DATA:servername} %{GREEDYDATA:systemmsg}"  
}  
}  
mutate {  
strip =\> ["systemmsg"]  
}  
if ([systemmsg] =~ "was;") {  
grok {  
match =\> {  
"systemmsg" =\> "(?m)%{DATA:appevn};%{DATA:netype};%{DATA:project};%{DATA:env} %{GREEDYDATA:newasmsg}"  
}  
}

```
        mutate {
            strip => ["newasmsg"]
        }

        if ([netype] == "sysout") {
            grok {
                match => {
                    "newasmsg" => "(?m)(?<wastimestamp>%{GREEDYDATA:mystimestamp}]) %{DATA:ThreadId} %{DATA:shortName} %{DATA:eventType} %{DATA:className} %{DATA:methodName} %{GREEDYDATA:appmsg}"
                }
            }

            mutate {
                add_field => {
                    "[fields][type]" => "server"
                }
            }
        }

        if ([netype] == "syserr") {
            grok {
                match => {
                    "newasmsg" => "(?m)(?<wastimestamp>%{GREEDYDATA:mystimestamp}]) %{DATA:ThreadId} %{DATA:shortName} %{DATA:eventType} %{DATA:className} %{DATA:methodName} %{GREEDYDATA:appmsg}"
                }
            }

            mutate {
                add_field => {
                    "[fields][type]" => "server"
                }
            }
        }

        # Remove the square bracket in timestamp
        mutate {
            gsub => ["wastimestamp", "\[", ""]
            gsub => ["wastimestamp", "\]", ""]
            add_field => {
                "[fields][env]" => "%{env}"
            }
            add_field => {
                "[fields][project]" => "%{project}"
            }
            add_field => {
                "[fields][wstream]" => "java"
            }
            remove_field => ["systemmsg", "newasmsg", "mystimestamp", "env", "project", "server"]
        }
    }

    if ([systemmsg] =~ "esb;") {
        grok {
            match => {
                "systemmsg" => "(?m)%{DATA:appevn};%{DATA:netype};%{DATA:nenode};%{DATA:env} %{GREEDYDATA:newasmsg}"
            }
        }

        mutate {
            strip => ["newasmsg"]
        }

        if ([netype] == "esblog") {
            grok {
                match => {
                    "newasmsg" => "(?m)%{MONTHDAY:day}-%{MONTHNUM:month}-%{YEAR:year} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}#011%{GREEDYDATA:newasmsg}#011%{GREEDYDATA:xmlcontent}"
                }
            }
            mutate {
                add_field => {
                    "timestamp" => "%{year}-%{month}-%{day} %{hour}:%{minute}%{second}"
                }
            }
            date {
                match => ["timestamp", "ISO8601"]
            }
            # Remove the square bracket in timestamp
            mutate {
                add_field => {
                    "[fields][wstream]" => "esb"
                }
                add_field => {
                     "[fields][env]" => "%{env}"
                }

                remove_field => ["newasmsg","year","month","day","hour","minute","second",systemmsg]
            }

        }# end of esblog if loop
    }# end of esb if loop

}

```

if ([systemmsg] =~ "wps;") {  
grok {  
patterns\_dir =\> ["/etc/logstash/patterns.d"]  
match =\> ["message", "%{RSYSLOGWPS1}"]  
}

```
mutate {
  add_field => {
    "[fields][project]" => "wps"
  }

```

# convert =\> { "finalstatuscode" =\> "integer" }

# convert =\> { "httpd\_responsetime" =\> "integer" }

# convert =\> { "httpd\_bytes" =\> "integer" }

```
    convert => { "httpd_response" => "integer" }
    convert => { "httpd_requesttimemilli" => "integer" }
    convert => { "httpd_bytes" => "integer" }

```

}

```
if !("_grokparsefailure" in [tags]) {
  mutate {
    remove_field => ["message","systemmsg",finalstatuscodestr,httpd_bytesstr,httpd_responsetimestr, servername1]
  }
}

```

}  
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 24, 2019, 1:09pm UTC](https://discuss.elastic.co/t/after-upgrade-5-7-logstash-to-6-0-unable-to-get-some-logs/209514/2 "2019-12-24T13:09:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.