# Need help - logstash pipeline isn't working anymore ...?

**URL:** <https://discuss.elastic.co/t/need-help-logstash-pipeline-isnt-working-anymore/210006>\
**Category:** Logstash\
**Created:** [November 29, 2019, 9:57pm UTC](https://discuss.elastic.co/t/need-help-logstash-pipeline-isnt-working-anymore/210006 "2019-11-29T21:57:45Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Cdnvballer](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Cdnvballer](https://discuss.elastic.co/u/Cdnvballer)\
**Post date:** [November 29, 2019, 9:57pm UTC](https://discuss.elastic.co/t/need-help-logstash-pipeline-isnt-working-anymore/210006/1 "2019-11-29T21:57:45Z")

</div>

Kibana -- Discover console is showing events coming in, but the indexing is not working right.  
used to be able to see the fields [system][syslog][hostname] and related but now they're gone??

==02-beats-input.conf ==  
input {  
beats {  
port =\> 5044  
}  
}

==10-syslog-filter.conf ==  
filter {  
if [fileset][module] == "system" {  
if [fileset][name] == "auth" {  
grok {  
match =\> { "message" =\> ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][p  
id]}])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[syste  
m][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",  
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: %{DAT  
A:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",  
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: Did n  
ot receive identification string from %{IPORHOST:[system][auth][ssh][dropped\_ip]}",  
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:[%{POSINT:[system][auth][pid]}])?: \s\*%{  
DATA:[system][auth][user]} ☹ %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; US  
ER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",  
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:[%{POSINT:[system][auth][pid]}])?: n  
ew group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",  
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:[%{POSINT:[system][auth][pid]}])?: ne  
w user: name=%{DATA:[system][auth][useradd][name]}, UID=%{NUMBER:[system][auth][useradd][uid]}, GID=%{NUMBER:[system][auth][useradd][gid]}, home=%{DATA:  
[system][auth][useradd][home]}, shell=%{DATA:[system][auth][useradd][shell]}$",  
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:[%{POSINT:[sys  
tem][auth][pid]}])?: %{GREEDYMULTILINE:[system][auth][message]}"] }  
pattern\_definitions =\> {  
"GREEDYMULTILINE"=\> "(.|\n)_"  
}  
remove\_field =\> "message"  
}  
date {  
match =\> ["[system][auth][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]  
}  
geoip {  
source =\> "[system][auth][ssh][ip]"  
target =\> "[system][auth][ssh][geoip]"  
}  
}  
else if [fileset][name] == "syslog" {  
grok {  
match =\> { "message" =\> ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][progra  
m]}(?:[%{POSINT:[system][syslog][pid]}])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }  
pattern\_definitions =\> { "GREEDYMULTILINE" =\> "(.|\n)_" }  
remove\_field =\> "message"  
}  
date {  
match =\> ["[system][syslog][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]  
}  
}  
}  
}

== 30-elasticsearch-output.conf ==  
output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
manage\_template =\> false  
index =\> "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"  
}

---

<div class="post-metadata">

**Author:** ![Cdnvballer](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Cdnvballer](https://discuss.elastic.co/u/Cdnvballer)\
**Post date:** [November 29, 2019, 10:00pm UTC](https://discuss.elastic.co/t/need-help-logstash-pipeline-isnt-working-anymore/210006/2 "2019-11-29T22:00:11Z")

</div>

> [@Cdnvballer](#):
>
> _match =\> { "message" =\> ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][progra  
> m]}(?:[%{POSINT:[system][syslog][pid]}])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }  
> pattern\_definitions =\> { "GREEDYMULTILINE" =\> "(.|\n)_ " }

this part works fine in GROK Debugger.. but those fields seemed to have disappeared.. I've tried to delete indexes, but no improvement...

---

<div class="post-metadata">

**Author:** ![Cdnvballer](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Cdnvballer](https://discuss.elastic.co/u/Cdnvballer)\
**Post date:** [November 30, 2019, 7:35pm UTC](https://discuss.elastic.co/t/need-help-logstash-pipeline-isnt-working-anymore/210006/3 "2019-11-30T19:35:34Z")

</div>

can any please give me some idea how to troubleshoot this?

1. I've tried deleting the indices and index pattern,
2. from Kibana discover console, I can see all incoming events with fileset.name = syslog and still containing ["message"]

3)however, there isn't anything that contains/assigned to  
[system][syslog][timestamp]  
[system][syslog][hostname]  
[system][syslog][message]

it's as if the grok match just stopped working...  
and it doesn't seem like  
remove\_field =\> "message"  
is happening either...

this is test environment, so I'm fine blowing things away.. help!

---

<div class="post-metadata">

**Author:** ![Cdnvballer](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Cdnvballer](https://discuss.elastic.co/u/Cdnvballer)\
**Post date:** [December 2, 2019, 7:08pm UTC](https://discuss.elastic.co/t/need-help-logstash-pipeline-isnt-working-anymore/210006/4 "2019-12-02T19:08:21Z")

</div>

well... I uninstalled 7.4.2 enviroment and re-installed 6.8.5 ... fields are matching again...  
assumption is that previous attempt to upgrade to 7.4.2 corrupted something..  
but now dealing with weird parsing/indexing.. =(

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 30, 2019, 7:08pm UTC](https://discuss.elastic.co/t/need-help-logstash-pipeline-isnt-working-anymore/210006/5 "2019-12-30T19:08:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.