# Problem with log processing
**URL:** https://discuss.elastic.co/t/problem-with-log-processing/227911
**Category:** Beats
**Tags:** elastic-stack-monitoring, filebeat
**Created:** [April 14, 2020, 12:35pm UTC](https://discuss.elastic.co/t/problem-with-log-processing/227911 "2020-04-14T12:35:56Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Vlad\_Piratov](https://discuss.elastic.co/u/Vlad_Piratov)
#### Post date: [April 14, 2020, 12:35pm UTC](https://discuss.elastic.co/t/problem-with-log-processing/227911/1 "2020-04-14T12:35:56Z")
Version 7.6.2
Data is processed incorrectly. All dashboards are empty (I created a user for this, used sudo commands, and so on)
Configured filebeat as follows: filebeat - \> Logstash - \> Elasticsearch.
I tried different actions for issuing syslog in kibana and all without result.
Please help me with this problem 🙂
`http://dl4.joxi.net/drive/2020/04/14/0003/0102/221286/86/849381b8d6.png`
/etc/logstash/logstash.yml
```
path.data: /var/lib/logstash
path.logs: /var/log/logstash
```
/etc/logstash/pipelines.yml
```
- pipeline.id: main
path.config: "/etc/logstash/conf.d/*.conf"
```
/etc/logstash/conf.d/logstash-simple.conf
```
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
beats {
port => 5044
}
}
filter {
if [fileset][module] == "system" {
if [fileset][name] == "auth" {
grok {
match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s*%{DATA:[system][auth][user]} :( %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$",
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}"] }
pattern_definitions => {
"GREEDYMULTILINE"=> "(.|\n)*"
}
remove_field => "message"
}
date {
match => ["[system][auth][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
geoip {
source => "[system][auth][ssh][ip]"
target => "[system][auth][ssh][geoip]"
}
}
else if [fileset][name] == "syslog" {
grok {
match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }
pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }
remove_field => "message"
}
date {
match => ["[system][syslog][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}
```
/etc/filebeat/filebeat.yml
```
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/*.log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
setup.template.settings:
index.number_of_shards: 1
output.logstash:
hosts: ["192.168.10.183:5044"]
```
/etc/filebeat/modules.d/system.yml
```
- module: system
syslog:
enabled: true
var.paths: ["/var/log/syslog*"]
auth:
enabled: true
var.paths: ["/var/log/auth.log*"]
```
Elasticsearch settings default
---
### Author:  [@mtojek](https://discuss.elastic.co/u/mtojek)
#### Post date: [April 14, 2020, 1:10pm UTC](https://discuss.elastic.co/t/problem-with-log-processing/227911/2 "2020-04-14T13:10:08Z")
Did you try to enable debug logging and verify step by step? I'm not sure if you're actually facing a problem with logstash not beats...
---
### Author:  [@Vlad\_Piratov](https://discuss.elastic.co/u/Vlad_Piratov)
#### Post date: [April 14, 2020, 5:44pm UTC](https://discuss.elastic.co/t/problem-with-log-processing/227911/3 "2020-04-14T17:44:47Z")
It doesn't work out of the "box" with the current settings?
---
### Author:  [@Vlad\_Piratov](https://discuss.elastic.co/u/Vlad_Piratov)
#### Post date: [April 15, 2020, 1:01pm UTC](https://discuss.elastic.co/t/problem-with-log-processing/227911/4 "2020-04-15T13:01:56Z")
[Logs](https://yadi.sk/d/YgIm-Y-u5yjT9g)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 13, 2020, 1:02pm UTC](https://discuss.elastic.co/t/problem-with-log-processing/227911/5 "2020-05-13T13:02:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.