# Filebeat 6.3.2 syslogs are not logging to elastcisearch
**URL:** https://discuss.elastic.co/t/filebeat-6-3-2-syslogs-are-not-logging-to-elastcisearch/149490
**Category:** Beats
**Tags:** filebeat
**Created:** [September 21, 2018, 4:47pm UTC](https://discuss.elastic.co/t/filebeat-6-3-2-syslogs-are-not-logging-to-elastcisearch/149490 "2018-09-21T16:47:32Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [September 21, 2018, 4:47pm UTC](https://discuss.elastic.co/t/filebeat-6-3-2-syslogs-are-not-logging-to-elastcisearch/149490/1 "2018-09-21T16:47:32Z")
System module is enabled, but syslogs are logging with 7hrs delay. Adjusted var.convert\_timezone: true in **/etc/filebeat/modules.d/system.yml** , still no luck. Can someone take a look into it.
/etc/filebeat/modules.d/system.yml
```auto
- module: system
# Syslog
syslog:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
var.convert_timezone: true
# Authorization logs
auth:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
var.convert_timezone: true
```
Filebeat\_log:
```auto
|2018-09-21T09:43:51.260-0700|INFO|[monitoring]|log/log.go:124|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1200,"time":{"ms":1204}},"total":{"ticks":10000,"time":{"ms":10012},"value":10000},"user":{"ticks":8800,"time":{"ms":8808}}},"info":{"ephemeral_id":"edceef24-4069-4d3a-a648-f7947d9927bf","uptime":{"ms":30009}},"memstats":{"gc_next":11220912,"memory_alloc":5683112,"memory_total":830369184,"rss":58142720}},"filebeat":{"events":{"added":147363,"done":147363},"harvester":{"open_files":18,"running":18,"started":18}},"libbeat":{"config":{"module":{"running":7,"starts":7},"reloads":1},"output":{"events":{"acked":146558,"batches":2936,"total":146558},"read":{"bytes":2045860},"type":"elasticsearch","write":{"bytes":87199374}},"pipeline":{"clients":17,"events":{"active":0,"filtered":805,"published":146558,"retry":50,"total":147363},"queue":{"acked":146558}}},"registrar":{"states":{"current":17,"update":147363},"writes":{"success":2923,"total":2923}},"system":{"cpu":{"cores":2},"load":{"1":0.69,"15":0.18,"5":0.3,"norm":{"1":0.345,"15":0.09,"5":0.15}}}}}}|
|---|---|---|---|---|---|
|2018-09-21T09:44:21.259-0700|INFO|[monitoring]|log/log.go:124|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1210,"time":{"ms":8}},"total":{"ticks":10030,"time":{"ms":24},"value":10030},"user":{"ticks":8820,"time":{"ms":16}}},"info":{"ephemeral_id":"edceef24-4069-4d3a-a648-f7947d9927bf","uptime":{"ms":60009}},"memstats":{"gc_next":11220912,"memory_alloc":7146264,"memory_total":831832336}},"filebeat":{"events":{"added":4,"done":4},"harvester":{"open_files":18,"running":18}},"libbeat":{"config":{"module":{"running":7}},"output":{"events":{"acked":4,"batches":4,"total":4},"read":{"bytes":1374},"write":{"bytes":4653}},"pipeline":{"clients":17,"events":{"active":0,"published":4,"total":4},"queue":{"acked":4}}},"registrar":{"states":{"current":17,"update":4},"writes":{"success":4,"total":4}},"system":{"load":{"1":0.42,"15":0.17,"5":0.27,"norm":{"1":0.21,"15":0.085,"5":0.135}}}}}}|
|2018-09-21T09:44:51.259-0700|INFO|[monitoring]|log/log.go:124|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1210},"total":{"ticks":10060,"time":{"ms":28},"value":10060},"user":{"ticks":8850,"time":{"ms":28}}},"info":{"ephemeral_id":"edceef24-4069-4d3a-a648-f7947d9927bf","uptime":{"ms":90009}},"memstats":{"gc_next":11220912,"memory_alloc":9304584,"memory_total":833990656}},"filebeat":{"events":{"added":11,"done":11},"harvester":{"open_files":18,"running":18}},"libbeat":{"config":{"module":{"running":7}},"output":{"events":{"acked":11,"batches":7,"total":11},"read":{"bytes":2454},"write":{"bytes":10085}},"pipeline":{"clients":17,"events":{"active":0,"published":11,"total":11},"queue":{"acked":11}}},"registrar":{"states":{"current":17,"update":11},"writes":{"success":7,"total":7}},"system":{"load":{"1":0.25,"15":0.17,"5":0.24,"norm":{"1":0.125,"15":0.085,"5":0.12}}}}}}|
|2018-09-21T09:45:21.259-0700|INFO|[monitoring]|log/log.go:124|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1210,"time":{"ms":4}},"total":{"ticks":10090,"time":{"ms":32},"value":10090},"user":{"ticks":8880,"time":{"ms":28}}},"info":{"ephemeral_id":"edceef24-4069-4d3a-a648-f7947d9927bf","uptime":{"ms":120008}},"memstats":{"gc_next":11219376,"memory_alloc":6261536,"memory_total":835962168}},"filebeat":{"events":{"added":12,"done":12},"harvester":{"open_files":18,"running":18}},"libbeat":{"config":{"module":{"running":7}},"output":{"events":{"acked":12,"batches":6,"total":12},"read":{"bytes":2125},"write":{"bytes":9585}},"pipeline":{"clients":17,"events":{"active":0,"published":12,"total":12},"queue":{"acked":12}}},"registrar":{"states":{"current":17,"update":12},"writes":{"success":6,"total":6}},"system":{"load":{"1":0.15,"15":0.16,"5":0.22,"norm":{"1":0.075,"15":0.08,"5":0.11}}}}}}|
```
I see auth logs are coming as expected with proper timestamp.
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [September 25, 2018, 5:59am UTC](https://discuss.elastic.co/t/filebeat-6-3-2-syslogs-are-not-logging-to-elastcisearch/149490/2 "2018-09-25T05:59:44Z")
Are the 7h difference the difference from your time zone?
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [September 25, 2018, 8:25am UTC](https://discuss.elastic.co/t/filebeat-6-3-2-syslogs-are-not-logging-to-elastcisearch/149490/3 "2018-09-25T08:25:54Z")
Yes; They are logging with 7hrs delay; I tried `curl -XDELETE 'http://localhost:9200/_ingest/pipeline/filebeat-*'`. Still i see it is loading to elasticsearch with delay.
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [September 25, 2018, 8:29am UTC](https://discuss.elastic.co/t/filebeat-6-3-2-syslogs-are-not-logging-to-elastcisearch/149490/4 "2018-09-25T08:29:22Z")
`curl -XGET "http://localhost:9200/filebeat-*/_search?_source=@timestamp,beat.timezone"`
```auto
{"took":39,"timed_out":false,"_shards":{"total":24,"successful":24,"skipped":0,"failed":0},"hits":{"total":5252536,"max_score":1.0,"hits":[{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"ZOTQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.311Z"}},{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"Z-TQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.311Z"}},{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"aeTQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.311Z"}},{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"cuTQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.311Z"}},{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"eOTQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.311Z"}},{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"euTQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.311Z"}},{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"fOTQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.311Z"}},{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"f-TQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.311Z"}},{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"guTQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.311Z"}},{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"ieTQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.314Z"}}]}}
```
`curl -XGET "http://localhost:9200/filebeat-*/_search?_source=@timestamp,beat.timezone"`
```auto
{"filebeat-6.3.2-system-syslog-pipeline":{"description":"Pipeline for parsing Syslog messages.","processors":[{"grok":{"field":"message","patterns":["%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{SYSLOGHOST:system.syslog.hostname} %{DATA:system.syslog.program}(?:\\[%{POSINT:system.syslog.pid}\\])?: %{GREEDYMULTILINE:system.syslog.message}","%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{GREEDYMULTILINE:system.syslog.message}"],"pattern_definitions":{"GREEDYMULTILINE":"(.|\n)*"},"ignore_missing":true}},{"remove":{"field":"message"}},{"date":{"field":"system.syslog.timestamp","target_field":"@timestamp","formats":["MMM d HH:mm:ss","MMM dd HH:mm:ss"],"ignore_failure":true}}],"on_failure":[{"set":{"value":"{{ _ingest.on_failure_message }}","field":"error.message"}}]}}
```
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [September 25, 2018, 10:14am UTC](https://discuss.elastic.co/t/filebeat-6-3-2-syslogs-are-not-logging-to-elastcisearch/149490/5 "2018-09-25T10:14:24Z")
Deleted filebeat ingest and restart filebeat.
```auto
curl -XDELETE 'http://localhost:9200/_ingest/pipeline/filebeat-*'
```
Now i see the syslogs are getting loaded but not authlogs.
```auto
2018-09-25T03:08:11.237-0700 WARN elasticsearch/client.go:502 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbee2a02174713ca8, ext:482829647661, loc:(*time.Location)(0x1f4bd20)}, Meta:common.MapStr{"pipeline":"filebeat-6.3.2-system-auth-pipeline"}, Fields:common.MapStr{"host":common.MapStr{"name":"elasticsearch"}, "source":"/var/log/auth.log", "offset":275099, "message":"Sep 25 03:08:04 elasticsearch sshd[2777]: Close session: user rdmon from 2600:1f18:2270:741e:bc52:4f7:4084:8855 port 43138 id 0", "fileset":common.MapStr{"module":"system", "name":"auth"}, "prospector":common.MapStr{"type":"log"}, "input":common.MapStr{"type":"log"}, "beat":common.MapStr{"timezone":"-07:00", "version":"6.3.2", "name":"elasticsearch", "hostname":"elasticsearch"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc42040ed00), Source:"/var/log/auth.log", Offset:275247, Timestamp:time.Time{wall:0xbee29fa8c7ccc9dd, ext:80672892, loc:(*time.Location)(0x1f4bd20)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x2964, Device:0xca01}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-6.3.2-system-auth-pipeline] does not exist"}
```
System module config yml:
```auto
- module: system
# Syslog
syslog:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false
# Authorization logs
auth:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
var.convert_timezone: true
```
I see the same issues for nginx module as well.
```auto
WARN elasticsearch/client.go:502 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbee2a19ec821c592, ext:306262386164928, loc:(*time.Location)(0x1f4bd20)}, Meta:common.MapStr{"pipeline":"filebeat-6.3.2-nginx-error-pipeline"}, Fields:common.MapStr{"source":"/var/log/nginx/error.log", "offset":51599584, "message":"2018/09/25 03:33:30 [debug] 19308#19308: *20729 accept: 10.201.21.48:31570 fd:16", "input":common.MapStr{"type":"log"}, "fileset":common.MapStr{"module":"nginx", "name":"error"}, "prospector":common.MapStr{"type":"log"}, "beat":common.MapStr{"name":"nginx", "hostname":"nginx", "version":"6.3.2"}, "host":common.MapStr{"name":"nginx"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc42048c8f0), Source:"/var/log/nginx/error.log", Offset:51599665, Timestamp:time.Time{wall:0xbee2951f6a1a6bc6, ext:293464956108620, loc:(*time.Location)(0x1f4bd20)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xfa4ec, Device:0xca01}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-6.3.2-nginx-error-pipeline] does not exist"}
2018-09-25T03:33:32.140-0700 WARN elasticsearch/client.go:502 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbee2a19ec821de3f, ext:306262386171246, loc:(*time.Location)(0x1f4bd20)}, Meta:common.MapStr{"pipeline":"filebeat-6.3.2-nginx-error-pipeline"}, Fields:common.MapStr{"source":"/var/log/nginx/error.log", "offset":51599665, "message":"2018/09/25 03:33:30 [debug] 19308#19308: *20729 event timer add: 16: 60000:1537871670747", "input":common.MapStr{"type":"log"}, "fileset":common.MapStr{"module":"nginx", "name":"error"}, "prospector":common.MapStr{"type":"log"}, "beat":common.MapStr{"hostname":"nginx", "version":"6.3.2", "name":"nginx"}, "host":common.MapStr{"name":"nginx"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc42048c8f0), Source:"/var/log/nginx/error.log", Offset:51599754, Timestamp:time.Time{wall:0xbee2951f6a1a6bc6, ext:293464956108620, loc:(*time.Location)(0x1f4bd20)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xfa4ec, Device:0xca01}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-6.3.2-nginx-error-pipeline] does not exist"}
2018-09-25T03:33:32.140-0700 WARN elasticsearch/client.go:502 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbee2a19ec82243dc, ext:306262386197271, loc:(*time.Location)(0x1f4bd20)}, Meta:common.MapStr{"pipeline":"filebeat-6.3.2-nginx-error-pipeline"}, Fields:common.MapStr{"offset":51599754, "message":"2018/09/25 03:33:30 [debug] 19308#19308: *20729 reusable connection: 1", "input":common.MapStr{"type":"log"}, "fileset":common.MapStr{"module":"nginx", "name":"error"}, "prospector":common.MapStr{"type":"log"}, "beat":common.MapStr{"version":"6.3.2", "name":"nginx", "hostname":"nginx"}, "host":common.MapStr{"name":"nginx"}, "source":"/var/log/nginx/error.log"}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc42048c8f0), Source:"/var/log/nginx/error.log", Offset:51599825, Timestamp:time.Time{wall:0xbee2951f6a1a6bc6, ext:293464956108620, loc:(*time.Location)(0x1f4bd20)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xfa4ec, Device:0xca01}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-6.3.2-nginx-error-pipeline] does not exist"}
2018-09-25T03:33:32.140-0700
```
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [September 25, 2018, 10:51am UTC](https://discuss.elastic.co/t/filebeat-6-3-2-syslogs-are-not-logging-to-elastcisearch/149490/6 "2018-09-25T10:51:05Z")
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [September 25, 2018, 10:52am UTC](https://discuss.elastic.co/t/filebeat-6-3-2-syslogs-are-not-logging-to-elastcisearch/149490/7 "2018-09-25T10:52:25Z")
I dont see auth/nginx pipelines/logs in `GET _ingest/pipeline/filebeat-6.3.2-*`
I also noticed system logs are logging with 7hrs delay and nginx logs are not even logging.
@ruflin, could you take a look.
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [September 26, 2018, 6:38pm UTC](https://discuss.elastic.co/t/filebeat-6-3-2-syslogs-are-not-logging-to-elastcisearch/149490/8 "2018-09-26T18:38:35Z")
Can someone take a look into it.
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [September 26, 2018, 9:37pm UTC](https://discuss.elastic.co/t/filebeat-6-3-2-syslogs-are-not-logging-to-elastcisearch/149490/9 "2018-09-26T21:37:29Z")
Are you sure there is a 7h delay or is it potentially a time zone issue and your timestamps are in the future? That is why I asked above about the time zone? What time zone are your servers in?
BTW: Instead of an image please share things as code and please be patient.
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [September 26, 2018, 10:32pm UTC](https://discuss.elastic.co/t/filebeat-6-3-2-syslogs-are-not-logging-to-elastcisearch/149490/10 "2018-09-26T22:32:51Z")
Here is the log:
```auto
2018-09-26T15:29:53.097-0700 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":17410,"time":{"ms":12}},"total":{"ticks":85510,"time":{"ms":32},"value":85510},"user":{"ticks":68100,"time":{"ms":20}}},"info":{"ephemeral_id":"6746f90b-2141-43fb-8275-dd6580801654","uptime":{"ms":100110023}},"memstats":{"gc_next":6927696,"memory_alloc":4833968,"memory_total":6773670520}},"filebeat":{"events":{"active":-1,"added":49,"done":50},"harvester":{"open_files":3,"running":3}},"libbeat":{"config":{"module":{"running":6}},"output":{"events":{"acked":32,"batches":9,"dropped":18,"total":50},"read":{"bytes":3337},"write":{"bytes":35236}},"pipeline":{"clients":17,"events":{"active":0,"published":49,"total":49},"queue":{"acked":50}}},"registrar":{"states":{"current":17,"update":50},"writes":{"success":9,"total":9}},"system":{"load":{"1":0.14,"15":0.02,"5":0.07,"norm":{"1":0.07,"15":0.01,"5":0.035}}}}}}
2018-09-26T15:30:05.516-0700 WARN elasticsearch/client.go:502 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbee31ffb1e9de2f1, ext:100121443294109, loc:(*time.Location)(0x1f4bd20)}, Meta:common.MapStr{"pipeline":"filebeat-6.3.2-system-auth-pipeline"}, Fields:common.MapStr{"offset":498030, "message":"Sep 26 15:30:01 elasticsearch-i-0c1f8e517148213d1 CRON[4286]: pam_unix(cron:session): session opened for user root by (uid=0)", "fileset":common.MapStr{"module":"system", "name":"auth"}, "prospector":common.MapStr{"type":"log"}, "input":common.MapStr{"type":"log"}, "beat":common.MapStr{"timezone":"-07:00", "hostname":"elasticsearch01.rep", "version":"6.3.2", "name":"elasticsearch01.rep"}, "host":common.MapStr{"name":"elasticsearch01.rep"}, "source":"/var/log/auth.log"}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc4202651e0), Source:"/var/log/auth.log", Offset:498156, Timestamp:time.Time{wall:0xbee2e97d55f25142, ext:44322297832655, loc:(*time.Location)(0x1f4bd20)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x252f, Device:0xca01}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-6.3.2-system-auth-pipeline] does not exist"}
2018-09-26T15:30:10.517-0700 WARN elasticsearch/client.go:502 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbee31ffb1e9df126, ext:100121443297745, loc:(*time.Location)(0x1f4bd20)}, Meta:common.MapStr{"pipeline":"filebeat-6.3.2-system-auth-pipeline"}, Fields:common.MapStr{"message":"Sep 26 15:30:01 elasticsearch-i-0c1f8e517148213d1 CRON[4286]: pam_unix(cron:session): session closed for user root", "fileset":common.MapStr{"name":"auth", "module":"system"}, "prospector":common.MapStr{"type":"log"}, "input":common.MapStr{"type":"log"}, "beat":common.MapStr{"version":"6.3.2", "name":"elasticsearch01.rep", "hostname":"elasticsearch01.rep", "timezone":"-07:00"}, "host":common.MapStr{"name":"elasticsearch01.rep"}, "source":"/var/log/auth.log", "offset":498156}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc4202651e0), Source:"/var/log/auth.log", Offset:498271, Timestamp:time.Time{wall:0xbee2e97d55f25142, ext:44322297832655, loc:(*time.Location)(0x1f4bd20)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x252f, Device:0xca01}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-6.3.2-system-auth-pipeline] does not exist"}
2018-09-26T15:30:23.097-0700 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":17420,"time":{"ms":4}},"total":{"ticks":85550,"time":{"ms":28},"value":85550},"user":{"ticks":68130,"time":{"ms":24}}},"info":{"ephemeral_id":"6746f90b-2141-43fb-8275-dd6580801654","uptime":{"ms":100140023}},"memstats":{"gc_next":6852368,"memory_alloc":3852688,"memory_total":6775627920}},"filebeat":{"events":{"active":1,"added":9,"done":8},"harvester":{"open_files":3,"running":3}},"libbeat":{"config":{"module":{"running":6}},"output":{"events":{"acked":6,"batches":6,"dropped":2,"total":8},"read":{"bytes":2017},"write":{"bytes":7470}},"pipeline":{"clients":17,"events":{"active":1,"published":9,"total":9},"queue":{"acked":8}}},"registrar":{"states":{"current":17,"update":8},"writes":{"success":6,"total":6}},"system":{"load":{"1":0.16,"15":0.02,"5":0.08,"norm":{"1":0.08,"15":0.01,"5":0.04}}}}}}
```
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [September 26, 2018, 10:37pm UTC](https://discuss.elastic.co/t/filebeat-6-3-2-syslogs-are-not-logging-to-elastcisearch/149490/11 "2018-09-26T22:37:12Z")
Instance TIMEZONE: PDT
Wed Sep 26 15:33:02 PDT 2018
System logs with 7Hrs delay:
filebeat nginx logs:
```auto
|2018-09-26T15:35:12.150-0700|WARN|elasticsearch/client.go:502|Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbee32047c8c643e1, ext:435962396945171, loc:(*time.Location)(0x1f4bd20)}, Meta:common.MapStr{"pipeline":"filebeat-6.3.2-nginx-error-pipeline"}, Fields:common.MapStr{"input":common.MapStr{"type":"log"}, "fileset":common.MapStr{"module":"nginx", "name":"error"}, "prospector":common.MapStr{"type":"log"}, "beat":common.MapStr{"name":"nginx-i-0d7ae58bf3bf1cc26.k.dev.eng-us", "hostname":"nginx-i-0d7ae58bf3bf1cc26.k.dev.eng-us", "version":"6.3.2"}, "host":common.MapStr{"name":"nginx-i-0d7ae58bf3bf1cc26.k.dev.eng-us"}, "source":"/var/log/nginx/error.log", "offset":161383543, "message":"2018/09/26 15:35:04 [notice] 21344#21344: *73119 \"^\" matches \"/social/ads/289/289_00018/870121669856913/productItems.csv\", client: 10.201.21.48, server: dev-kaiser.*, request: \"GET /social/ads/289/289_00018/870121669856913/productItems.csv HTTP/1.1\", host: \"dev-service.reputation.com\""}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc4201e9d40), Source:"/var/log/nginx/error.log", Offset:161383829, Timestamp:time.Time{wall:0xbee32047c8bf27bb, ext:435962396479213, loc:(*time.Location)(0x1f4bd20)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xfb001, Device:0xca01}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-6.3.2-nginx-error-pipeline] does not exist"}|
|---|---|---|---|
|2018-09-26T15:35:12.151-0700|WARN|elasticsearch/client.go:502|Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbee32047c8cba876, ext:435962397298609, loc:(*time.Location)(0x1f4bd20)}, Meta:common.MapStr{"pipeline":"filebeat-6.3.2-nginx-error-pipeline"}, Fields:common.MapStr{"beat":common.MapStr{"name":"nginx-i-0d7ae58bf3bf1cc26.k.dev.eng-us", "hostname":"nginx-i-0d7ae58bf3bf1cc26.k.dev.eng-us", "version":"6.3.2"}, "host":common.MapStr{"name":"nginx-i-0d7ae58bf3bf1cc26.k.dev.eng-us"}, "source":"/var/log/nginx/error.log", "offset":161383829, "message":"2018/09/26 15:35:04 [notice] 21344#21344: *73119 rewritten redirect: \"https://dev-service.reputation.com/social/ads/289/289_00018/870121669856913/productItems.csv\", client: 10.201.21.48, server: dev-kaiser.*, request: \"GET /social/ads/289/289_00018/870121669856913/productItems.csv HTTP/1.1\", host: \"dev-service.reputation.com\"", "input":common.MapStr{"type":"log"}, "fileset":common.MapStr{"module":"nginx", "name":"error"}, "prospector":common.MapStr{"type":"log"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc4201e9d40), Source:"/var/log/nginx/error.log", Offset:161384157, Timestamp:time.Time{wall:0xbee32047c8bf27bb, ext:435962396479213, loc:(*time.Location)(0x1f4bd20)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xfb001, Device:0xca01}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-6.3.2-nginx-error-pipeline] does not exist"}|
```
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [September 28, 2018, 6:53pm UTC](https://discuss.elastic.co/t/filebeat-6-3-2-syslogs-are-not-logging-to-elastcisearch/149490/12 "2018-09-28T18:53:31Z")
I am expecting response bit faster. Can someone help me in fixing it. system and other defined modules are loading logs with 7hrs delay. I set true in the module yml files, still no luck.
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [October 1, 2018, 7:02am UTC](https://discuss.elastic.co/t/filebeat-6-3-2-syslogs-are-not-logging-to-elastcisearch/149490/13 "2018-10-01T07:02:50Z")
It seems you get an ingestion error because the pipeline does not exist:
```auto
"reason":"pipeline with id [filebeat-6.3.2-system-auth-pipeline] does not exist"}
```
How did you load the pipelines?
If you are expecting faster response times I would recommend you to look into our commercial support subscriptions: [https://www.elastic.co/subscriptions](https://www.elastic.co/subscriptions)
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [October 1, 2018, 8:03am UTC](https://discuss.elastic.co/t/filebeat-6-3-2-syslogs-are-not-logging-to-elastcisearch/149490/14 "2018-10-01T08:03:58Z")
I have not used any method to load the pipeline. I just installed filebeat on our instances and enabled system module on it.
I experienced the similar issues recently on our dev environment and i found a solution in beat's forums to deleted filebeat system pipelines by doing `curl -XDELETE 'http://localhost:9200/_ingest/pipeline/filebeat-*'`. and it worked. System module logs[syslog and auth.logs] are started populating to elasticsearch.
After i upgrade version, it started giving the same issues, and i gave try with most of options sugegsted in beat forums, but no luck.
Need help on it.
---
### Author:  [@shaunak](https://discuss.elastic.co/u/shaunak)
#### Post date: [October 2, 2018, 8:34pm UTC](https://discuss.elastic.co/t/filebeat-6-3-2-syslogs-are-not-logging-to-elastcisearch/149490/15 "2018-10-02T20:34:55Z")
@mouli_v, I'm having a hard time following this post as it is covering syslog, auth log, nginx logs, and timezone issues with some of these logs. May I suggest that we start with a very narrow surface area, get things working as expected and then expand from there?
For starters, could you disable all modules in your Filebeat. Then enable just the `system` module and within `modules.d/system.yml`, enable just the `syslog` metricset. Do not customize any further settings within the `syslog` metricset (that is, leave everything as default for now).
Next, for good measure, delete your Filebeat pipelines:
```auto
curl -XDELETE 'http://localhost:9200/_ingest/pipeline/filebeat-*'
```
Finally, with this setup, start up Filebeat. And let me know whether you are a) getting syslogs in Elasticsearch/Kibana and b) if their timezone is correct or wrong. We can then debug from here until we get syslogs working as expected. Once we've done that, we'll move on to auth logs, then nginx logs.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 30, 2018, 8:34pm UTC](https://discuss.elastic.co/t/filebeat-6-3-2-syslogs-are-not-logging-to-elastcisearch/149490/16 "2018-10-30T20:34:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.