# Fitebeat 6.3.1 system module issue
**URL:** https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587
**Category:** Beats
**Tags:** filebeat
**Created:** [July 25, 2018, 1:33pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587 "2018-07-25T13:33:35Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [July 25, 2018, 1:33pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/1 "2018-07-25T13:33:35Z")
recently I have installed fitebeat 6.3.1. I see the system logs are propagating to elasticsearch with 7hours delay.
I have set var.convert\_timezone true, still, I don't see those logs with current time.
system module file:
/etc/filebeat/modules.d/system.yml
- module: system
@andrewkroh, @ruflin , Would you please take a look into it.
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [July 25, 2018, 4:18pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/2 "2018-07-25T16:18:15Z")
**Here is the log:**
```auto
2018-07-25T09:15:38.932-0700 WARN elasticsearch/client.go:502 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbece466a77105996, ext:747401007978008, loc:(*time.Location)(0x1f4a300)}, Meta:common.MapStr{"pipeline":"filebeat-6.3.1-system-auth-pipeline"}, Fields:common.MapStr{"message":"Jul 25 09:15:36 autofind-i-0250f2df10123c4a3 sshd[25035]: User child is on pid 25085", "prospector":common.MapStr{"type":"log"}, "input":common.MapStr{"type":"log"}, "fileset":common.MapStr{"module":"system", "name":"auth"}, "beat":common.MapStr{"timezone":"-07:00", "name":"autofind-i-0250f2df10123c4a3.k.dev.eng-us", "hostname":"autofind-i-0250f2df10123c4a3.k.dev.eng-us", "version":"6.3.1"}, "host":common.MapStr{"name":"autofind-i-0250f2df10123c4a3.k.dev.eng-us"}, "source":"/var/log/auth.log", "offset":95506}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc420a5cc30), Source:"/var/log/auth.log", Offset:95591, Timestamp:time.Time{wall:0xbece25dd388af459, ext:714068032790436, loc:(*time.Location)(0x1f4a300)}, TTL:-1, Type:"log", Meta:map[string]string{}, FileStateOS:file.StateOS{Inode:0x10db, Device:0xca01}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-6.3.1-system-auth-pipeline] does not exist"}
2018-07-25T09:19:26.942-0700 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":53440,"time":{"ms":8}},"total":{"ticks":238730,"time":{"ms":8},"value":238730},"user":{"ticks":185290}},"info":{"ephemeral_id":"363fa537-770b-4948-8dae-afdf30a094a2","uptime":{"ms":747630020}},"memstats":{"gc_next":5694272,"memory_alloc":3940440,"memory_total":21587357176}},"filebeat":{"harvester":{"open_files":4,"running":4}},"libbeat":{"config":{"module":{"running":5}},"pipeline":{"clients":15,"events":{"active":0}}},"registrar":{"states":{"current":14}},"system":{"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}}}}
2018-07-25T09:19:56.942-0700 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":53440,"time":{"ms":4}},"total":{"ticks":238730,"time":{"ms":4},"value":238730},"user":{"ticks":185290}},"info":{"ephemeral_id":"363fa537-770b-4948-8dae-afdf30a094a2","uptime":{"ms":747660020}},"memstats":{"gc_next":5694272,"memory_alloc":4350864,"memory_total":21587767600}},"filebeat":{"harvester":{"open_files":4,"running":4}},"libbeat":{"config":{"module":{"running":5}},"pipeline":{"clients":15,"events":{"active":0}}},"registrar":{"states":{"current":14}},"system":{"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}}}}
2018-07-25T09:20:08.934-0700 WARN elasticsearch/client.go:502 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbece46adf79e7df0, ext:747671017293480, loc:(*time.Location)(0x1f4a300)}, Meta:common.MapStr{"pipeline":"filebeat-6.3.1-system-auth-pipeline"}, Fields:common.MapStr{"message":"Jul 25 09:20:01 autofind-i-0250f2df10123c4a3 CRON[25161]: pam_unix(cron:session): session opened for user root by (uid=0)", "input":common.MapStr{"type":"log"}, "fileset":common.MapStr{"module":"system", "name":"auth"}, "prospector":common.MapStr{"type":"log"}, "beat":common.MapStr{"timezone":"-07:00", "version":"6.3.1", "name":"autofind-i-0250f2df10123c4a3.k.dev.eng-us", "hostname":"autofind-i-0250f2df10123c4a3.k.dev.eng-us"}, "host":common.MapStr{"name":"autofind-i-0250f2df10123c4a3.k.dev.eng-us"}, "source":"/var/log/auth.log", "offset":95972}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc420a5cc30), Source:"/var/log/auth.log", Offset:96094, Timestamp:time.Time{wall:0xbece25dd388af459, ext:714068032790436, loc:(*time.Location)(0x1f4a300)}, TTL:-1, Type:"log", Meta:map[string]string{}, FileStateOS:file.StateOS{Inode:0x10db, Device:0xca01}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-6.3.1-system-auth-pipeline] does not exist"}
2018-07-25T09:20:13.934-0700 WARN elasticsearch/client.go:502 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbece46adf79e8d0c, ext:747671017297335, loc:(*time.Location)(0x1f4a300)}, Meta:common.MapStr{"pipeline":"filebeat-6.3.1-system-auth-pipeline"}, Fields:common.MapStr{"input":common.MapStr{"type":"log"}, "fileset":common.MapStr{"name":"auth", "module":"system"}, "prospector":common.MapStr{"type":"log"}, "beat":common.MapStr{"timezone":"-07:00", "name":"autofind-i-0250f2df10123c4a3.k.dev.eng-us", "hostname":"autofind-i-0250f2df10123c4a3.k.dev.eng-us", "version":"6.3.1"}, "host":common.MapStr{"name":"autofind-i-0250f2df10123c4a3.k.dev.eng-us"}, "source":"/var/log/auth.log", "offset":96094, "message":"Jul 25 09:20:01 autofind-i-0250f2df10123c4a3 CRON[25161]: pam_unix(cron:session): session closed for user root"}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc420a5cc30), Source:"/var/log/auth.log", Offset:96205, Timestamp:time.Time{wall:0xbece25dd388af459, ext:714068032790436, loc:(*time.Location)(0x1f4a300)}, TTL:-1, Type:"log", Meta:map[string]string{}, FileStateOS:file.StateOS{Inode:0x10db, Device:0xca01}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-6.3.1-system-auth-pipeline] does not exist"}
2018-07-25T09:20:26.942-0700 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":53440,"time":{"ms":8}},"total":{"ticks":238740,"time":{"ms":16},"value":238740},"user":{"ticks":185300,"time":{"ms":8}}},"info":{"ephemeral_id":"363fa537-770b-4948-8dae-afdf30a094a2","uptime":{"ms":747690019}},"memstats":{"gc_next":5695840,"memory_alloc":3572904,"memory_total":21588652112}},"filebeat":{"events":{"added":5,"done":5},"harvester":{"open_files":4,"running":4}},"libbeat":{"config":{"module":{"running":5}},"output":{"events":{"acked":3,"batches":2,"dropped":2,"total":5},"read":{"bytes":763},"write":{"bytes":3473}},"pipeline":{"clients":15,"events":{"active":0,"published":5,"total":5},"queue":{"acked":5}}},"registrar":{"states":{"current":14,"update":5},"writes":{"success":2,"total":2}},"system":{"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}}}}
}}}}
```
---
### Author:  [@shaunak](https://discuss.elastic.co/u/shaunak)
#### Post date: [July 25, 2018, 6:10pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/3 "2018-07-25T18:10:02Z")
Hi @mouli_v,
Could you provide the output of `curl -XGET "http://localhost:9200/filebeat-*/_search?_source=@timestamp,beat.timezone"`, replacing `localhost:9200` with your Elasticsearch host and port, if necessary?
Also, are the timestamps in your syslog in UTC or in your local time zone?
Shaunak
---
### Author:  [@shaunak](https://discuss.elastic.co/u/shaunak)
#### Post date: [July 25, 2018, 6:21pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/4 "2018-07-25T18:21:14Z")
Also, one more request:
Could you please provide the output of `curl -XGET "http://localhost:9200/_ingest/pipeline/file*syslog*"` as well?
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [July 25, 2018, 6:30pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/5 "2018-07-25T18:30:40Z")
> [@shaunak](#):
>
> curl -XGET "[http://elasticsearch:9200/\_ingest/pipeline/file\*syslog](http://elasticsearch:9200/_ingest/pipeline/file*syslog)\*"
* * *
curl -XGET "[http://localhost:9200/filebeat-\*/\_search?\_source=@timestamp,beat.timezone](http://localhost:9200/filebeat-*/_search?_source=@timestamp,beat.timezone)"
{"took":197,"timed\_out":false,"\_shards":{"total":36,"successful":36,"skipped":0,"failed":0},"hits":{"total":76008730,"max\_score":1.0,"hits":[{"\_index":"filebeat-6.3.1-2018.07.16","\_type":"doc","_id":"uSnypWQBdaushYXi0Dp_","\_score":1.0,"\_source":{"@timestamp":"2018-07-17T01:55:06.032Z"}},{"\_index":"filebeat-6.3.1-2018.07.16","\_type":"doc","_id":"uinypWQBdaushYXi0Dp_","\_score":1.0,"\_source":{"@timestamp":"2018-07-17T01:55:06.032Z"}},{"\_index":"filebeat-6.3.1-2018.07.16","\_type":"doc","_id":"vCnypWQBdaushYXi0Dp_","\_score":1.0,"\_source":{"@timestamp":"2018-07-17T01:55:06.032Z"}},{"\_index":"filebeat-6.3.1-2018.07.16","\_type":"doc","_id":"vSnypWQBdaushYXi0Dp_","\_score":1.0,"\_source":{"@timestamp":"2018-07-17T01:55:06.032Z"}},{"\_index":"filebeat-6.3.1-2018.07.16","\_type":"doc","_id":"vynypWQBdaushYXi0Dp_","\_score":1.0,"\_source":{"@timestamp":"2018-07-17T01:55:06.032Z"}},{"\_index":"filebeat-6.3.1-2018.07.16","\_type":"doc","_id":"wSnypWQBdaushYXi0Dp_","\_score":1.0,"\_source":{"@timestamp":"2018-07-17T01:55:06.032Z"}},{"\_index":"filebeat-6.3.1-2018.07.16","\_type":"doc","_id":"wynypWQBdaushYXi0Dp_","\_score":1.0,"\_source":{"@timestamp":"2018-07-17T01:55:06.032Z"}},{"\_index":"filebeat-6.3.1-2018.07.16","\_type":"doc","_id":"yCnypWQBdaushYXi0Dp_","\_score":1.0,"\_source":{"@timestamp":"2018-07-17T01:55:06.032Z"}},{"\_index":"filebeat-6.3.1-2018.07.16","\_type":"doc","_id":"zCnypWQBdaushYXi0Dp_","\_score":1.0,"\_source":{"@timestamp":"2018-07-17T01:55:06.032Z"}},{"\_index":"filebeat-6.3.1-2018.07.16","\_type":"doc","_id":"zynypWQBdaushYXi0Dp_","\_score":1.0,"\_source":{"@timestamp":"2018-07-17T01:55:06.032Z"}}]}}[quote="shaunak, post:3, topic:141587, full:true"]
Hi @mouli_v,
Could you provide the output of `curl -XGET "http://localhost:9200/filebeat-*/_search?_source=@timestamp,beat.timezone"`, replacing `localhost:9200` with your Elasticsearch host and port, if necessary?
Also, are the timestamps in your syslog in UTC or in your local time zone?
Shaunak
[/quote]
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [July 25, 2018, 6:31pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/6 "2018-07-25T18:31:17Z")
curl -XGET "[http://localhost:9200/\_ingest/pipeline/file](http://localhost:9200/_ingest/pipeline/file)_syslog_"
```auto
{
"filebeat-6.3.1-system-syslog-pipeline":{
"description":"Pipeline for parsing Syslog messages.",
"processors":[
{
"grok":{
"pattern_definitions":{
"GREEDYMULTILINE":"(.|\n)*"
},
"ignore_missing":true,
"field":"message",
"patterns":[
"%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{SYSLOGHOST:system.syslog.hostname} %{DATA:system.syslog.program}(?:\\[%{POSINT:system.syslog.pid}\\])?: %{GREEDYMULTILINE:system.syslog.message}",
"%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{GREEDYMULTILINE:system.syslog.message}"
]
}
},
{
"remove":{
"field":"message"
}
},
{
"date":{
"target_field":"@timestamp",
"formats":[
"MMM d HH:mm:ss",
"MMM dd HH:mm:ss"
],
"ignore_failure":true,
"field":"system.syslog.timestamp"
}
}
],
"on_failure":[
{
"set":{
"field":"error.message",
"value":"{{ _ingest.on_failure_message }}"
}
}
]
},
"filebeat-6.3.2-system-syslog-pipeline":{
"description":"Pipeline for parsing Syslog messages.",
"processors":[
{
"grok":{
"field":"message",
"patterns":[
"%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{SYSLOGHOST:system.syslog.hostname} %{DATA:system.syslog.program}(?:\\[%{POSINT:system.syslog.pid}\\])?: %{GREEDYMULTILINE:system.syslog.message}",
"%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{GREEDYMULTILINE:system.syslog.message}"
],
"pattern_definitions":{
"GREEDYMULTILINE":"(.|\n)*"
},
"ignore_missing":true
}
},
{
"remove":{
"field":"message"
}
},
{
"date":{
"field":"system.syslog.timestamp",
"target_field":"@timestamp",
"formats":[
"MMM d HH:mm:ss",
"MMM dd HH:mm:ss"
],
"ignore_failure":true
}
}
],
"on_failure":[
{
"set":{
"field":"error.message",
"value":"{{ _ingest.on_failure_message }}"
}
}
]
}
}
```
I ran on the elastic node.
---
### Author:  [@shaunak](https://discuss.elastic.co/u/shaunak)
#### Post date: [July 25, 2018, 6:33pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/7 "2018-07-25T18:33:02Z")
Sorry, the host name in the curl request should be `localhost`, not `elasticsearch`. Copy-paste error from me!
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [July 25, 2018, 6:35pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/8 "2018-07-25T18:35:40Z")
Yup; I have adjusted that; Please find the result i have shared.
---
### Author:  [@shaunak](https://discuss.elastic.co/u/shaunak)
#### Post date: [July 25, 2018, 6:38pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/9 "2018-07-25T18:38:54Z")
Okay, I see the issue now. Looks like your filebeat ingest pipelines need to be updated. The easiest way to do this is:
1. Stop `filebeat`
2. Run `curl -XDELETE "http://localhost:9200/_ingest/pipeline/filebeat-*"`
3. Start `filebeat`
To confirm that your filebeat ingest pipelines were indeed updated correctly, please run `curl -XGET "http://localhost:9200/_ingest/pipeline/file*syslog*"` again, and check that the `date` processor has the `"timezone": "{{ beat.timezone }}"` setting in it.
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [July 25, 2018, 6:46pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/10 "2018-07-25T18:46:33Z")
> [@shaunak](#):
>
> curl -XGET "[http://localhost:9200/\_ingest/pipeline/file\*syslog](http://localhost:9200/_ingest/pipeline/file*syslog)\*"
I already gave a try deleting old ingest/pipeline, still no luck.
`curl -XGET "http://localhost:9200/_ingest/pipeline/file*syslog*"`
```auto
{
"filebeat-6.3.1-system-syslog-pipeline":{
"description":"Pipeline for parsing Syslog messages.",
"processors":[
{
"grok":{
"pattern_definitions":{
"GREEDYMULTILINE":"(.|\n)*"
},
"ignore_missing":true,
"field":"message",
"patterns":[
"%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{SYSLOGHOST:system.syslog.hostname} %{DATA:system.syslog.program}(?:\\[%{POSINT:system.syslog.pid}\\])?: %{GREEDYMULTILINE:system.syslog.message}",
"%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{GREEDYMULTILINE:system.syslog.message}"
]
}
},
{
"remove":{
"field":"message"
}
},
{
"date":{
"target_field":"@timestamp",
"formats":[
"MMM d HH:mm:ss",
"MMM dd HH:mm:ss"
],
"ignore_failure":true,
"field":"system.syslog.timestamp"
}
}
],
"on_failure":[
{
"set":{
"field":"error.message",
"value":"{{ _ingest.on_failure_message }}"
}
}
]
}
}
```
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [July 25, 2018, 6:53pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/11 "2018-07-25T18:53:06Z")
> [@shaunak](#):
>
> curl -XGET "[http://localhost:9200/\_ingest/pipeline/file\*syslog](http://localhost:9200/_ingest/pipeline/file*syslog)\*"
We have 2 environments: i have verified on the both environments:
result for prod env:
curl -XGET "[http://localhost:9200/filebeat-\*/\_search?\_source=@timestamp,beat.timezone](http://localhost:9200/filebeat-*/_search?_source=@timestamp,beat.timezone)"
{"took":6,"timed\_out":false,"\_shards":{"total":18,"successful":18,"skipped":0,"failed":0},"hits":{"total":773273,"max\_score":1.0,"hits":[{"\_index":"filebeat-6.2.1-2018.07.25","\_type":"doc","\_id":"AE6k0WQB2cu211gImLq4","\_score":1.0,"\_source":{"@timestamp":"2018-07-25T13:33:01.320Z"}},{"\_index":"filebeat-6.2.1-2018.07.25","\_type":"doc","\_id":"A06k0WQB2cu211gImLq4","\_score":1.0,"\_source":{"@timestamp":"2018-07-25T13:33:01.320Z"}},{"\_index":"filebeat-6.2.1-2018.07.25","\_type":"doc","\_id":"Bk6k0WQB2cu211gImLq4","\_score":1.0,"\_source":{"@timestamp":"2018-07-25T13:33:01.320Z"}},{"\_index":"filebeat-6.2.1-2018.07.25","\_type":"doc","\_id":"CE6k0WQB2cu211gImLq4","\_score":1.0,"\_source":{"@timestamp":"2018-07-25T13:33:01.321Z"}},{"\_index":"filebeat-6.2.1-2018.07.25","\_type":"doc","\_id":"Ck6k0WQB2cu211gImLq4","\_score":1.0,"\_source":{"@timestamp":"2018-07-25T13:33:01.321Z"}},{"\_index":"filebeat-6.2.1-2018.07.25","\_type":"doc","\_id":"C06k0WQB2cu211gImLq4","\_score":1.0,"\_source":{"@timestamp":"2018-07-25T13:33:01.321Z"}},{"\_index":"filebeat-6.2.1-2018.07.25","\_type":"doc","\_id":"DE6k0WQB2cu211gImLq4","\_score":1.0,"\_source":{"@timestamp":"2018-07-25T13:33:01.321Z"}},{"\_index":"filebeat-6.2.1-2018.07.25","\_type":"doc","\_id":"D06k0WQB2cu211gImLq4","\_score":1.0,"\_source":{"@timestamp":"2018-07-25T13:33:01.321Z"}},{"\_index":"filebeat-6.2.1-2018.07.25","\_type":"doc","\_id":"EE6k0WQB2cu211gImLq4","\_score":1.0,"\_source":{"@timestamp":"2018-07-25T13:33:01.321Z"}},{"\_index":"filebeat-6.2.1-2018.07.25","\_type":"doc","\_id":"EU6k0WQB2cu211gImLq4","\_score":1.0,"\_source":{"@timestamp":"2018-07-25T13:33:01.321Z"}}]}}
---
### Author:  [@shaunak](https://discuss.elastic.co/u/shaunak)
#### Post date: [July 25, 2018, 6:57pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/12 "2018-07-25T18:57:42Z")
The problem definitely seems to be with the `filebeat-6.3.1-system-syslog-pipeline` ingest pipeline not getting updated even after you delete it and restart filebeat. I have to admit, I'm a bit stumped as to why that' s not working for you as it seems to be working for me (I've downloaded and tested with filebeat 6.3.1 as well).
After you run `curl -XDELETE "http://localhost:9200/_ingest/pipeline/filebeat-*"` but before you restart filebeat, can you run `curl -XGET "http://localhost:9200/_ingest/pipeline/file*syslog*"` again? I want to check if the pipeline is actually being deleted or not.
Shaunak
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [July 25, 2018, 7:02pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/13 "2018-07-25T19:02:44Z")
> [@shaunak](#):
>
> curl -XDELETE "[http://localhost:9200/\_ingest/pipeline/filebeat-](http://localhost:9200/_ingest/pipeline/filebeat-)\*"
curl -XGET "[http://localhost:9200/\_ingest/pipeline/file\*syslog](http://localhost:9200/_ingest/pipeline/file*syslog)\*"
{"filebeat-6.3.1-system-syslog-pipeline":{"description":"Pipeline for parsing Syslog messages.","processors":[{"grok":{"pattern\_definitions":{"GREEDYMULTILINE":"(.|\n)_"},"ignore\_missing":true,"field":"message","patterns":["%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{SYSLOGHOST:system.syslog.hostname} %{DATA:system.syslog.program}(?:\[%{POSINT:system.syslog.pid}\])?: %{GREEDYMULTILINE:system.syslog.message}","%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{GREEDYMULTILINE:system.syslog.message}"]}},{"remove":{"field":"message"}},{"date":{"target\_field":"@timestamp","formats":["MMM d HH:mm:ss","MMM dd HH:mm:ss"],"ignore\_failure":true,"field":"system.syslog.timestamp"}}],"on\_failure":[{"set":{"field":"error.message","value":"{{ \_ingest.on\_failure\_message }}"}}]},"filebeat-6.3.2-system-syslog-pipeline":{"description":"Pipeline for parsing Syslog messages.","processors":[{"grok":{"field":"message","patterns":["%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{SYSLOGHOST:system.syslog.hostname} %{DATA:system.syslog.program}(?:\[%{POSINT:system.syslog.pid}\])?: %{GREEDYMULTILINE:system.syslog.message}","%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{GREEDYMULTILINE:system.syslog.message}"],"pattern\_definitions":{"GREEDYMULTILINE":"(.|\n)_"},"ignore\_missing":true}},{"remove":{"field":"message"}},{"date":{"field":"system.syslog.timestamp","target\_field":"@timestamp","formats":["MMM d HH:mm:ss","MMM dd HH:mm:ss"],"ignore\_failure":true}}],"on\_failure":[{"set":{"value":"{{ \_ingest.on\_failure\_message }}","field":"error.message"}}]}}root@sherlock-i-0c82683e261bb829e:~#
systemctl stop filebeat.service
curl -XDELETE "[http://localhost:9200/\_ingest/pipeline/filebeat-](http://localhost:9200/_ingest/pipeline/filebeat-)\*"
{"acknowledged":true}
systemctl start filebeat.service
systemctl status filebeat.service
● filebeat.service - filebeat
Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2018-07-25 12:00:56 PDT; 3s ago
Docs: [Filebeat Reference [8.11] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/index.html)
Main PID: 29980 (filebeat)
Tasks: 12
Memory: 16.1M
CPU: 219ms
CGroup: /system.slice/filebeat.service
└─29980 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat
Jul 25 12:00:56 sherlock-i-0c82683e261bb829e.k.dev.eng-us systemd[1]: Started filebeat.
curl -XGET "[http://localhost:9200/\_ingest/pipeline/file\*syslog](http://localhost:9200/_ingest/pipeline/file*syslog)\*"
{"filebeat-6.3.1-system-syslog-pipeline":{"description":"Pipeline for parsing Syslog messages.","processors":[{"grok":{"patterns":["%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{SYSLOGHOST:system.syslog.hostname} %{DATA:system.syslog.program}(?:\[%{POSINT:system.syslog.pid}\])?: %{GREEDYMULTILINE:system.syslog.message}","%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{GREEDYMULTILINE:system.syslog.message}"],"pattern\_definitions":{"GREEDYMULTILINE":"(.|\n)\*"},"ignore\_missing":true,"field":"message"}},{"remove":{"field":"message"}},{"date":{"formats":["MMM d HH:mm:ss","MMM dd HH:mm:ss"],"ignore\_failure":true,"field":"system.syslog.timestamp","target\_field":"@timestamp"}}],"on\_failure":[{"set":{"field":"error.message","value":"{{ \_ingest.on\_failure\_message }}"}}]}}
---
### Author:  [@shaunak](https://discuss.elastic.co/u/shaunak)
#### Post date: [July 25, 2018, 7:05pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/14 "2018-07-25T19:05:24Z")
Sorry if I wasn't clear earlier. Lets ignore filebeat altogether for a moment. Stop it but **don't start it up at all**.
Then:
1. Run `curl -XDELETE "http://localhost:9200/_ingest/pipeline/filebeat-*"`
2. Run `curl -XGET "http://localhost:9200/_ingest/pipeline/file*syslog*"` and post it's response here.
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [July 25, 2018, 7:08pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/15 "2018-07-25T19:08:07Z")
> [@shaunak](#):
>
> curl -XGET "[http://localhost:9200/\_ingest/pipeline/file\*syslog](http://localhost:9200/_ingest/pipeline/file*syslog)\*"
i-0c82683e261bb829e:~# systemctl stop filebeat.service
i-0c82683e261bb829e:~# curl -XDELETE "[http://localhost:9200/\_ingest/pipeline/filebeat-](http://localhost:9200/_ingest/pipeline/filebeat-)\*"
{"acknowledged":true}
i-0c82683e261bb829e:~# curl -XGET "[http://localhost:9200/\_ingest/pipeline/file\*syslog](http://localhost:9200/_ingest/pipeline/file*syslog)\*"
{"filebeat-6.3.1-system-syslog-pipeline":{"on\_failure":[{"set":{"field":"error.message","value":"{{ \_ingest.on\_failure\_message }}"}}],"description":"Pipeline for parsing Syslog messages.","processors":[{"grok":{"patterns":["%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{SYSLOGHOST:system.syslog.hostname} %{DATA:system.syslog.program}(?:\[%{POSINT:system.syslog.pid}\])?: %{GREEDYMULTILINE:system.syslog.message}","%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{GREEDYMULTILINE:system.syslog.message}"],"pattern\_definitions":{"GREEDYMULTILINE":"(.|\n)\*"},"ignore\_missing":true,"field":"message"}},{"remove":{"field":"message"}},{"date":{"formats":["MMM d HH:mm:ss","MMM dd HH:mm:ss"],"ignore\_failure":true,"field":"system.syslog.timestamp","target\_field":"@timestamp"}}]}}
---
### Author:  [@shaunak](https://discuss.elastic.co/u/shaunak)
#### Post date: [July 25, 2018, 7:55pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/16 "2018-07-25T19:55:52Z")
Wow, this is bizarre. It would appear that you are unable to delete the ingest pipeline! Or something is immediately recreating it. Is it possible that there's a filebeat process somewhere that keeps restarting?
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [July 25, 2018, 8:04pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/17 "2018-07-25T20:04:11Z")
> [@shaunak](#):
>
> bizarre
Is it possible that there's a filebeat process somewhere that keeps restarting?
No. service still on stopped state on elastisearch instance.
Filebeat service is stopped on on elasticsearch instance.I have installed filebeat on 50+ servers and those all will be pointing to same elasticserver instance. Do you think that keeps recreating the file, as the service filebeat is running on them.
---
### Author:  [@shaunak](https://discuss.elastic.co/u/shaunak)
#### Post date: [July 25, 2018, 8:29pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/18 "2018-07-25T20:29:43Z")
Alright, lets try one more thing.
1. Run `curl -XDELETE "http://localhost:9200/_ingest/pipeline/filebeat-*"`
2. Edit your `filebeat.yml` and add: `filebeat.overwrite_pipelines: true`
3. Start up filebeat
4. In your filebeat logs, check if you have two lines like these:
5. Run `curl -XGET "http://localhost:9200/_ingest/pipeline/file*syslog*"` and post it's response here.
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [July 25, 2018, 8:44pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/19 "2018-07-25T20:44:04Z")
> [@shaunak](#):
>
> curl -XGET "[http://localhost:9200/\_ingest/pipeline/file\*syslog](http://localhost:9200/_ingest/pipeline/file*syslog)\*"
No luck,
I have enabled other custom modules, which i could see them in log, not system:
2018-07-25T13:41:45.099-0700 INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.2.4
2018-07-25T13:41:45.100-0700 INFO template/load.go:73 Template already exists and will not be overwritten.
2018-07-25T13:41:45.119-0700 INFO fileset/pipelines.go:62 Elasticsearch pipeline with ID 'filebeat-6.3.1-ansible-log-pipeline' loaded
2018-07-25T13:41:45.132-0700 INFO fileset/pipelines.go:62 Elasticsearch pipeline with ID 'filebeat-6.3.1-aptitude-log-pipeline' loaded
2018-07-25T13:41:45.178-0700 INFO fileset/pipelines.go:62 Elasticsearch pipeline with ID 'filebeat-6.3.1-dpkg-log-pipeline' loaded
2018-07-25T13:41:45.227-0700 INFO fileset/pipelines.go:62 Elasticsearch pipeline with ID 'filebeat-6.3.1-mail-log-pipeline' loaded
2018-07-25T13:42:10.929-0700 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":60,"time":{"ms":8}},"total":{"ticks":310,"time":{"ms":12},"value":310},"user":{"ticks":250,"time":{"ms":4}}},"info":{"ephemeral\_id":"84990714-8d64-461a-899c-0a4e1946f224","uptime":{"ms":210009}},"memstats":{"gc\_next":15048528,"memory\_alloc":9175640,"memory\_total":36340152,"rss":-8192}},"filebeat":{"harvester":{"open\_files":3,"running":3}},"libbeat":{"config":{"module":{"running":5}},"output":{"read":{"bytes":1415},"write":{"bytes":4508}},"pipeline":{"clients":13,"events":{"active":4119,"retry":50}}},"registrar":{"states":{"current":13}},"system":{"load":{"1":1.69,"15":0.97,"5":1.24,"norm":{"1":0.4225,"15":0.2425,"5":0.31}}}}}}
2018-07-25T13:42:40.929-0700 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":70,"time":{"ms":4}},"total":{"ticks":320,"time":{"ms":8},"value":320},"user":{"ticks":250,"time":{"ms":4}}},"info":{"ephemeral\_id":"84990714-8d64-461a-899c-0a4e1946f224","uptime":{"ms":240009}},"memstats":{"gc\_next":15048528,"memory\_alloc":9541408,"memory\_total":36705920}},"filebeat":{"harvester":{"open\_files":3,"running":3}},"libbeat":{"config":{"module":{"running":5}},"pipeline":{"clients":13,"events":{"active":4119}}},"registrar":{"states":{"current":13}},"system":{"load":{"1":1.39,"15":0.97,"5":1.22,"norm":{"1":0.3475,"15":0.2425,"5":0.305}}}}}}
2018-07-25T13:42:45.238-0700 ERROR pipeline/output.go:74 Failed to connect: Connection marked as failed because the onConnect callback failed: Error loading pipeline for fileset system/auth: This module requires the ingest-geoip plugin to be installed in Elasticsearch. You can install it using the following command in the Elasticsearch home directory:
sudo bin/elasticsearch-plugin install ingest-geoip
--
2018-07-25T13:52:10.929-0700 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":120,"time":{"ms":4}},"total":{"ticks":560,"time":{"ms":12},"value":560},"user":{"ticks":440,"time":{"ms":8}}},"info":{"ephemeral\_id":"84990714-8d64-461a-899c-0a4e1946f224","uptime":{"ms":810008}},"memstats":{"gc\_next":14949840,"memory\_alloc":9194328,"memory\_total":47956416}},"filebeat":{"harvester":{"open\_files":3,"running":3}},"libbeat":{"config":{"module":{"running":5}},"output":{"read":{"bytes":1415},"write":{"bytes":4508}},"pipeline":{"clients":13,"events":{"active":4119,"retry":50}}},"registrar":{"states":{"current":13}},"system":{"load":{"1":1.85,"15":1.2,"5":1.46,"norm":{"1":0.4625,"15":0.3,"5":0.365}}}}}}
2018-07-25T13:52:40.929-0700 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":130},"total":{"ticks":570,"time":{"ms":4},"value":570},"user":{"ticks":440,"time":{"ms":4}}},"info":{"ephemeral\_id":"84990714-8d64-461a-899c-0a4e1946f224","uptime":{"ms":840008}},"memstats":{"gc\_next":14949840,"memory\_alloc":9556672,"memory\_total":48318760}},"filebeat":{"harvester":{"open\_files":3,"running":3}},"libbeat":{"config":{"module":{"running":5}},"pipeline":{"clients":13,"events":{"active":4119}}},"registrar":{"states":{"current":13}},"system":{"load":{"1":2.32,"15":1.25,"5":1.6,"norm":{"1":0.58,"15":0.3125,"5":0.4}}}}}}
2018-07-25T13:52:46.993-0700 ERROR pipeline/output.go:74 Failed to connect: Connection marked as failed because the onConnect callback failed: Error loading pipeline for fileset system/auth: This module requires the ingest-geoip plugin to be installed in Elasticsearch. You can install it using the following command in the Elasticsearch home directory:
sudo bin/elasticsearch-plugin install ingest-geoip
2018-07-25T13:52:46.993-0700 INFO [publish] pipeline/retry.go:172 retryer: send unwait-signal to consumer
2018-07-25T13:52:46.993-0700 INFO [publish] pipeline/retry.go:174 done
2018-07-25T13:52:46.993-0700 INFO [publish] pipeline/retry.go:149 retryer: send wait signal to consumer
2018-07-25T13:52:46.994-0700 INFO [publish] pipeline/retry.go:151 done
2018-07-25T13:52:46.994-0700 INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.2.4
2018-07-25T13:52:46.997-0700 INFO template/load.go:73 Template already exists and will not be overwritten.
2018-07-25T13:52:47.013-0700 INFO fileset/pipelines.go:62 Elasticsearch pipeline with ID 'filebeat-6.3.1-ansible-log-pipeline' loaded
2018-07-25T13:52:47.048-0700 INFO fileset/pipelines.go:62 Elasticsearch pipeline with ID 'filebeat-6.3.1-aptitude-log-pipeline' loaded
2018-07-25T13:52:47.063-0700 INFO fileset/pipelines.go:62 Elasticsearch pipeline with ID 'filebeat-6.3.1-dpkg-log-pipeline' loaded
2018-07-25T13:52:47.098-0700 INFO fileset/pipelines.go:62 Elasticsearch pipeline with ID 'filebeat-6.3.1-mail-log-pipeline' loaded
2018-07-25T13:52:47.136-0700 INFO fileset/pipelines.go:62 Elasticsearch pipeline with ID 'filebeat-6.3.1-system-syslog-pipeline' loaded
---
### Author:  [@mouli\_v](https://discuss.elastic.co/u/mouli_v)
#### Post date: [July 25, 2018, 8:48pm UTC](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587/20 "2018-07-25T20:48:56Z")
> [@mouli\_v](#):
>
> 2018-07-25T13:41:45.099-0700 INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.2.4
Do you think, it is causing issues:
2018-07-25T13:41:45.099-0700 INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.2.4
Elasticsearch version 6.2.4
[Next page](https://discuss.elastic.co/t/fitebeat-6-3-1-system-module-issue/141587.md?page=2)