Filebeat 6.3.2 syslogs are not logging to elastcisearch

Elastic Stack Beats
1

System module is enabled, but syslogs are logging with 7hrs delay. Adjusted var.convert_timezone: true in /etc/filebeat/modules.d/system.yml, still no luck. Can someone take a look into it.
/etc/filebeat/modules.d/system.yml

- module: system
  # Syslog
  syslog:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    var.convert_timezone: true

  # Authorization logs
  auth:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    var.convert_timezone: true

Filebeat_log:

|2018-09-21T09:43:51.260-0700|INFO|[monitoring]|log/log.go:124|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1200,"time":{"ms":1204}},"total":{"ticks":10000,"time":{"ms":10012},"value":10000},"user":{"ticks":8800,"time":{"ms":8808}}},"info":{"ephemeral_id":"edceef24-4069-4d3a-a648-f7947d9927bf","uptime":{"ms":30009}},"memstats":{"gc_next":11220912,"memory_alloc":5683112,"memory_total":830369184,"rss":58142720}},"filebeat":{"events":{"added":147363,"done":147363},"harvester":{"open_files":18,"running":18,"started":18}},"libbeat":{"config":{"module":{"running":7,"starts":7},"reloads":1},"output":{"events":{"acked":146558,"batches":2936,"total":146558},"read":{"bytes":2045860},"type":"elasticsearch","write":{"bytes":87199374}},"pipeline":{"clients":17,"events":{"active":0,"filtered":805,"published":146558,"retry":50,"total":147363},"queue":{"acked":146558}}},"registrar":{"states":{"current":17,"update":147363},"writes":{"success":2923,"total":2923}},"system":{"cpu":{"cores":2},"load":{"1":0.69,"15":0.18,"5":0.3,"norm":{"1":0.345,"15":0.09,"5":0.15}}}}}}|
|---|---|---|---|---|---|
|2018-09-21T09:44:21.259-0700|INFO|[monitoring]|log/log.go:124|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1210,"time":{"ms":8}},"total":{"ticks":10030,"time":{"ms":24},"value":10030},"user":{"ticks":8820,"time":{"ms":16}}},"info":{"ephemeral_id":"edceef24-4069-4d3a-a648-f7947d9927bf","uptime":{"ms":60009}},"memstats":{"gc_next":11220912,"memory_alloc":7146264,"memory_total":831832336}},"filebeat":{"events":{"added":4,"done":4},"harvester":{"open_files":18,"running":18}},"libbeat":{"config":{"module":{"running":7}},"output":{"events":{"acked":4,"batches":4,"total":4},"read":{"bytes":1374},"write":{"bytes":4653}},"pipeline":{"clients":17,"events":{"active":0,"published":4,"total":4},"queue":{"acked":4}}},"registrar":{"states":{"current":17,"update":4},"writes":{"success":4,"total":4}},"system":{"load":{"1":0.42,"15":0.17,"5":0.27,"norm":{"1":0.21,"15":0.085,"5":0.135}}}}}}|
|2018-09-21T09:44:51.259-0700|INFO|[monitoring]|log/log.go:124|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1210},"total":{"ticks":10060,"time":{"ms":28},"value":10060},"user":{"ticks":8850,"time":{"ms":28}}},"info":{"ephemeral_id":"edceef24-4069-4d3a-a648-f7947d9927bf","uptime":{"ms":90009}},"memstats":{"gc_next":11220912,"memory_alloc":9304584,"memory_total":833990656}},"filebeat":{"events":{"added":11,"done":11},"harvester":{"open_files":18,"running":18}},"libbeat":{"config":{"module":{"running":7}},"output":{"events":{"acked":11,"batches":7,"total":11},"read":{"bytes":2454},"write":{"bytes":10085}},"pipeline":{"clients":17,"events":{"active":0,"published":11,"total":11},"queue":{"acked":11}}},"registrar":{"states":{"current":17,"update":11},"writes":{"success":7,"total":7}},"system":{"load":{"1":0.25,"15":0.17,"5":0.24,"norm":{"1":0.125,"15":0.085,"5":0.12}}}}}}|
|2018-09-21T09:45:21.259-0700|INFO|[monitoring]|log/log.go:124|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1210,"time":{"ms":4}},"total":{"ticks":10090,"time":{"ms":32},"value":10090},"user":{"ticks":8880,"time":{"ms":28}}},"info":{"ephemeral_id":"edceef24-4069-4d3a-a648-f7947d9927bf","uptime":{"ms":120008}},"memstats":{"gc_next":11219376,"memory_alloc":6261536,"memory_total":835962168}},"filebeat":{"events":{"added":12,"done":12},"harvester":{"open_files":18,"running":18}},"libbeat":{"config":{"module":{"running":7}},"output":{"events":{"acked":12,"batches":6,"total":12},"read":{"bytes":2125},"write":{"bytes":9585}},"pipeline":{"clients":17,"events":{"active":0,"published":12,"total":12},"queue":{"acked":12}}},"registrar":{"states":{"current":17,"update":12},"writes":{"success":6,"total":6}},"system":{"load":{"1":0.15,"15":0.16,"5":0.22,"norm":{"1":0.075,"15":0.08,"5":0.11}}}}}}|

I see auth logs are coming as expected with proper timestamp.

2

Are the 7h difference the difference from your time zone?

3

Yes; They are logging with 7hrs delay; I tried curl -XDELETE 'http://localhost:9200/_ingest/pipeline/filebeat-*'. Still i see it is loading to elasticsearch with delay.

4

curl -XGET "http://localhost:9200/filebeat-*/_search?_source=@timestamp,beat.timezone"

{"took":39,"timed_out":false,"_shards":{"total":24,"successful":24,"skipped":0,"failed":0},"hits":{"total":5252536,"max_score":1.0,"hits":[{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"ZOTQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.311Z"}},{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"Z-TQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.311Z"}},{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"aeTQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.311Z"}},{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"cuTQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.311Z"}},{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"eOTQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.311Z"}},{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"euTQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.311Z"}},{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"fOTQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.311Z"}},{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"f-TQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.311Z"}},{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"guTQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.311Z"}},{"_index":"filebeat-6.2.4-2018.09.19","_type":"doc","_id":"ieTQ8GUBLr4RUOpqb0WB","_score":1.0,"_source":{"@timestamp":"2018-09-19T07:51:56.314Z"}}]}}

curl -XGET "http://localhost:9200/filebeat-*/_search?_source=@timestamp,beat.timezone"


{"filebeat-6.3.2-system-syslog-pipeline":{"description":"Pipeline for parsing Syslog messages.","processors":[{"grok":{"field":"message","patterns":["%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{SYSLOGHOST:system.syslog.hostname} %{DATA:system.syslog.program}(?:\\[%{POSINT:system.syslog.pid}\\])?: %{GREEDYMULTILINE:system.syslog.message}","%{SYSLOGTIMESTAMP:system.syslog.timestamp} %{GREEDYMULTILINE:system.syslog.message}"],"pattern_definitions":{"GREEDYMULTILINE":"(.|\n)*"},"ignore_missing":true}},{"remove":{"field":"message"}},{"date":{"field":"system.syslog.timestamp","target_field":"@timestamp","formats":["MMM d HH:mm:ss","MMM dd HH:mm:ss"],"ignore_failure":true}}],"on_failure":[{"set":{"value":"{{ _ingest.on_failure_message }}","field":"error.message"}}]}}
5

Deleted filebeat ingest and restart filebeat.

 curl -XDELETE 'http://localhost:9200/_ingest/pipeline/filebeat-*'

Now i see the syslogs are getting loaded but not authlogs.

2018-09-25T03:08:11.237-0700	WARN	elasticsearch/client.go:502	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbee2a02174713ca8, ext:482829647661, loc:(*time.Location)(0x1f4bd20)}, Meta:common.MapStr{"pipeline":"filebeat-6.3.2-system-auth-pipeline"}, Fields:common.MapStr{"host":common.MapStr{"name":"elasticsearch"}, "source":"/var/log/auth.log", "offset":275099, "message":"Sep 25 03:08:04 elasticsearch sshd[2777]: Close session: user rdmon from 2600:1f18:2270:741e:bc52:4f7:4084:8855 port 43138 id 0", "fileset":common.MapStr{"module":"system", "name":"auth"}, "prospector":common.MapStr{"type":"log"}, "input":common.MapStr{"type":"log"}, "beat":common.MapStr{"timezone":"-07:00", "version":"6.3.2", "name":"elasticsearch", "hostname":"elasticsearch"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc42040ed00), Source:"/var/log/auth.log", Offset:275247, Timestamp:time.Time{wall:0xbee29fa8c7ccc9dd, ext:80672892, loc:(*time.Location)(0x1f4bd20)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x2964, Device:0xca01}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-6.3.2-system-auth-pipeline] does not exist"}

System module config yml:

- module: system
  # Syslog
  syslog:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    #var.convert_timezone: false

  # Authorization logs
  auth:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    var.convert_timezone: true

I see the same issues for nginx module as well.

WARN	elasticsearch/client.go:502	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbee2a19ec821c592, ext:306262386164928, loc:(*time.Location)(0x1f4bd20)}, Meta:common.MapStr{"pipeline":"filebeat-6.3.2-nginx-error-pipeline"}, Fields:common.MapStr{"source":"/var/log/nginx/error.log", "offset":51599584, "message":"2018/09/25 03:33:30 [debug] 19308#19308: *20729 accept: 10.201.21.48:31570 fd:16", "input":common.MapStr{"type":"log"}, "fileset":common.MapStr{"module":"nginx", "name":"error"}, "prospector":common.MapStr{"type":"log"}, "beat":common.MapStr{"name":"nginx", "hostname":"nginx", "version":"6.3.2"}, "host":common.MapStr{"name":"nginx"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc42048c8f0), Source:"/var/log/nginx/error.log", Offset:51599665, Timestamp:time.Time{wall:0xbee2951f6a1a6bc6, ext:293464956108620, loc:(*time.Location)(0x1f4bd20)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xfa4ec, Device:0xca01}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-6.3.2-nginx-error-pipeline] does not exist"}
2018-09-25T03:33:32.140-0700	WARN	elasticsearch/client.go:502	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbee2a19ec821de3f, ext:306262386171246, loc:(*time.Location)(0x1f4bd20)}, Meta:common.MapStr{"pipeline":"filebeat-6.3.2-nginx-error-pipeline"}, Fields:common.MapStr{"source":"/var/log/nginx/error.log", "offset":51599665, "message":"2018/09/25 03:33:30 [debug] 19308#19308: *20729 event timer add: 16: 60000:1537871670747", "input":common.MapStr{"type":"log"}, "fileset":common.MapStr{"module":"nginx", "name":"error"}, "prospector":common.MapStr{"type":"log"}, "beat":common.MapStr{"hostname":"nginx", "version":"6.3.2", "name":"nginx"}, "host":common.MapStr{"name":"nginx"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc42048c8f0), Source:"/var/log/nginx/error.log", Offset:51599754, Timestamp:time.Time{wall:0xbee2951f6a1a6bc6, ext:293464956108620, loc:(*time.Location)(0x1f4bd20)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xfa4ec, Device:0xca01}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-6.3.2-nginx-error-pipeline] does not exist"}
2018-09-25T03:33:32.140-0700	WARN	elasticsearch/client.go:502	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbee2a19ec82243dc, ext:306262386197271, loc:(*time.Location)(0x1f4bd20)}, Meta:common.MapStr{"pipeline":"filebeat-6.3.2-nginx-error-pipeline"}, Fields:common.MapStr{"offset":51599754, "message":"2018/09/25 03:33:30 [debug] 19308#19308: *20729 reusable connection: 1", "input":common.MapStr{"type":"log"}, "fileset":common.MapStr{"module":"nginx", "name":"error"}, "prospector":common.MapStr{"type":"log"}, "beat":common.MapStr{"version":"6.3.2", "name":"nginx", "hostname":"nginx"}, "host":common.MapStr{"name":"nginx"}, "source":"/var/log/nginx/error.log"}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc42048c8f0), Source:"/var/log/nginx/error.log", Offset:51599825, Timestamp:time.Time{wall:0xbee2951f6a1a6bc6, ext:293464956108620, loc:(*time.Location)(0x1f4bd20)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xfa4ec, Device:0xca01}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-6.3.2-nginx-error-pipeline] does not exist"}
2018-09-25T03:33:32.140-0700
6

34%20PM

7

I dont see auth/nginx pipelines/logs in GET _ingest/pipeline/filebeat-6.3.2-*

I also noticed system logs are logging with 7hrs delay and nginx logs are not even logging.

@ruflin, could you take a look.

8

Can someone take a look into it.

9

Are you sure there is a 7h delay or is it potentially a time zone issue and your timestamps are in the future? That is why I asked above about the time zone? What time zone are your servers in?

BTW: Instead of an image please share things as code and please be patient.

10

Here is the log:

2018-09-26T15:29:53.097-0700	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":17410,"time":{"ms":12}},"total":{"ticks":85510,"time":{"ms":32},"value":85510},"user":{"ticks":68100,"time":{"ms":20}}},"info":{"ephemeral_id":"6746f90b-2141-43fb-8275-dd6580801654","uptime":{"ms":100110023}},"memstats":{"gc_next":6927696,"memory_alloc":4833968,"memory_total":6773670520}},"filebeat":{"events":{"active":-1,"added":49,"done":50},"harvester":{"open_files":3,"running":3}},"libbeat":{"config":{"module":{"running":6}},"output":{"events":{"acked":32,"batches":9,"dropped":18,"total":50},"read":{"bytes":3337},"write":{"bytes":35236}},"pipeline":{"clients":17,"events":{"active":0,"published":49,"total":49},"queue":{"acked":50}}},"registrar":{"states":{"current":17,"update":50},"writes":{"success":9,"total":9}},"system":{"load":{"1":0.14,"15":0.02,"5":0.07,"norm":{"1":0.07,"15":0.01,"5":0.035}}}}}}
2018-09-26T15:30:05.516-0700	WARN	elasticsearch/client.go:502	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbee31ffb1e9de2f1, ext:100121443294109, loc:(*time.Location)(0x1f4bd20)}, Meta:common.MapStr{"pipeline":"filebeat-6.3.2-system-auth-pipeline"}, Fields:common.MapStr{"offset":498030, "message":"Sep 26 15:30:01 elasticsearch-i-0c1f8e517148213d1 CRON[4286]: pam_unix(cron:session): session opened for user root by (uid=0)", "fileset":common.MapStr{"module":"system", "name":"auth"}, "prospector":common.MapStr{"type":"log"}, "input":common.MapStr{"type":"log"}, "beat":common.MapStr{"timezone":"-07:00", "hostname":"elasticsearch01.rep", "version":"6.3.2", "name":"elasticsearch01.rep"}, "host":common.MapStr{"name":"elasticsearch01.rep"}, "source":"/var/log/auth.log"}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc4202651e0), Source:"/var/log/auth.log", Offset:498156, Timestamp:time.Time{wall:0xbee2e97d55f25142, ext:44322297832655, loc:(*time.Location)(0x1f4bd20)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x252f, Device:0xca01}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-6.3.2-system-auth-pipeline] does not exist"}
2018-09-26T15:30:10.517-0700	WARN	elasticsearch/client.go:502	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbee31ffb1e9df126, ext:100121443297745, loc:(*time.Location)(0x1f4bd20)}, Meta:common.MapStr{"pipeline":"filebeat-6.3.2-system-auth-pipeline"}, Fields:common.MapStr{"message":"Sep 26 15:30:01 elasticsearch-i-0c1f8e517148213d1 CRON[4286]: pam_unix(cron:session): session closed for user root", "fileset":common.MapStr{"name":"auth", "module":"system"}, "prospector":common.MapStr{"type":"log"}, "input":common.MapStr{"type":"log"}, "beat":common.MapStr{"version":"6.3.2", "name":"elasticsearch01.rep", "hostname":"elasticsearch01.rep", "timezone":"-07:00"}, "host":common.MapStr{"name":"elasticsearch01.rep"}, "source":"/var/log/auth.log", "offset":498156}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc4202651e0), Source:"/var/log/auth.log", Offset:498271, Timestamp:time.Time{wall:0xbee2e97d55f25142, ext:44322297832655, loc:(*time.Location)(0x1f4bd20)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x252f, Device:0xca01}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-6.3.2-system-auth-pipeline] does not exist"}
2018-09-26T15:30:23.097-0700	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":17420,"time":{"ms":4}},"total":{"ticks":85550,"time":{"ms":28},"value":85550},"user":{"ticks":68130,"time":{"ms":24}}},"info":{"ephemeral_id":"6746f90b-2141-43fb-8275-dd6580801654","uptime":{"ms":100140023}},"memstats":{"gc_next":6852368,"memory_alloc":3852688,"memory_total":6775627920}},"filebeat":{"events":{"active":1,"added":9,"done":8},"harvester":{"open_files":3,"running":3}},"libbeat":{"config":{"module":{"running":6}},"output":{"events":{"acked":6,"batches":6,"dropped":2,"total":8},"read":{"bytes":2017},"write":{"bytes":7470}},"pipeline":{"clients":17,"events":{"active":1,"published":9,"total":9},"queue":{"acked":8}}},"registrar":{"states":{"current":17,"update":8},"writes":{"success":6,"total":6}},"system":{"load":{"1":0.16,"15":0.02,"5":0.08,"norm":{"1":0.08,"15":0.01,"5":0.04}}}}}}
11

Instance TIMEZONE: PDT
Wed Sep 26 15:33:02 PDT 2018
System logs with 7Hrs delay:

07%20AM
Screen Shot 2018-09-27 at 3.59.07 AM.png2452×658 78.6 KB

filebeat nginx logs:

|2018-09-26T15:35:12.150-0700|WARN|elasticsearch/client.go:502|Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbee32047c8c643e1, ext:435962396945171, loc:(*time.Location)(0x1f4bd20)}, Meta:common.MapStr{"pipeline":"filebeat-6.3.2-nginx-error-pipeline"}, Fields:common.MapStr{"input":common.MapStr{"type":"log"}, "fileset":common.MapStr{"module":"nginx", "name":"error"}, "prospector":common.MapStr{"type":"log"}, "beat":common.MapStr{"name":"nginx-i-0d7ae58bf3bf1cc26.k.dev.eng-us", "hostname":"nginx-i-0d7ae58bf3bf1cc26.k.dev.eng-us", "version":"6.3.2"}, "host":common.MapStr{"name":"nginx-i-0d7ae58bf3bf1cc26.k.dev.eng-us"}, "source":"/var/log/nginx/error.log", "offset":161383543, "message":"2018/09/26 15:35:04 [notice] 21344#21344: *73119 \"^\" matches \"/social/ads/289/289_00018/870121669856913/productItems.csv\", client: 10.201.21.48, server: dev-kaiser.*, request: \"GET /social/ads/289/289_00018/870121669856913/productItems.csv HTTP/1.1\", host: \"dev-service.reputation.com\""}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc4201e9d40), Source:"/var/log/nginx/error.log", Offset:161383829, Timestamp:time.Time{wall:0xbee32047c8bf27bb, ext:435962396479213, loc:(*time.Location)(0x1f4bd20)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xfb001, Device:0xca01}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-6.3.2-nginx-error-pipeline] does not exist"}|
|---|---|---|---|
|2018-09-26T15:35:12.151-0700|WARN|elasticsearch/client.go:502|Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbee32047c8cba876, ext:435962397298609, loc:(*time.Location)(0x1f4bd20)}, Meta:common.MapStr{"pipeline":"filebeat-6.3.2-nginx-error-pipeline"}, Fields:common.MapStr{"beat":common.MapStr{"name":"nginx-i-0d7ae58bf3bf1cc26.k.dev.eng-us", "hostname":"nginx-i-0d7ae58bf3bf1cc26.k.dev.eng-us", "version":"6.3.2"}, "host":common.MapStr{"name":"nginx-i-0d7ae58bf3bf1cc26.k.dev.eng-us"}, "source":"/var/log/nginx/error.log", "offset":161383829, "message":"2018/09/26 15:35:04 [notice] 21344#21344: *73119 rewritten redirect: \"https://dev-service.reputation.com/social/ads/289/289_00018/870121669856913/productItems.csv\", client: 10.201.21.48, server: dev-kaiser.*, request: \"GET /social/ads/289/289_00018/870121669856913/productItems.csv HTTP/1.1\", host: \"dev-service.reputation.com\"", "input":common.MapStr{"type":"log"}, "fileset":common.MapStr{"module":"nginx", "name":"error"}, "prospector":common.MapStr{"type":"log"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc4201e9d40), Source:"/var/log/nginx/error.log", Offset:161384157, Timestamp:time.Time{wall:0xbee32047c8bf27bb, ext:435962396479213, loc:(*time.Location)(0x1f4bd20)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xfb001, Device:0xca01}}}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-6.3.2-nginx-error-pipeline] does not exist"}|

03%20AM
Screen Shot 2018-09-27 at 4.08.03 AM.png2458×832 223 KB

12

I am expecting response bit faster. Can someone help me in fixing it. system and other defined modules are loading logs with 7hrs delay. I set true in the module yml files, still no luck.

13

It seems you get an ingestion error because the pipeline does not exist:

"reason":"pipeline with id [filebeat-6.3.2-system-auth-pipeline] does not exist"}

How did you load the pipelines?

If you are expecting faster response times I would recommend you to look into our commercial support subscriptions: https://www.elastic.co/subscriptions

14

I have not used any method to load the pipeline. I just installed filebeat on our instances and enabled system module on it.

I experienced the similar issues recently on our dev environment and i found a solution in beat's forums to deleted filebeat system pipelines by doing curl -XDELETE 'http://localhost:9200/_ingest/pipeline/filebeat-*'. and it worked. System module logs[syslog and auth.logs] are started populating to elasticsearch.

After i upgrade version, it started giving the same issues, and i gave try with most of options sugegsted in beat forums, but no luck.

Need help on it.

15

@mouli_v, I'm having a hard time following this post as it is covering syslog, auth log, nginx logs, and timezone issues with some of these logs. May I suggest that we start with a very narrow surface area, get things working as expected and then expand from there?

For starters, could you disable all modules in your Filebeat. Then enable just the system module and within modules.d/system.yml, enable just the syslog metricset. Do not customize any further settings within the syslog metricset (that is, leave everything as default for now).

Next, for good measure, delete your Filebeat pipelines:

 curl -XDELETE 'http://localhost:9200/_ingest/pipeline/filebeat-*'

Finally, with this setup, start up Filebeat. And let me know whether you are a) getting syslogs in Elasticsearch/Kibana and b) if their timezone is correct or wrong. We can then debug from here until we get syslogs working as expected. Once we've done that, we'll move on to auth logs, then nginx logs.

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.