Filebeat is not shipping the log data to Elasticsearch

I am following the use case for Machine Learning for Elastic Stack found at the link below:
Suspicious Login Activity

My system:
Ubuntu 16.04, Elastic Stack 5.6.7 (Elasticsearch, Kibana and Filebeat) with X-Pack and Ingest-Geoip plugins installed. All of my services are running.

I was provided a test dataset auth.log and saved it in a folder:
/opt/data

My filebeat.yml configuration:

filebeat-yml%20config

After using the following command to start Filebeat

sudo /etc/init.d/filebeat start

I go to check if the data has been indexed into Elasticsearch using the following command

curl localhost:9200/filebeat-/_refresh -u elastic:changeme
curl localhost:9200/filebeat-
/doc/_count -u elastic:changeme

However, I get a response that non have indexed.

{"count":0,"_shards":{"total":0,"successful":0,"skipped":0,"failed":0}

I ran the command in the following directory /user/share/filebeat/bin

sudo filebeat.sh -configtest -e

I am not able to figure out the issue.

Hello, lets try a few thing to get a bit debug information.

Make sure you stop the original filebeat

sudo /etc/init.d/filebeat stop

Can you start filebeat with the following command and add the output here?

sudo filebeat.sh -e -d "*"

I am interested to know if there is any error.

This is the output from running the above command. How do I know if it is reading the given path to read the test dataset?
/opt/data

Hello @t.d.garland100,

I've installed the last 5.6.8 release and tested with the provided configuration from our repository and you should see something like this:

2018/02/20 20:35:41.989341 prospector.go:83: DBG  File Configs: [/opt/demos/ml-recipes/recipes/security/suspicious_login_activity/data/auth.log]
2018/02/20 20:35:41.989357 prospector_log.go:44: DBG  exclude_files: [(?-s:.)gz(?-m:$)]
2018/02/20 20:35:41.989366 prospector_log.go:65: INFO Prospector with previous states loaded: 0
2018/02/20 20:35:41.989454 prospector.go:124: INFO Starting prospector of type: log; id: 13996711275384129127
2018/02/20 20:35:41.989466 crawler.go:58: INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2018/02/20 20:35:41.989485 prospector_log.go:70: DBG  Start next scan
2018/02/20 20:35:41.989532 prospector_log.go:91: DBG  Prospector states cleaned up. Before: 0, After: 0
2018/02/20 20:35:46.989916 spooler.go:89: DBG  Flushing spooler because of timeout. Events flushed: 0
2018/02/20 20:35:51.990794 spooler.go:89: DBG  Flushing spooler because of timeout. Events flushed: 0

So I think it could be one of the following:

  1. By default Filebeat will read from the /etc/filebeat/filebeat.yml, is that the one you have modified?

  2. The YAML syntax in the configuration is sensitive to spaces, and It might be a problem if something is out of order, sadly the -configtest flag will not help you in that case, but we have the ideas to make it better.

The fact that I don't see any see any logs statement related from the prospector make me that either 1 or 2 could be the source of the problem. Since I just have a screenshot of the configuration, I can't verify my hypotheses.

So I downloaded the repository and copied the data to /opt/data. I also copied the configuration file filebeat.yml to the following /etc/filebeat/. I modified the this file to contain the above path for the test dataset. I reran the sudo filebeat.sh -e -d "*" command. This was my response:

tabitha@CaseV-611-ESML:~$ sudo filebeat.sh -e -d "*"
2018/02/21 16:10:35.243665 beat.go:297: INFO Home path:    [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018/02/21 16:10:35.243714 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.7
2018/02/21 16:10:35.243785 metrics.go:23: INFO Metrics logging every 30s
2018/02/21 16:10:35.243900 output.go:258: INFO Loading template enabled. Reading template file: /etc/filebeat/filebeat.template.json
2018/02/21 16:10:35.245186 output.go:269: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /etc/filebeat/filebeat.template-es2x.json
2018/02/21 16:10:35.246569 output.go:281: INFO Loading template enabled for Elasticsearch 6.x. Reading template file: /etc/filebeat/filebeat.template-es6x.json
2018/02/21 16:10:35.247763 client.go:128: INFO Elasticsearch url: http://localhost:9200
2018/02/21 16:10:35.247804 outputs.go:108: INFO Activated elasticsearch as output plugin.
2018/02/21 16:10:35.247912 publish.go:300: INFO Publisher name: test
2018/02/21 16:10:35.248153 async.go:63: INFO Flush Interval set to: 1s
2018/02/21 16:10:35.248172 async.go:64: INFO Max Bulk Size set to: 50
2018/02/21 16:10:35.249217 filebeat.go:46: INFO Enabled modules/filesets: system (auth),  ()
2018/02/21 16:10:35.249850 beat.go:233: INFO filebeat start running.
2018/02/21 16:10:35.249967 client.go:128: INFO Elasticsearch url: http://localhost:9200
2018/02/21 16:10:35.251796 client.go:667: INFO Connected to Elasticsearch version 5.6.7
2018/02/21 16:10:35.253113 registrar.go:85: INFO Registry file set to: /var/lib/filebeat/registry
2018/02/21 16:10:35.253181 registrar.go:106: INFO Loading registrar data from /var/lib/filebeat/registry
2018/02/21 16:10:35.253591 registrar.go:123: INFO States Loaded from registrar: 5
2018/02/21 16:10:35.253633 crawler.go:38: INFO Loading Prospectors: 1
2018/02/21 16:10:35.253658 registrar.go:236: INFO Starting Registrar
2018/02/21 16:10:35.253696 sync.go:41: INFO Start sending events to output
2018/02/21 16:10:35.253763 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2018/02/21 16:10:35.253918 prospector_log.go:65: INFO Prospector with previous states loaded: 1
2018/02/21 16:10:35.254175 prospector.go:124: INFO Starting prospector of type: log; id: 17023599358903200319 
2018/02/21 16:10:35.254205 crawler.go:58: INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2018/02/21 16:10:35.254574 log.go:91: INFO Harvester started for file: /opt/data/auth.log
2018/02/21 16:10:35.309176 client.go:667: INFO Connected to Elasticsearch version 5.6.7
2018/02/21 16:10:35.309201 output.go:317: INFO Trying to load template for client: http://localhost:9200
2018/02/21 16:10:35.310270 output.go:341: INFO Template already exists and will not be overwritten.
2018/02/21 16:11:05.244449 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.es.call_count.PublishEvents=144 libbeat.es.publish.read_bytes=69755 libbeat.es.publish.write_bytes=3255130 libbeat.es.published_and_acked_events=7121 libbeat.publisher.published_events=7121 publish.events=7123 registrar.states.current=6 registrar.states.update=7123 registrar.writes=5
2018/02/21 16:11:35.244094 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/21 16:12:05.244091 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/21 16:12:35.244050 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/21 16:13:05.244089 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/21 16:13:35.244109 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/21 16:14:05.244117 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/21 16:14:35.244105 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/21 16:15:05.244105 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/21 16:15:35.244062 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/21 16:15:40.900903 log.go:116: INFO File is inactive: /opt/data/auth.log. Closing because close_inactive of 5m0s reached.
2018/02/21 16:16:05.244142 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.closed=1 filebeat.harvester.open_files=-1 filebeat.harvester.running=-1 publish.events=1 registrar.states.update=1 registrar.writes=1

This output was different, however when using the curl cmd, I am still not seeing the data indexed.

Here a copy of the filebeat.yml configuration file

filebeat.modules:
- module: system
  syslog:
    enabled: false
  auth:
    enabled: true
    var.paths: ["opt/data/auth.log"]
name: test
output.elasticsearch:
  hosts: ["localhost:9200"]
  #protocol: "https"
  username: "elastic"
  password: "changeme"
  index: "filebeat"
logging.level: debug
logging.selectors: ["publish"}

If you look at the look, I see that your configuration correctly it tries to read auth.log in /opt/data/auth, but filebeat close it because it cannot find new data written to the file.

I think the registry has some information about this file so I doesn't detect new content.

Question:

  • Does this file has content?
  • I think there is no more writes in that file?

Can you add the content of the following file:

cat /var/lib/filebeat/registry

The registry does have some content.

tabitha@CaseV-611-ESML:~$ sudo cat /var/lib/filebeat/registry
[sudo] password for tabitha: 
[{"source":"/home/tabitha/Data/auth.log","offset":797638,"FileStateOS": {"inode":4729038,"device":2049},"timestamp":"2018-02-15T10:02:24.10256429-05:00","ttl":- 2},{"source":"/var/log/data/testdata-auth.log","offset":797638,"FileStateOS":{"inode":17705737,"device":2049},"timestamp":"2018-02-16T11:51:33.488626541-05:00","ttl":-2},{"source":"/opt/data/auth.log","offset":797637,"FileStateOS":{"inode":18612227,"device":2049},"timestamp":"2018-02-21T14:03:38.875454424-05:00","ttl":-1},{"source":"/var/log/auth.log","offset":47979,"FileStateOS":{"inode":17706302,"device":2049},"timestamp":"2018-02-20T11:57:32.43111871-05:00","ttl":-2},{"source":"/var/log/auth.log.1","offset":72313,"FileStateOS":{"inode":17696223,"device":2049},"timestamp":"2018-02-20T11:31:17.394728752-05:00","ttl":-2},{"source":"/opt/data/auth.log","offset":797637,"FileStateOS":{"inode":18612228,"device":2049},"timestamp":"2018-02-21T14:03:38.875456235-05:00","ttl":-1}]

When I was trying to figure out why it was not working. I moved the file auth.log from another directory (/home/tabitha/Data/), renamed it (testdata-auth.log), and of course have tried it several times since then.

Question:

  • So how can I clear the registry to start over reading the file, auth.log?

To clear the registry you must stop Filebeat and remove the /var/lib/filebeat/registry file.

ok so with a clean registry, I started filebeat in the debug mode again and this is the output

tabitha@CASEV-611-ESML:~$ sudo filebeat.sh -e -d "*"
2018/02/27 15:28:09.269591 beat.go:297: INFO Home path: [/usr/share/filebeat]   Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018/02/27 15:28:09.269637 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.8
2018/02/27 15:28:09.269729 metrics.go:23: INFO Metrics logging every 30s
2018/02/27 15:28:09.269805 output.go:258: INFO Loading template enabled. Reading template file: /etc/filebeat/filebeat.template.json
2018/02/27 15:28:09.271066 output.go:269: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /etc/filebeat/filebeat.template-es2x.json
2018/02/27 15:28:09.272417 output.go:281: INFO Loading template enabled for Elasticsearch 6.x. Reading template file: /etc/filebeat/filebeat.template-es6x.json
2018/02/27 15:28:09.273635 client.go:128: INFO Elasticsearch url: http://localhost:9200
2018/02/27 15:28:09.273681 outputs.go:108: INFO Activated elasticsearch as output plugin.
2018/02/27 15:28:09.273792 publish.go:300: INFO Publisher name: test
2018/02/27 15:28:09.274032 async.go:63: INFO Flush Interval set to: 1s
2018/02/27 15:28:09.274047 async.go:64: INFO Max Bulk Size set to: 50
2018/02/27 15:28:09.274820 filebeat.go:46: INFO Enabled modules/filesets: system (auth),  ()
2018/02/27 15:28:09.275232 beat.go:233: INFO filebeat start running.
2018/02/27 15:28:09.275371 client.go:128: INFO Elasticsearch url: http://localhost:9200
2018/02/27 15:28:09.313324 client.go:667: INFO Connected to Elasticsearch version 5.6.8
2018/02/27 15:28:10.346044 modules.go:376: INFO Elasticsearch pipeline with ID 'filebeat-5.6.8-system-auth-pipeline' loaded
2018/02/27 15:28:10.346103 registrar.go:68: INFO No registry file found under: /var/lib/filebeat/registry. Creating a new registry file.
2018/02/27 15:28:10.346621 registrar.go:106: INFO Loading registrar data from /var/lib/filebeat/registry
2018/02/27 15:28:10.346662 registrar.go:123: INFO States Loaded from registrar: 0
2018/02/27 15:28:10.346728 crawler.go:38: INFO Loading Prospectors: 1
2018/02/27 15:28:10.346806 registrar.go:236: INFO Starting Registrar
2018/02/27 15:28:10.346818 sync.go:41: INFO Start sending events to output
2018/02/27 15:28:10.346904 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2018/02/27 15:28:10.346949 prospector_log.go:65: INFO Prospector with previous states loaded: 0
2018/02/27 15:28:10.347087 prospector.go:124: INFO Starting prospector of type: log; id: 17853336057650699177 
2018/02/27 15:28:10.347106 crawler.go:58: INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2018/02/27 15:28:10.347598 log.go:91: INFO Harvester started for file: /opt/data/auth.log
2018/02/27 15:28:10.453666 client.go:667: INFO Connected to Elasticsearch version 5.6.8
2018/02/27 15:28:10.453690 output.go:317: INFO Trying to load template for client: http://localhost:9200
2018/02/27 15:28:10.509959 client.go:592: INFO Elasticsearch template with name 'filebeat' loaded
2018/02/27 15:28:39.269988 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.es.call_count.PublishEvents=144 libbeat.es.publish.read_bytes=69620 libbeat.es.publish.write_bytes=3265344 libbeat.es.published_and_acked_events=7121 libbeat.publisher.published_events=7121 publish.events=7122 registrar.states.current=1 registrar.states.update=7122 registrar.writes=6
2018/02/27 15:29:09.270015 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/27 15:29:39.270077 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/27 15:30:09.269955 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/27 15:30:39.270048 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/27 15:31:09.269997 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/27 15:31:39.270057 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/27 15:32:09.270060 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/27 15:32:39.270077 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/27 15:33:09.269978 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/27 15:33:16.885860 log.go:116: INFO File is inactive: /opt/data/auth.log. Closing because close_inactive of 5m0s reached.
2018/02/27 15:33:39.270065 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.closed=1 filebeat.harvester.open_files=-1 filebeat.harvester.running=-1 publish.events=1 registrar.states.update=1 registrar.writes=1
2018/02/27 15:34:09.270021 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/27 15:34:39.270075 metrics.go:34: INFO No non-zero metrics in the last 30s

However, filebeat stopped again due to inactivity. I checked the contents of the registry file:

tabitha@CASEV-611-ESML:~$ sudo cat /var/lib/filebeat/registry
[sudo] password for tabitha: 
[{"source":"/opt/data/auth.log","offset":797637,"FileStateOS":{"inode":9830403,"device":64512},"timestamp":"2018-02-27T10:33:21.881100771-05:00","ttl":-1}]

I see that filebeat is opening the data file but I am not uderstanding why it then becomes inactive and does not ship the data to Elasticsearch.
In the web browser I put the following: localhost:9200/_cat/indices and the output is below. I shows the index filebeat was indexed however, Kibana is telling me there is not mapping.

The problem is that you set the index to filebeat and not use the default daily indices. So the pattern in Kibana is then also just filebeat and not filebeat-*.

I just had used the configuration from the repository filebeat.yml.
So this is my new configuration file.

filebeat.modules:
- module: system
  syslog:
    enabled: false
  auth:
    enabled: true
    var.paths: ["/opt/data/auth.log"]
name: test
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "elastic"
  password: "changeme"
logging.level: debug
logging.selectors: ["publish"]

After restarting filebeat in debug mode, my output is:

tabitha@CASEV-611-ESML:~$ sudo filebeat.sh -e -d "*"
2018/03/02 18:23:42.713010 beat.go:297: INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018/03/02 18:23:42.713054 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.8
2018/03/02 18:23:42.713244 metrics.go:23: INFO Metrics logging every 30s
2018/03/02 18:23:42.713254 output.go:258: INFO Loading template enabled. Reading template file: /etc/filebeat/filebeat.template.json
2018/03/02 18:23:42.714444 output.go:269: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /etc/filebeat/filebeat.template-es2x.json
2018/03/02 18:23:42.715678 output.go:281: INFO Loading template enabled for Elasticsearch 6.x. Reading template file: /etc/filebeat/filebeat.template-es6x.json
2018/03/02 18:23:42.716768 client.go:128: INFO Elasticsearch url: http://localhost:9200
2018/03/02 18:23:42.716801 outputs.go:108: INFO Activated elasticsearch as output plugin.
2018/03/02 18:23:42.716901 publish.go:300: INFO Publisher name: test
2018/03/02 18:23:42.717122 async.go:63: INFO Flush Interval set to: 1s
2018/03/02 18:23:42.717168 async.go:64: INFO Max Bulk Size set to: 50
2018/03/02 18:23:42.717942 filebeat.go:46: INFO Enabled modules/filesets: system (auth),  ()
2018/03/02 18:23:42.718377 beat.go:233: INFO filebeat start running.
2018/03/02 18:23:42.718506 client.go:128: INFO Elasticsearch url: http://localhost:9200
2018/03/02 18:23:42.720236 client.go:667: INFO Connected to Elasticsearch version 5.6.8
2018/03/02 18:23:42.721441 registrar.go:68: INFO No registry file found under: /var/lib/filebeat/registry. Creating a new registry file.
2018/03/02 18:23:42.722116 registrar.go:106: INFO Loading registrar data from /var/lib/filebeat/registry
2018/03/02 18:23:42.722156 registrar.go:123: INFO States Loaded from registrar: 0
2018/03/02 18:23:42.722189 crawler.go:38: INFO Loading Prospectors: 1
2018/03/02 18:23:42.722200 registrar.go:236: INFO Starting Registrar
2018/03/02 18:23:42.722261 sync.go:41: INFO Start sending events to output
2018/03/02 18:23:42.722345 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2018/03/02 18:23:42.722383 prospector_log.go:65: INFO Prospector with previous states loaded: 0
2018/03/02 18:23:42.722573 prospector.go:124: INFO Starting prospector of type: log; id: 17853336057650699177 
2018/03/02 18:23:42.722601 crawler.go:58: INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2018/03/02 18:23:42.723014 log.go:91: INFO Harvester started for file: /opt/data/auth.log
2018/03/02 18:23:42.790197 client.go:667: INFO Connected to Elasticsearch version 5.6.8
2018/03/02 18:23:42.790220 output.go:317: INFO Trying to load template for client: http://localhost:9200
2018/03/02 18:23:42.791546 output.go:341: INFO Template already exists and will not be overwritten.
2018/03/02 18:24:12.713659 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.es.call_count.PublishEvents=144 libbeat.es.publish.read_bytes=71423 libbeat.es.publish.write_bytes=3333461 libbeat.es.published_and_acked_events=7121 libbeat.publisher.published_events=7121 publish.events=7122 registrar.states.current=1 registrar.states.update=7122 registrar.writes=6
2018/03/02 18:24:42.713592 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/03/02 18:25:12.713600 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/03/02 18:25:42.713486 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/03/02 18:26:12.713605 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/03/02 18:26:42.713604 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/03/02 18:27:12.713503 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/03/02 18:27:42.713596 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/03/02 18:28:12.713577 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/03/02 18:28:42.713577 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/03/02 18:28:48.355345 log.go:116: INFO File is inactive: /opt/data/auth.log. Closing because close_inactive of 5m0s reached.
2018/03/02 18:29:12.713598 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.closed=1 filebeat.harvester.open_files=-1 filebeat.harvester.running=-1 publish.events=1 registrar.states.update=1 registrar.writes=1

As you can see, it becomes inactive again.
I ran the curl cmd to confirm the data had been indexed.
curl localhost:9200/filebeat-*/_refresh
curl localhost:9200/filebeat-*/doc/_count
and my output was
So I loaded the job ./reset_job.sh suspicious_login_activity and issued the cmd to start the ML curl -s -X POST localhost:9200/_xpack/ml/datafeeds/datafeed-suspicious_login_activity/_start, however I get an error: that the datafeed lookback retrieved no data.
In the /var/log/filebeat/filebeat only one event was sent.

When you see the close_inactive message it means the file was not updated or it's size did not change. Are there new log lines in /opt/data/auth.log? If you just restarted filebeat it remembers the last state. So if there are log lines in there which were already sent it will not read them again. If you want that, you need to remove the registry file.

If you think the file changed since you started filebeat, it would be interesting to post here the output of ls -li before you start it and when it shows close_inactive log message.

For the ML part I don't really know what the code does. Let's get the Filebeat side sorted out first.

Please do not post pictures of command line output but aways the command line output.

Rufin,
The auth.log file does not receive any updated log files. It is a data set that was provided for a use case. I have removed the registry file and then restarted filebeat. Can you please clarify your last statement, it contradicts itself.

So my updated configuration file is as seen below:
filebeat.yml

filebeat.modules:
- module: system
  syslog:
    enabled: false
  auth:
    enabled: true
    var.paths: ["/opt/data/auth.log"]
name: test
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "elastic"
  password: "changeme"
  index: "filebeat-%{+yyyy.MM.dd}"
dashboards.enabled: true
logging.level: debug
logging.selectors: ["publish"]

I removed the /var/lib/filebeat/registry and /opt/data/auth.log and recopied the auth.log file for a fresh start. I restarted filebeat with the new index configuration. I was able to check the index using
curl localhost:9200/filebeat-*/_refresh and
curl localhost:9200/filebeat-*/doc/_count
and it is showing the document is indexed

{"count":21363,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0}

So I am still having an issue of seeing this indexed in Kibana.

Good news we have data in. Lets try to sort out the Kibana issue.

Below is the cat command showing the list of indices.

Indices look good. Can you now also check which index patterns you already have in Kibana?

I did change my setup a little from the original posting. Since your March 12 posting and forgot to mention. I have updated to 5.6.8, and I am no longer using the Ingest-geoip plugin for elasticsearch. I have installed Logstash. My configuration files are below.

filebeat.yml File

###################### Filebeat Configuration #########################
#=========================== Filebeat prospectors =============================

filebeat.prospectors:
- input_type: log
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    #- /var/log/auth.log
    #- /var/log/syslog
    - /opt/data/auth.log
    #- c:\programdata\elasticsearch\logs\*

#================================ Outputs =====================================
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "elastic"
  password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

#================================ Logging =====================================
# Sets log level. 
logging.level: debug
# At debug level, you can selectively enable logging only for   some components.
logging.selectors: ["publish"]

10-syslog-filter.conf File

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" =>   "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
  add_field => [ "received_at", "%{@timestamp}" ]
  add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
  match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

02-beats-input.conf File

input {
  beats {
    port => 5044
  }
}

30-elasticsearch-output.conf

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    user => elastic
    password => changeme
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

So this is the first picture showing the index was created for filebeat-*

This is now my discover tab

However, this is not allowing me to use the ML tools from x-pack. After creating the job for the ML, I got this error message

Datafeed lookback retrieved no data

And the datafeed preview showed this.

I realized that the above filebeat.yml configuration file was sending the log data to elasticsearch and not to logstash. I have fixed this and the configuration is below.

###################### Filebeat Configuration #########################
#=========================== Filebeat prospectors =============================

filebeat.prospectors:
- input_type: log
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    #- /var/log/auth.log
    #- /var/log/syslog
    - /opt/data/auth.log
    #- c:\programdata\elasticsearch\logs\*

#================================ Outputs =====================================
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]
  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:./reset_job.sh suspicious_login_activity
  # The Logstash hosts
  hosts: ["localhost:5044"]

#================================ Logging =====================================
# Sets log level. 
logging.level: debug
# At debug level, you can selectively enable logging only for   some components.
logging.selectors: ["publish"]

I updated the beats input plugin for logstash. I start Filebeat in debug mode and it ran fine. However, when I go to the Management Tab in Kibana, I am able to index the data now.

However, I am still getting the same error after creating the job for the ML.

So I just want to make sure that I have the data in there correctly before moving to another forum concerning the ML job

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.