Filebeat index not created

dev9 · October 24, 2021, 12:51am

Hi Guys,
I spent hours trying to get filebeat to index a custom log file but I am not having any luck in seeing any of the data ingest.
I provided all of my setup details on stack overflow (formatting of yaml here is off for some reason) elasticsearch - elastic filebeat index not created - Stack Overflow

Please let me know if you need the same details here

Any idea what am I missing?

Thanks

stephenb · October 24, 2021, 2:25am

Hi @dev9 Welcome to the community!

BTW We are not all guys

First thing I would do is just comment out all the setup stuff Just to reduce the variables.

# setup.ilm.enabled: false
# setup.ilm.rollover_alias: "filebeat"
# setup.ilm.pattern: "{now/d}-000001" 
# setup.template.name: "myindex"
# setup.template.pattern: "myindex-*"

I suspect what may be happening... Is the file has been already been read once (which it's a good chance it has) filebeat Will only read the file once because he keeps track of that.

So I would clean out the registry under the data directory and try again.

This is a common thing people get tripped up on.

Also you can start filebeat and watch the logs and should be able to see any error messages.

You can post the logs here.

dev9 · October 24, 2021, 2:45am

Thanks, Stephen!
My apology, my intention was not to sound sexist :). I usually use that guys term instead of ladies and gentlemen
I reduced the configuration to the following and I am still having issues;
elastic yaml

xpack.security.enabled: true
cluster.name: my-elastic-cluster
discovery.type: single-node
node.name: es-node-1
network.host: localhost
http.port: 9200

Kibana's yaml

elasticsearch.username: "kibana_system"
server.port: 5601
server.host: "localhost"
elasticsearch.hosts: ["http://localhost:9200"]

filebeat yaml


filebeat.inputs:
- type: log
  enabled: true
  paths: /Users/me/Downloads/logs/mylog.log
  json.message_key: severity 
  json.keys_under_root: true
    #json.overwrite_keys: true
    #json.add_error_key: true
    #json.expand_keys: true

    #setup.template.overwrite: true
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s
# ======================= Elasticsearch template setting =======================
#setup.template.settings:
#index.number_of_shards: 1
setup.kibana:
output.elasticsearch:
  hosts: ["localhost:9200"]
  template.name: filebeat
  template.path: filebeat.template.json

I also followed the instructions on Set up minimal security for Elasticsearch | Elasticsearch Guide [7.15] | Elastic and used ./bin/Elasticsearch-setup-passwords interactive option. Now whenever I try to executed filebeat setup, I get

Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at http://localhost:9200: 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}]

The reason I changed the original configuration was this blog post Structured logging with Filebeat | Elastic Blog but unfortunately I am unable to proceed due the authentication error when I execute filebeat setup.

Thanks for the support.

stephenb · October 24, 2021, 2:56am

Since you set up the passwords..

If So you need to include the elastic username and password in that output.elasticsearch section.

(Technically you could have used the filebeat user but I'd start with the elastic user first)

Not sure why you're including all the template stuff.

The way I recommend doing this is getting all the connectivity working and then start adding your templates and etc afterwards.

Just a recommendation.

stephenb · October 24, 2021, 3:04am

BTW If you're interested on how to create secure stack on a single server I wrote a step-by-step walkthrough.

github.com

bvader/howtos/blob/master/basic-security-elasticsearch/README.md


## Single Node Secured Elasticsearch + Kibana with Elastic generated self signed certs (updated)

#### NOTE / DISCLAIMER: **This configuration should only be used for Dev / POC purposes this is NOT suitable for production use.**

For Further Details Please Refer to the Official Documentation: [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) and [Kibana](https://www.elastic.co/guide/en/kibana/current/index.html)

### What we are doing?

This is simple / minimal quickstart to create a single Elasticsearch node and Kibana with basic authentication and SSL/TLS enabled (we will enable SSL for both HTTPS and Transport layer even though it is just a single node). This is a condensed / direct path to the Basic Security + HTTPs shown in the diagram [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-stack-security.html) and described [Configure security for the Elastic Stack](https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-stack-security.html#security-basic-https-overview). We will then be able to bind the Elasticsearch and Kibana to the network so it can be safely reached from another system.  
**Do NOT bind your Elasticsearch node or cluster to the network unless you secure your cluster and Kibana FIRST!** 

![Elastic Securty Layers](https://www.elastic.co/guide/en/elasticsearch/reference/current/images/elastic-security-overview.png)

**Notes:** 
* This example is using Elastic Stack 7.15.1 and Ubuntu 20.04 LTS using a Deb Package
* We are colocating Elasticsearch and Kibana for Dev / POC only for POC / Dev only
* Many of these commands / directores require `root` access so either be prepared to `sudo` most of the commands or just do a `sudo -i` for the duration of the session... your choice...
* There are many important settings for Elasticsearch that are not in the example please review them [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html) before moving on from POC / Dev mode
* All the IP Adresses are just examples and are not to be taken literally.

This file has been truncated. show original

dev9 · October 24, 2021, 3:30am

Thanks again Stephen.
I added the username and password as per your recommendation and removed these two lines from the filebeat.yaml

template.name: filebeat
  template.path: filebeat.template.json

executed filebeat setup and it ran Ok then started filebeat using nohup ./filebeat -e -c filebeat.yml & followed by tail -f nohup.out. I see the events being outputted but when I visit Kibana's discover, I still dont see any data. No errors in elastic or kibana, just no data.

stephenb · October 24, 2021, 3:50am

Did you clean out the filebeat registry?

Can you show those log lines?

Most people that want help post logs... You don't need to start in the background just start it in the foreground and capture the logs first 50 lines or so put in here

Also have you run

GET _cat/indices?v

In Kibana -> Dev Tools

dev9 · October 24, 2021, 4:07am

I am sorry, I have no idea how to clean the registry. I see folder called registry, should I delete it?
Logs

warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME
warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2021-10-23T20:39:06,551][INFO ][o.e.n.Node               ] [es-node-1] version[7.15.1], pid[53007], build[default/tar/83c34f456ae29d60e94d886e455e6a3409bba9ed/2021-10-07T21:56:19.031608185Z], OS[Mac OS X/11.5/x86_64], JVM[Eclipse Foundation/OpenJDK 64-Bit Server VM/11.0.12/11.0.12+7]
[2021-10-23T20:39:06,559][INFO ][o.e.n.Node               ] [es-node-1] JVM home [/Library/Java/JavaVirtualMachines/temurin-11.jre/Contents/Home], using bundled JDK [false]
[2021-10-23T20:39:06,559][INFO ][o.e.n.Node               ] [es-node-1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/var/folders/6f/r3ynyh013yx7wbs52xjyptnc0000gn/T/elasticsearch-4888122999279150445, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1024m, -Xmx1024m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/Users/sammy/Documents/elk/elastic, -Des.path.conf=/Users/sammy/Documents/elk/elastic/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]

executing GET _cat/indices?v yields the following

health status index                             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana_7.15.1_001                NJlrb97eT6aGuGZik2b7Eg   1   0       2308          505      3.1mb          3.1mb
green  open   .geoip_databases                  8JTvUiJFTXutDwV8KNpmhg   1   0         41           10     44.9mb         44.9mb
green  open   .security-7                       GMvs7im3RiiXcbwFdFG_gA   1   0         57            0    306.3kb        306.3kb
green  open   .apm-custom-link                  6TlRLLE-TG2tF0oWFd2dGQ   1   0          0            0       208b           208b
green  open   .kibana-event-log-7.15.1-000001   vQ-s7-PrS3yy08159iNLRw   1   0         20            0     50.8kb         50.8kb
green  open   .apm-agent-configuration          mL904RRxSE6faXIfDvlq_g   1   0          0            0       208b           208b
yellow open   filebeat-7.15.1-2021.10.23-000001 B0_RNKWySr2x5nyC0LQPfw   1   1          0            0       208b           208b
green  open   .async-search                     9quJYk7LRmuZjZlJT6s7yg   1   0          0          168       46kb           46kb
green  open   .tasks                            KTTMVQLaRICD0ngyay9r-Q   1   0         22            0     30.8kb         30.8kb
green  open   .kibana_task_manager_7.15.1_001   Z7hWEqQvSgutJ1LcXYDtjw   1   0         16         4986        1mb            1mb

I really appreciate your patience and support.

stephenb · October 24, 2021, 4:19am

The _cat looks good I see the filebeat bootstrap index.

Actually just delete that whole filebeat data directory

Logs : That is only 6 lines of logs...Not much I can do with that...

Clean out the data directory and start filebeat again... see if the data shows up ... if not.. I need about 50+ lines of the logs from the start command...

dev9 · October 24, 2021, 4:45am

Deleted the data directory, ran setup, still no data. the first 53 lines of logs were too much for this forum to take but I added them to google drive here logdata.log - Google Drive

stephenb · October 24, 2021, 5:15am

See all these these are mapping errors...and thus the docs are failing to index.
Not sure exactly what kind of logs these are but I have a bit of a theory but let's first see if we can get this running.

2021-10-24T00:27:14.324-0400    WARN    [elasticsearch] elasticsearch/client.go:412 Cannot index event 

....

{"type":"mapper_parsing_exception","reason":"object mapping for [source] tried to parse field [source] as object, but found a concrete value"}, dropping event!

Cleanup the registry...

Comment out

#json.message_key: severity 
#json.keys_under_root: true

In Dev Tools

DELETE filebeat*

Run setup again then start filebeat

I think you're JSON is trying to overwrite fields that have already been defined a specific types but your log is trying to write them as different types.

See if you can get it to ingest some and then I can perhaps help with the JSON part of it.

It looks it looks like you're trying to tell it to parse severity as a JSON object but it's just a simple field That's causing a parsing exception.

See The logs help :slight_smile

BTW https://pastebin.com/ works good for logs

dev9 · October 24, 2021, 5:41am

First of all, Thank you very much for your patience and support. You are the definition of a professional and I am grateful for all you do.

Now I see the logs but I have one more question for you. since these logs could contain error details, is it possible to query them using the json key values? for example, in this entry, the value of "environment" is "Env1". how do I query against these keys?

{"schemaVersion":1,"timestamp":"2021-10-12T13:30:33.648Z","severity":"info","details":"This is a test message, please ignore fe7a9860-26db-4474-a8f6-7a376e3251ce","clientIp":null,"serverName":"Server1","serverIp":"192.168.0.2","serviceComponent":null,"serviceProcessId":1708,"serviceRealm":null,"identity":null,"identityDelegate":null,"correlationId":null,"correlationRole":"PARTICIPANT","requestId":null,"requestRole":null,"messageId":null,"messageRole":null,"category":"TEST","class":"logger.test","action":null,"target":null,"result":null,"dataClassification":"CONFIDENTIAL","resultReason":null,"duration":0,"source":"logger","serverPort":2001,"serviceName":"Service1","serviceVersion":"1.2.1","environment":"Env1"}

Once again, thank you for your help

stephenb · October 24, 2021, 5:55am

You're welcome

Well, Perhaps you should read the docs about mappings, field data types and the query DSL.

In short you'll use a term query.

You should open a new thread on query once you get used to looking at your data.

You can see your data with this

GET filebeat*/_search

Your query would look something like this,

Since you're using a default map each field will create a keyword field and a text field which you will want to define your own when you go to production

GET /filebeat*/_search
{
  "query": {
    "term": {
      "environment.keyword": {
        "value": "Env1"
      }
    }
  }
}

Oh and when you're ready open a new thread on searching and mappings

dev9 · October 24, 2021, 5:59am

I will definitely read up on that topic, try a few things then open a new thread only if I need to.
Again, thank you very much for your support.

system · November 21, 2021, 6:00am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.