Events added from rsyslog have only _index, _type and _id

Hello, I created a template for ES 5.5: http://termbin.com/kxdx

I imported it successfully:

# curl -X PUT "localhost:9200/_template/project.foreman-2018-11-07" -H
'Content-Type: application/json'
-d@org.foreman.viaq-cdm.5.5.2.template.json
{"acknowledged":true}

Configured rsyslog as follows:

And it starts sending something, this is how JSON should look like:

The problem now is that these documents are kinda empty:

# curl -XGET localhost:9200/project.foreman-2018-11-07/events/_search
| json_reformat
{
    "took": 0,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": 52,
        "max_score": 1.0,
        "hits": [
            {
                "_index": "project.foreman-2018-11-07",
                "_type": "events",
                "_id": "AWSPPEOnN8zQOY_iyj6u",
                "_score": 1.0
            },
            {
                "_index": "project.foreman-2018-11-07",
                "_type": "events",
                "_id": "AWSPPEQhN8zQOY_iyj6v",
                "_score": 1.0
            },
            {
                "_index": "project.foreman-2018-11-07",
                "_type": "events",
                "_id": "AWSPPESuN8zQOY_iyj6x",
                "_score": 1.0
            },
            {
                "_index": "project.foreman-2018-11-07",
                "_type": "events",
                "_id": "AWSPPEWmN8zQOY_iyj61",
                "_score": 1.0
            },
            {
                "_index": "project.foreman-2018-11-07",
                "_type": "events",
                "_id": "AWSPPEYrN8zQOY_iyj62",
                "_score": 1.0
            },
            {
                "_index": "project.foreman-2018-11-07",
                "_type": "events",
                "_id": "AWSPPEbVN8zQOY_iyj66",
                "_score": 1.0
            },
            {
                "_index": "project.foreman-2018-11-07",
                "_type": "events",
                "_id": "AWSPPEfkN8zQOY_iyj7A",
                "_score": 1.0
            },
            {
                "_index": "project.foreman-2018-11-07",
                "_type": "events",
                "_id": "AWSPPEgXN8zQOY_iyj7B",
                "_score": 1.0
            },
            {
                "_index": "project.foreman-2018-11-07",
                "_type": "events",
                "_id": "AWSPPEwpN8zQOY_iyj7R",
                "_score": 1.0
            },
            {
                "_index": "project.foreman-2018-11-07",
                "_type": "events",
                "_id": "AWSPPEy8N8zQOY_iyj7T",
                "_score": 1.0
            }
        ]
    }
}

I don't see the fields at all. Apparently rsyslog does format right data:

Debug line with all properties:
FROMHOST: 'next', fromhost-ip: '127.0.0.1', HOSTNAME: 'next', PRI: 14,
syslogtag 'foreman:', programname: 'foreman', APP-NAME: 'foreman',
PROCID: '-', MSGID: '-',
TIMESTAMP: 'Jul 12 17:12:33', STRUCTURED-DATA: '-',
msg: '  Rendered common/_searchbar.html.erb (0.6ms)'
escaped msg: '  Rendered common/_searchbar.html.erb (0.6ms)'
inputname: imjournal rawmsg: '  Rendered common/_searchbar.html.erb (0.6ms)'
$!:{ "message": "  Rendered common\/_searchbar.html.erb (0.6ms)\n",
"level": "info", "hostname": "next", "@timestamp": "2018-07
-12T17:12:33.592494+01:00", "systemd": { "t": { "MACHINE_ID":
"ce37bff78932646b63dbb85e691a3808", "BOOT_ID": "88a49b5077b64ee19
4fbe6e16b23a641", "CAP_EFFECTIVE": "0", "CMDLINE": "Passenger RackApp:
\/usr\/share\/foreman
   ", "COMM": "ruby", "EXE":
"\/opt\/rh\/rh-ruby24\/root\/usr\/bin\/ruby", "GID": "991",
"HOSTNAME": "next.nat.lan", "PID": "10
341", "SELINUX_CONTEXT": "system_u:system_r:passenger_t:s0",
"SOURCE_REALTIME_TIMESTAMP": "1531411953592176", "SYSTEMD_CGROUP":
 "\/system.slice\/httpd.service", "SYSTEMD_SLICE": "system.slice",
"SYSTEMD_UNIT": "httpd.service", "TRANSPORT": "journal", "UI
D": "993" }, "u": { "SYSLOG_FACILITY": "176", "SYSLOG_IDENTIFIER":
"foreman" } }, "MESSAGE": "  Rendered common\/_searchbar.htm
l.erb (0.6ms)\n", "msg": "Rendered common\/_searchbar.html.erb
(0.6ms)", "foreman": { "logger": "app", "user_login": "admin", "
remote_ip": "192.168.199.1", "request":
"af1afd88-afdd-4dcb-9458-689d633e286c", "session":
"61feb0a7-479a-40cd-a4f2-50a36525873
4" } }
$.:{ }
$/:

Not sure why all fields are ignored, can you give me hint? I am not much experienced with ES.

Your template has:

      "_source": {
        "enabled": false
      },

So Elasticsearch isn't storing a copy of your JSON source. That means the search results have nothing to show you except the index/id/type.
That doesn't mean the fields aren't there to search on, but when you retrieve results ES won't be able to show you the original document.

Ooooh thank you so much, this was a copy-n-paste error I missed!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.