# Links status based on sysloghost and message

**URL:** <https://discuss.elastic.co/t/links-status-based-on-sysloghost-and-message/215544>\
**Category:** Kibana\
**Created:** [January 18, 2020, 5:19am UTC](https://discuss.elastic.co/t/links-status-based-on-sysloghost-and-message/215544 "2020-01-18T05:19:52Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![mushfiq](https://avatars.discourse-cdn.com/v4/letter/m/ce7236/32.png) [@mushfiq](https://discuss.elastic.co/u/mushfiq)\
**Post date:** [January 18, 2020, 5:19am UTC](https://discuss.elastic.co/t/links-status-based-on-sysloghost-and-message/215544/1 "2020-01-18T05:19:52Z")

</div>

Hi,

I am new ELK stack & started testing it from last week. I must say, **I just love it**.

**What I Have**  
Basic SOHO dual WAN routers, first link leased line and second link broadband (no static IP)  
These routers support syslog.  
Example logs that appears on Kibana are below  
**NSD FAIL WAN[2]** (WAN1 is Leased line and WAN2 is Broadband)  
**NSD SUCCESS WAN[2]**

**What I did**  
Installed and configured ELK stack and Rsyslog on single Ubuntu server.  
Redirected routers syslogs to this server.  
Configured filter to drop unwanted logs.

**What I want**  
Kibana dashboard where I can monitor both the links, based on "sysloghost" unique entries and it's log message (logs example above)

I tried to search but no luck, need help to accomplish this.

---

<div class="post-metadata">

**Author:** ![mushfiq](https://avatars.discourse-cdn.com/v4/letter/m/ce7236/32.png) [@mushfiq](https://discuss.elastic.co/u/mushfiq)\
**Post date:** [January 20, 2020, 9:05am UTC](https://discuss.elastic.co/t/links-status-based-on-sysloghost-and-message/215544/2 "2020-01-20T09:05:56Z")

</div>

Elaborating

Is it possible to visualize based on last log message? Like, two types of logs appears on Kibana from routers as below.

 ![Screenshot from 2020-01-20 11-19-38](https://us1.discourse-cdn.com/elastic/original/3X/5/7/576eaed12f05e63c90f13e19b5f75281153d9efa.png)

Log: NSD SUCCESS WAN[1]  
Log: NSD FAIL WAN[1]

So, when the last log from a router is "NSD SUCCESS WAN[1]" then it should appear in "up" list and if the last log is "NSD FAIL WAN[1]" then it should appear in "down" list

Guys I can provide more details if required.

---

<div class="post-metadata">

**Author:** ![azasypkin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/azasypkin/32/42216_2.png) [@azasypkin](https://discuss.elastic.co/u/azasypkin)\
**Post date:** [January 20, 2020, 10:36am UTC](https://discuss.elastic.co/t/links-status-based-on-sysloghost-and-message/215544/3 "2020-01-20T10:36:51Z")

</div>

Hi @mushfiq,

I think in general you need to consider [entity-centric index](https://www.elastic.co/videos/entity-centric-indexing-mark-harwood) approach. But if you just want to get the last record for every unique status message you could:

- term split on the router so that you get one bucket for every unique router
- top hits on timestamp field (you get the last N entries for each bucket and you can limit it just to one).

Something like this maybe (assuming I understood your use case correctly), but experiment and see what works for you and what doesn't:

 ![Peek 2020-01-20 11-35](https://us1.discourse-cdn.com/elastic/original/3X/1/d/1dc7050e9ba1f1c0c747e468ba63ed1f9d86731e.gif)

Best,  
Oleg

---

<div class="post-metadata">

**Author:** ![mushfiq](https://avatars.discourse-cdn.com/v4/letter/m/ce7236/32.png) [@mushfiq](https://discuss.elastic.co/u/mushfiq)\
**Post date:** [January 21, 2020, 10:29am UTC](https://discuss.elastic.co/t/links-status-based-on-sysloghost-and-message/215544/4 "2020-01-21T10:29:31Z")

</div>

Hi @azasypkin

Thanks for reply 🙂

There is one issue, while creating visualization it does not show the "message", not aggregatable.

![Screenshot from 2020-01-20 16-32-53](https://us1.discourse-cdn.com/elastic/original/3X/d/8/d878bc4ec203db6228d6bc3e94c4e2de6aa7ed0a.png)

 ![Screenshot from 2020-01-20 16-31-37](https://us1.discourse-cdn.com/elastic/original/3X/e/1/e1f1aff20a1ad6614436ac499f9dc894f5feb37c.png)

Also when search "NSD FAIL WAN[2]" on discover it does not show the result, I believe "[" "]" are causing it.

---

<div class="post-metadata">

**Author:** ![azasypkin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/azasypkin/32/42216_2.png) [@azasypkin](https://discuss.elastic.co/u/azasypkin)\
**Post date:** [January 21, 2020, 11:03am UTC](https://discuss.elastic.co/t/links-status-based-on-sysloghost-and-message/215544/5 "2020-01-21T11:03:34Z")

</div>

Your index mapping must be wrong, can you share it: [https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html)

---

<div class="post-metadata">

**Author:** ![mushfiq](https://avatars.discourse-cdn.com/v4/letter/m/ce7236/32.png) [@mushfiq](https://discuss.elastic.co/u/mushfiq)\
**Post date:** [January 22, 2020, 7:43am UTC](https://discuss.elastic.co/t/links-status-based-on-sysloghost-and-message/215544/6 "2020-01-22T07:43:51Z")

</div>

Hi @azasypkin

Sorry for bothering you, as I am not developer, I am having little difficulty rectifying mapping.

Output of " curl -XGET [http://localhost:9200/logstash-2020.01.18?pretty](http://localhost:9200/logstash-2020.01.18?pretty) "

{  
"logstash-2020.01.18" : {  
"aliases" : { },  
"mappings" : {  
"doc" : {  
"dynamic\_templates" : [  
{  
"message\_field" : {  
"path\_match" : "message",  
"match\_mapping\_type" : "string",  
"mapping" : {  
"norms" : false,  
"type" : "text"  
}  
}  
},  
{  
"string\_fields" : {  
"match" : "_",  
"match\_mapping\_type" : "string",  
"mapping" : {  
"fields" : {  
"keyword" : {  
"ignore\_above" : 256,  
"type" : "keyword"  
}  
},  
"norms" : false,  
"type" : "text"  
}  
}  
}  
],  
"properties" : {  
"@timestamp" : {  
"type" : "date"  
},  
"@version" : {  
"type" : "keyword"  
},  
"facility" : {  
"type" : "text",  
"norms" : false,  
"fields" : {  
"keyword" : {  
"type" : "keyword",  
"ignore\_above" : 256  
}  
}  
},  
"geoip" : {  
"dynamic" : "true",  
"properties" : {  
"ip" : {  
"type" : "ip"  
},  
"latitude" : {  
"type" : "half\_float"  
},  
"location" : {  
"type" : "geo\_point"  
},  
"longitude" : {  
"type" : "half\_float"  
}  
}  
},  
"host" : {  
"type" : "text",  
"norms" : false,  
"fields" : {  
"keyword" : {  
"type" : "keyword",  
"ignore\_above" : 256  
}  
}  
},  
"message" : {  
"type" : "text",  
"norms" : false  
},  
"procid" : {  
"type" : "text",  
"norms" : false,  
"fields" : {  
"keyword" : {  
"type" : "keyword",  
"ignore\_above" : 256  
}  
}  
},  
"programname" : {  
"type" : "text",  
"norms" : false,  
"fields" : {  
"keyword" : {  
"type" : "keyword",  
"ignore\_above" : 256  
}  
}  
},  
"severity" : {  
"type" : "text",  
"norms" : false,  
"fields" : {  
"keyword" : {  
"type" : "keyword",  
"ignore\_above" : 256  
}  
}  
},  
"sysloghost" : {  
"type" : "text",  
"norms" : false,  
"fields" : {  
"keyword" : {  
"type" : "keyword",  
"ignore\_above" : 256  
}  
}  
},  
"tags" : {  
"type" : "text",  
"norms" : false,  
"fields" : {  
"keyword" : {  
"type" : "keyword",  
"ignore\_above" : 256  
}  
}  
},  
"type" : {  
"type" : "text",  
"norms" : false,  
"fields" : {  
"keyword" : {  
"type" : "keyword",  
"ignore\_above" : 256  
}  
}  
}  
}  
},  
"default" : {  
"dynamic\_templates" : [  
{  
"message\_field" : {  
"path\_match" : "message",  
"match\_mapping\_type" : "string",  
"mapping" : {  
"norms" : false,  
"type" : "text"  
}  
}  
},  
{  
"string\_fields" : {  
"match" : "_",  
"match\_mapping\_type" : "string",  
"mapping" : {  
"fields" : {  
"keyword" : {  
"ignore\_above" : 256,  
"type" : "keyword"  
}  
},  
"norms" : false,  
"type" : "text"  
}  
}  
}  
],  
"properties" : {  
"@timestamp" : {  
"type" : "date"  
},  
"@version" : {  
"type" : "keyword"  
},  
"geoip" : {  
"dynamic" : "true",  
"properties" : {  
"ip" : {  
"type" : "ip"  
},  
"latitude" : {  
"type" : "half\_float"  
},  
"location" : {  
"type" : "geo\_point"  
},  
"longitude" : {  
"type" : "half\_float"  
}  
}  
}  
}  
}  
},  
"settings" : {  
"index" : {  
"refresh\_interval" : "5s",  
"number\_of\_shards" : "5",  
"provided\_name" : "logstash-2020.01.18",  
"creation\_date" : "1579306701290",  
"number\_of\_replicas" : "1",  
"uuid" : "YN35vCHXR2CxIGONMygV0Q",  
"version" : {  
"created" : "6080699"  
}  
}  
}  
}  
}

Indices being created automatically

![Screenshot from 2020-01-22 13-02-32](https://us1.discourse-cdn.com/elastic/original/3X/2/f/2f996692603cdf139245f23b0c1fb203158501f3.png)

I have created index in Kibana as below.

![Screenshot from 2020-01-22 13-05-45](https://us1.discourse-cdn.com/elastic/original/3X/2/4/2497048a0e88173cb7d02ed83606272e4e058b77.png)

I am not able understand where should I make changes to get the mapping right.

---

<div class="post-metadata">

**Author:** ![mushfiq](https://avatars.discourse-cdn.com/v4/letter/m/ce7236/32.png) [@mushfiq](https://discuss.elastic.co/u/mushfiq)\
**Post date:** [January 22, 2020, 8:40am UTC](https://discuss.elastic.co/t/links-status-based-on-sysloghost-and-message/215544/7 "2020-01-22T08:40:58Z")

</div>

```
{
  "logstash-2020.01.18" : {
"aliases" : { },
"mappings" : {
  "doc" : {
    "dynamic_templates" : [
      {
        "message_field" : {
          "path_match" : "message",
          "match_mapping_type" : "string",
          "mapping" : {
            "norms" : false,
            "type" : "text"
          }
        }
      },
      {
        "string_fields" : {
          "match" : "*",
          "match_mapping_type" : "string",
          "mapping" : {
            "fields" : {
              "keyword" : {
                "ignore_above" : 256,
                "type" : "keyword"
              }
            },
            "norms" : false,
            "type" : "text"
          }
        }
      }
    ],
    "properties" : {
      "@timestamp" : {
        "type" : "date"
      },
      "@version" : {
        "type" : "keyword"
      },
      "facility" : {
        "type" : "text",
        "norms" : false,
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      },
      "geoip" : {
        "dynamic" : "true",
        "properties" : {
          "ip" : {
            "type" : "ip"
          },
          "latitude" : {
            "type" : "half_float"
          },
          "location" : {
            "type" : "geo_point"
          },
          "longitude" : {
            "type" : "half_float"
          }
        }
      },
      "host" : {
        "type" : "text",
        "norms" : false,
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      },
      "message" : {
        "type" : "text",
        "norms" : false
      },
      "procid" : {
        "type" : "text",
        "norms" : false,
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      },
      "programname" : {
        "type" : "text",
        "norms" : false,
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      },
      "severity" : {
        "type" : "text",
        "norms" : false,
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      },
      "sysloghost" : {
        "type" : "text",
        "norms" : false,
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      },
      "tags" : {
        "type" : "text",
        "norms" : false,
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      },
      "type" : {
        "type" : "text",
        "norms" : false,
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      }
    }
  },
  "_default_" : {
    "dynamic_templates" : [
      {
        "message_field" : {
          "path_match" : "message",
          "match_mapping_type" : "string",
          "mapping" : {
            "norms" : false,
            "type" : "text"
          }
        }
      },
      {
        "string_fields" : {
          "match" : "*",
          "match_mapping_type" : "string",
          "mapping" : {
            "fields" : {
              "keyword" : {
                "ignore_above" : 256,
                "type" : "keyword"
              }
            },
            "norms" : false,
            "type" : "text"
          }
        }
      }
    ],
    "properties" : {
      "@timestamp" : {
        "type" : "date"
      },
      "@version" : {
        "type" : "keyword"
      },
      "geoip" : {
        "dynamic" : "true",
        "properties" : {
          "ip" : {
            "type" : "ip"
          },
          "latitude" : {
            "type" : "half_float"
          },
          "location" : {
            "type" : "geo_point"
          },
          "longitude" : {
            "type" : "half_float"
          }
        }
      }
    }
  }
},
"settings" : {
  "index" : {
    "refresh_interval" : "5s",
    "number_of_shards" : "5",
    "provided_name" : "logstash-2020.01.18",
    "creation_date" : "1579306701290",
    "number_of_replicas" : "1",
    "uuid" : "YN35vCHXR2CxIGONMygV0Q",
    "version" : {
      "created" : "6080699"
    }
  }
}
  }
}

```

in better view.

---

<div class="post-metadata">

**Author:** ![mushfiq](https://avatars.discourse-cdn.com/v4/letter/m/ce7236/32.png) [@mushfiq](https://discuss.elastic.co/u/mushfiq)\
**Post date:** [January 22, 2020, 10:17am UTC](https://discuss.elastic.co/t/links-status-based-on-sysloghost-and-message/215544/8 "2020-01-22T10:17:26Z")

</div>

Hi @azasypkin

I believe below template is being used.

```
{
  "logstash" : {
    "order" : 0,
    "version" : 60001,
    "index_patterns" : [
      "logstash-*"
    ],
    "settings" : {
      "index" : {
        "refresh_interval" : "5s"
      }
    },
    "mappings" : {
      "_default_" : {
        "dynamic_templates" : [
          {
            "message_field" : {
              "path_match" : "message",
              "mapping" : {
                "norms" : false,
                "type" : "text"
              },
              "match_mapping_type" : "string"
            }
          },
          {
            "string_fields" : {
              "mapping" : {
                "norms" : false,
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "ignore_above" : 256,
                    "type" : "keyword"
                  }
                }
              },
              "match_mapping_type" : "string",
              "match" : "*"
            }
          }
        ],
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "geoip" : {
            "dynamic" : true,
            "properties" : {
              "ip" : {
                "type" : "ip"
              },
              "latitude" : {
                "type" : "half_float"
              },
              "location" : {
                "type" : "geo_point"
              },
              "longitude" : {
                "type" : "half_float"
              }
            }
          },
          "@version" : {
            "type" : "keyword"
          }
        }
      }
    },
    "aliases" : { }
  }
}
```

---

<div class="post-metadata">

**Author:** ![mushfiq](https://avatars.discourse-cdn.com/v4/letter/m/ce7236/32.png) [@mushfiq](https://discuss.elastic.co/u/mushfiq)\
**Post date:** [January 22, 2020, 10:44am UTC](https://discuss.elastic.co/t/links-status-based-on-sysloghost-and-message/215544/9 "2020-01-22T10:44:15Z")

</div>

Made bellow changes

```
    "message_field" : {
      "path_match" : "message",
      "mapping" : {
        "norms" : false,
        "type" : "text",
        "fields" : {
          "keyword" : {
            "ignore_above" : 256,
            "type" : "keyword"
          }
        }   
      },

```

got below out put

```
{
  "acknowledged" : true
}

```

Deleted old index and created new one but still "message is not aggregatable.

---

<div class="post-metadata">

**Author:** ![azasypkin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/azasypkin/32/42216_2.png) [@azasypkin](https://discuss.elastic.co/u/azasypkin)\
**Post date:** [January 23, 2020, 8:30am UTC](https://discuss.elastic.co/t/links-status-based-on-sysloghost-and-message/215544/10 "2020-01-23T08:30:43Z")

</div>

> [@mushfiq](#):
>
> Deleted old index and created new one but still "message is not aggregatable.

So you updated `message_field`, and should have `message_field.keyword`. Don't you see it in Kibana? If not, try to refresh index pattern mapping cache like described [here](https://discuss.elastic.co/t/kibana-doesnt-recognize-keyword-fields/194586/2)

Best,  
Oleg

---

<div class="post-metadata">

**Author:** ![mushfiq](https://avatars.discourse-cdn.com/v4/letter/m/ce7236/32.png) [@mushfiq](https://discuss.elastic.co/u/mushfiq)\
**Post date:** [January 23, 2020, 10:36am UTC](https://discuss.elastic.co/t/links-status-based-on-sysloghost-and-message/215544/11 "2020-01-23T10:36:12Z")

</div>

Thanks @azasypkin

It helped, yesterday I tried it but no luck, to day it worked.

Trying different visuals now.

Thanks once again 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 20, 2020, 10:36am UTC](https://discuss.elastic.co/t/links-status-based-on-sysloghost-and-message/215544/12 "2020-02-20T10:36:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.