Dealing with no timestamp (Bro integration)

Elastic Stack Elasticsearch
1

So I'm currently testing using bro's bro ES integration plugin. My first challenge is the ts field, timestamp, isn't anything ES knows. Example:

How do I create a mapping to make ES see ts as a timestamp? Thank you.

2

The first example on the date mapping page should do it.

3

Thanks Nik that's helpful. A question I've always wanted to know...a lot of the examples show things like:

PUT my_index
{
  "mappings": {
    "my_type": {
      "properties": {
        "date": {
          "type": "date" 
        }
      }
    }
  }
}

but exactly HOW does one input that? Using a curl command? Is there no front-end that would allow making these changes? Thanks again.

4

Examples that look like that should have two links under them - one that says "view in Sense" or "view in Console" which pops open the snippet in sense (being renamed to console) which is a fancy Kibana/browser app with nice stuff like autocomplete. The other link says "copy as curl" and it ought turn the that sense syntax into valid bash/curl syntax and stick it on your clipboard.

We like the sense syntax because:

  1. It is pretty.
  2. It pushes folks to Sense which is a fairly friendly interface.
  3. It is reasonably easy to turn it into testable code, which we do in 5.0.
  4. The syntax highlighter does a decent job of highlighting it. So long as you don't have Privacy Badger on for the site. I haven't figured out why Privacy Badger is blocking the syntax highlighter. Something to do with the CDN....
5

Thanks again Nik. So OK I got Sense installed (neat tool). It appear that I can't modify an already existing index yes? How do I create a mapping and apply it to current and future indexes? Thank you.

6

For the most part you can't modify a field that has already been created. You can add new fields or new properties to existing fields.

You can use templates to control the mappings for new indexes. I never use them myself though. I prefer manually creating indexes with the mappings that I want for everything except testing.

You can modify some stuff about an existing index easily (number_of_replicas) or through special processes (number_of_shards through _shrink which is 5.0+). The rules are all about what is efficient to do for large indexes. If you don't like your index you can always create a new index and use the _reindex to copy all the docs to it.

7

Awesome thanks so much again Nik. Bro creates its indexes every 3 hours, so uh yea I think templates will be the way to go :slight_smile: I'll post my results once I'm done. Sense is pretty cool all in all.

8

I hope it doesn't plan on keeping them for very long then. There are practical limits on the number of indexes you can have in Elasticsearch cluster. You start to notice somethings (mapping changes, moving indexes from one node to another) start to take longer when you have too many. If used to be a couple thousand in 1.x, it is higher in 2.x but I'm not sure anyone is exactly sure how much higher.

9

Yea this is just testing and they are really small (like...187k). So here's my line:

PUT /_template/bro_template
{
  "template": "bro-*",
    "mappings": {
        "bro_ts": {
          "properties": {
            "ts": {
              "type": "date",
              "format": "epoch_millis"
            }
        }
      }  
  }
}

Fingers crossed that this works!

10

WOOT: