Logstash/JBDC for ES when using nested attributes?

Elastic Stack Logstash
1

Hi

We wish to index physical events as documents. Each event will have multiple attributes, such as artists who are attending, and dates.

We're migrating from Sphinx, and to deal with this, we produced a mysql query which produced a single row, with some columns containing the attributes, eg:

|id |name | artists |
|001 | event 1 | artist1 artist2 artist3 |
|002 | event 2 | artist1 artist4 artist9 |

Could someone point me in the correct direction for doing this with Logstash/ES? I can replicate the same layout and it will perform a search query, but I want to be able to use filters, etc to pull out all 'events with artist 1' for example so I presume I need to nest those artists within the event? Whats the best way to do this using Logstash+ JDBC?

Many thanks

2

I think the best option would be to do a 1:1 mapping of RDBMS row to ES documents, i.e. turn the first row above into this document:

{
  "id": 1,
  "name": "event 1",
  "artist": [
    "artist1",
    "artist2",
    "artist3"
  ]
}

To split the artists field produced by Logstash's jdbc input you can use the mutate filter's split option.

3

Ah that looks promising, thanks. Do the plugins needs installing seperately?

4

Some do, some are core plugins that are installed by default. The mutate filter is definitely a core plugin but I'm not sure about the jdbc input.