# Multiple inputs to multiple indices

**URL:** <https://discuss.elastic.co/t/multiple-inputs-to-multiple-indices/290171>\
**Category:** Logstash\
**Created:** [November 25, 2021, 12:21pm UTC](https://discuss.elastic.co/t/multiple-inputs-to-multiple-indices/290171 "2021-11-25T12:21:47Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Katgust](https://avatars.discourse-cdn.com/v4/letter/k/c4cdca/32.png) [@Katgust](https://discuss.elastic.co/u/Katgust)\
**Post date:** [November 25, 2021, 12:21pm UTC](https://discuss.elastic.co/t/multiple-inputs-to-multiple-indices/290171/1 "2021-11-25T12:21:47Z")

</div>

Hi!

I'm new to the Elastic stack but have tried to learn to use it the last couple of weeks. I have Elasticsearch and Kibana running (I can at least use the DevTools) and I'm using Logstash to retrieve data from an MSSQL database. Right now I'm using some sample data (the goal is to be able to implement this with a similar database at work). My config file is now as follows:

```auto
input {
  jdbc {
    jdbc_driver_library => "C:\Program Files\Microsoft JDBC Driver 7.4 for SQL Server\sqljdbc_7.4\enu\mssql-jdbc-7.4.1.jre8.jar"
    jdbc_driver_class => "Java::com.microsoft.sqlserver.jdbc.SQLServerDriver"
    jdbc_connection_string => "jdbc:sqlserver://localhost:1433;databaseName=AdventureWorks2019"
    jdbc_user => "test_user"
    jdbc_password => "pw1"
    statement => " SELECT * FROM Person.Person where modifieddate > :sql_last_value "
    schedule => "* * * * *"
  }
} 

output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "person_person" 
        action => 'update'
        document_id => "%{businessentityid}"
        user => "elastic"
        password => "OI4OZjzp7JjPuv4yIGwE"
    } 
}

```

With this configuration I get out the table Person.Person (I can access it in DevTools) and every minute all rows that have been updated in MSSQL since the last check (aka where the modifieddate has been changed) gets updated in Elasticsearch. So far so good.

The problem for me now is that I have at least 20 different tables that I want to index. Maybe an index per table isn't the best solution but it's what makes the most sense to me right now. Is there a way to modify my config file to update all these indices at the same time or do I have to have multiple config files "running" (both to retrieve the data the first time and to update it)?

I've found this solution [multiple-inputs-on-logstash-jdbc](https://stackoverflow.com/questions/37613611/multiple-inputs-on-logstash-jdbc) so I get that I can have many inputs but I would like to output to different indices too.

Also I'm not doing any mapping, which I guess happens automatically, but I've understood that's a smart thing to do. Is that something I can do in this config file or do I need to set that up in Elasticsearch before I import the data?

I'm very happy for any guidance I could get, thank you!

---

<div class="post-metadata">

**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\
**Post date:** [December 3, 2021, 5:10pm UTC](https://discuss.elastic.co/t/multiple-inputs-to-multiple-indices/290171/2 "2021-12-03T17:10:44Z")

</div>

You can use an if statement in your output to define where you want stuff to go OR the smarter thing would be to add an index name field to each of your inputs in the `@metadata` field.

```auto
input {
  jdbc {
    jdbc_driver_library => "C:\Program Files\Microsoft JDBC Driver 7.4 for SQL Server\sqljdbc_7.4\enu\mssql-jdbc-7.4.1.jre8.jar"
    jdbc_driver_class => "Java::com.microsoft.sqlserver.jdbc.SQLServerDriver"
    jdbc_connection_string => "jdbc:sqlserver://localhost:1433;databaseName=AdventureWorks2019"
    jdbc_user => "test_user"
    jdbc_password => "pw1"
    statement => " SELECT * FROM Person.Person where modifieddate > :sql_last_value "
    schedule => "* * * * *"
    add_field => { "[@metadata][target_index]" => "person_person" }
  }
  jdbc {
    jdbc_driver_library => "C:\Program Files\Microsoft JDBC Driver 7.4 for SQL Server\sqljdbc_7.4\enu\mssql-jdbc-7.4.1.jre8.jar"
    jdbc_driver_class => "Java::com.microsoft.sqlserver.jdbc.SQLServerDriver"
    jdbc_connection_string => "jdbc:sqlserver://localhost:1433;databaseName=AdventureWorks2019"
    jdbc_user => "test_user"
    jdbc_password => "pw1"
    statement => " SELECT * FROM Address.Address where modifieddate > :sql_last_value "
    schedule => "* * * * *"
    add_field => { "[@metadata][target_index]" => "address_address" }
  }
} 

output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "%{[@metadata][target_index]}" 
        action => 'update'
        document_id => "%{businessentityid}"
        user => "elastic"
        password => "OI4OZjzp7JjPuv4yIGwE"
    } 
}

```

You can read more here: [Elasticsearch output plugin | Logstash Reference [7.15] | Elastic](https://www.elastic.co/guide/en/logstash/7.15/plugins-outputs-elasticsearch.html#_writing_to_different_indices_best_practices)

---

<div class="post-metadata">

**Author:** ![Katgust](https://avatars.discourse-cdn.com/v4/letter/k/c4cdca/32.png) [@Katgust](https://discuss.elastic.co/u/Katgust)\
**Post date:** [December 8, 2021, 8:39am UTC](https://discuss.elastic.co/t/multiple-inputs-to-multiple-indices/290171/3 "2021-12-08T08:39:49Z")

</div>

Nice! Thank you!

I would like to be able to do something similar where the document\_id corresponds to the primary key of the imported table (which is known). Do you know how (or where I could read about how) to add something like

```auto
document_id => "%{[@metadata][target_id]}"

```

Everything I've tried at [target\_id] has given Invalid FieldReference error and I cant seem to find the right documentation

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [December 8, 2021, 10:37am UTC](https://discuss.elastic.co/t/multiple-inputs-to-multiple-indices/290171/4 "2021-12-08T10:37:17Z")

</div>

Select will return the "message" field. You have to extract id from there by Grok or similar plugin. Use the filter section for data transformation. Also use ruby debugger mode to see processed data. You can use a field or add tags in jdbc to make difference between data for the output section.

```auto
output {
if ([field-name] == "something") {
  elasticsearch
   {
      hosts => ["http://elkhost:9200"]
      index => "indexname1"
      document_id => "%{[@metadata][id_from_select]}"
   }
 }  
 else if ([field-name] == "something") {
   elasticsearch {
      hosts => ["http://elkhost:9200"]
      index => "indexname2"
      document_id => "%{[@metadata][id_from_select]}"
   }
 stdout {codec => rubydebug}
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 5, 2022, 10:38am UTC](https://discuss.elastic.co/t/multiple-inputs-to-multiple-indices/290171/5 "2022-01-05T10:38:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.