How to identify JDBC pipeline

[feijiangnan]

Hello,

I have pipelines of Filebeat which are identified by [fields][document_type] in logstash, and the indices are named with [fields][document_type] . Now, I am adding JDBC pipeline, how can I set [fields][document_type] in JDBC pipeline?

input {
jdbc {
jdbc_validate_connection => true
jdbc_connection_string => "jdbc:oracle:thin:@oradb:1521/orcl"
jdbc_user => "system"
jdbc_password => "Admin123"
jdbc_driver_library => "/opt/ojdbc7.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
statement => "SELECT * FROM V$ACTIVE_SESSION_HISTORY WHERE SAMPLE_TIME > :sql_last_value"
last_run_metadata_path => "/tmp/logstash-oradb.lastrun"
record_last_run => true
schedule => "*/2 * * * *"
}
}
filter {
# Set the timestamp to that of the ASH sample, not current time.
mutate { convert => [ "sample_time" , "string" ]}
date { match => ["sample_time", "ISO8601"]}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[fields][document_type]}-%{+YYYY.MM.dd}"
}
}

Thanks

Fei

[warkolm (Mark Walkom)]

Do you mean how can you extract [fields][document_type] from the JDBC input?

[feijiangnan]

Hello Mark,

I tried to use same variable/array [fields][document_type] to identify different pipeline so that I can create according Index name. For instance:
in filebeat.yml, it is defined as below

fields:
document_type: appserver-systemout-server1

in logstash configuration file, it is used as below

    elasticsearch {
        hosts => ["localhost:9200"]
        manage_template => false
        index => "%{[fields][document_type]}-%{+YYYY.MM.dd}"
    }

I would like to know if I can define something in jdbc pipeline similar to the entries in filebeat.

Thanks and Regards,
Fei

[Rory]

You can add a couple of columns in the your select query that have the values you want to index in the fields and document_type column.

OR

You can use mutate filter in logstash config to add the fields you want....coming from the jdbc input.

[guyboertje]

No need to add the mutate filter. The JDBC input supports add_field and other common config options.