Import specific fields only - Elasticsearch bulk import


(Abhilash Sathe) #1

Is there a way where we can ignore certain JSON keys inside a JSON dump while doing a bulk import in elasticsearch.

I'm currently working with Node JS. As mentioned in the below code, data.json contains certain JSON keys that I would want elasticsearch to ignore while importing. These keys can be of any type, example an Object, a string, a boolean, etc.

var bulkIndex = function bulkIndex(index, type, data) {
    let bulkBody = [];

    data.forEach(item => {
        bulkBody.push({
            index: {
                _index: index,
                _type: type,
                _id: item.id
            }
        });

        bulkBody.push(item);
    });

    esClient.bulk({
            body: bulkBody
        })
        .then(response => {
            let errorCount = 0;
            response.items.forEach(item => {
                if (item.index && item.index.error) {
                    console.log(++errorCount, item.index.error);
                }
            });
            console.log(`Successfully indexed ${data.length - errorCount} out of ${data.length} items`);
        })
        .catch(console.err);
};
const test = function test() {
    const rawJSON = fs.readFileSync('data.json');
    const processedJSON = JSON.parse(rawJSON);
    console.log(`${processedJSON.length} items parsed from data file`);
    bulkIndex('myIndex', 'myType', processedJSON);
};
test();

(David Pilato) #2

Can't you do that in your JS code?


(Abhilash Sathe) #3

Yes, that's what I'm going to do now. The JSON is basically Shopify's storefront product data. I've decided to use GraphQL and get specific fields only from the corresponding API.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.