XML filter, doc_as_upsert and pipeline.workers

Hello,

My colleague @Anabella_Cristaldi and me have an XML like the following structure

<animal>
    <h>feline</h>
                <h>meow</h>
                <p>
                               <property>
                                               <character>
                                               <breed>
                                               <age>
                               </property>
                               <cat name="gardfield">
                                               <c>good</c>
                                               <c>persian</c>
                                               <c>1year</c>
                               </cat>
                               <cat name="felix">
                                               <c>regular</c>
                                               <c>common</c>
                                               <c>3year</c>
                               </cat>
                               <cat name="misifu">
                                               <c>sweet</c>
                                               <c>common</c>
                                               <c>1year</c>
                               </cat>
                </p>
                <p>
                               <property>
                                               <height>
                               </property>
                               <cat name="gardfield">
                                               <c>3</c>
                               </cat>
                               <cat name="felix">
                                               <c>8</c>
                               </cat>
                               <cat name="misifu">
                                               <c>2</c>
                               </cat>
                </p>
</animal>

and we have an similar XML for dogs

We need to have in ES a document by animal with all the properties.
We managed to do it using multine codec in the input plugin, xml and split filter plugin and in the output plugin action => "update" and doc_as_upsert => true and fix id for every cat/dog.
Originally our pipeline.workers was set up to 3 but we have noticed that sometimes we had erratic results. In a cat document we had dog properties.
Then we set the pipeline.workers to 1 and we have the expected results.

How pipeline.workers works and why we should use only one?

Thanks a lot!

PS: we use cats and dogs for simplicity, but we have data from network equipments in XML format

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.