Bulk API ID

[Pierre2]

Hi

To explain the problem there is a schema :

I have 3 log sources. First, log files that come from the SBC are processed by logstash and index in elasticsearch. The document ID (_id) is set to the call ID.

Then, the second source of log is the SVI. In this log file I have the same call ID as in SBC log file. Log files are processed by logstash and the output is a file that contains a lot of bulk instruction like :

{ "update" : {"_id" : "SD86mha1-d2915bef605b544a7a6f5da290549d57-v300g00060", "_type" : "type", "_index" : call"}}
{ "doc": {"a" : true , "b" : "lala", "c" : "dfe", "id2" : "abcdefghij123"}}

The "SD86mha1-d2915bef605b544a7a6f5da290549d57-v300g00060" is the call ID that is already the _id of an elasticsearch document.

When I execute this request it works and my existing document win 4 fieds (a,b,c,id2)

This is what is in the green rectangle and it works.

Now I want to add what is in the red rectangle. That means that I want to add a third log source (SIP server). The problem is that I can't use the bulk API because the common field beetween the existing doccument and the SVI server log is the id2 that was previously added and not the id of the document (_id). If I sent this request

{ "update" : {"id2" : "abcdefghij123", "_type" : "type", "_index" : call"}}
{ "doc": {"d" : true , "e" : "hoho"}}

I obtain :

{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "Action/metadata line [1] contains an unknown parameter [id2]"
}
],
"type": "illegal_argument_exception",
"reason": "Action/metadata line [1] contains an unknown parameter [id2]"
},
"status": 400
}

How can I add field from the third log source ?

[pastechecker (Charlie)]

Are you using one logstash instance that you can use pipelines.yml to connect things?

[Pierre2]

For the moment I run Elasticsearch on my PC. I call the first Logstash filter to process log comming from SBC with Logstash -f MyFilter.conf When my logs are indexed into Elastic search I call the second filter .conf that process the log from SVI and create the text file that contain bulk request. Then I POST this request (and it works). To finish I call the third filter that process the log from SIP server and that create the second text document with bulk request. When I try to post then I obtain the error bellow because in my request I use id2 that is a simple text field and not _id that is the ID of Elasticsearch's documents :

{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "Action/metadata line [1] contains an unknown parameter [id2]"
}
],
"type": "illegal_argument_exception",
"reason": "Action/metadata line [1] contains an unknown parameter [id2]"
},
"status": 400
}

If I understand what you are asking, you would like to know if I use a unique Logstash filter that parse all 3 log type with 3 pipelines ? If it's your question, as you can guess the answer is no. Should I ? Why ?

[pastechecker (Charlie)]

I have asked how do you run the logstash because from the picture I saw that you rather run the logstash on demand and change the configuration (by invoking different config files with -f option). You do not have to do it like that, you can use (default /etc/logstash/pipelines.yml to add paths to different configuration files).

Example of pipelines.yml:

• pipeline.id: whatever_unique_name1
  path.config: "/etc/logstash/yourconfigfile1.conf"
  pipeline.workers: 4
• pipeline.id: whatever_unique_name2
  path.config: "/etc/logstash/yourconfigfile2.conf"
  pipeline.workers: 4
• pipeline.id: whatever_unique_name3
  path.config: "/etc/logstash/yourconfigfile3.conf"
  pipeline.workers: 4

1. Correct me if I understand it wrongly: You let elasticsearch to set the _id for you (or you manually force in your config to override it with the document_id => "%{YOUR_FIELD} )? You try to update a record without identifying the unique _id, you try to go after the field directly and that is why it fails as the _id is not specified.

2. If the 1st one is not the case, you can assign the id to be your elasticsearch id and reference it in your update call as above and the problem will be gone.