Combining log lines

Elastic Stack Elasticsearch
1

Hello,

I am asking as a newbie.

I have two kinds of log type. One type gives me ID's of items along with
other information. The other type gives me a particular id and its relevant
information such as its name and so on.

I want to perform a search and perform an update on the first type of log
such that the item's other information (name and so on) can be added to
that log line based on the matching Id.

As an example =>
First Type:
{

  • "_index": "logstash-2014.10.19",
  • "_type": "logs",
  • "_id": "VRh2iqsiRMmgS87BEIuduA",
  • "_version": 1,
  • "_score": 1,
  • "_source": {
    • "@timestamp": "2014-10-19T22:12:30.470Z",
    • "message": "18 Oct 2014 02:21:48,640 DEBUG [http-8080-1]
      (com.rtx.db.Where:45) -
      [2459BC896ECDFF92E85797F84E9DFCB4][page4][pwu2]EmptyPolicyApplicable =
      false ",
    • "@version": "1",
    • "host": "kilsedar-N55SL",
    • "path": [
      • "/home/kilsedar/Downloads/RTX.log",
      • "com.rtx.db.Where:45"
        ],
    • "tags": [
      • "_xmlparsefailure",
      • "LogData"
        ],
    • "day": "18",
    • "month": "Oct",
    • "year": "2014",
    • "time": "02:21:48,640",
    • "mode": "DEBUG",
    • "httpPort": "http-8080-1",
    • "sessionId": "2459BC896ECDFF92E85797F84E9DFCB4",
    • "pageId": "page4",
    • "Info": "EmptyPolicyApplicable = false ",
    • "pwuId": "pwu2"
      }

}
Second Type:
{

  • "_index": "logstash-2014.10.19",
  • "_type": "logs",
  • "_id": "LqDWHES9S1qS2WBYxq9fRA",
  • "_version": 1,
  • "_score": 1,
  • "_source": {
    • "@timestamp": "2014-10-19T22:12:20.322Z",
    • "message": "


      ",
    • "@version": "1",
    • "tags": [
      • "multiline",
      • "_xmlparsefailure",
      • "_grokparsefailure",
      • "WebModel"
        ],
    • "host": "kilsedar-N55SL",
    • "path":"/home/kilsedar/Downloads/WebModel/sv1/area10/page4.wr",
    • "powerIndexUnitId": [
      • "sv1#area10#page4#pwu2"
        ],
    • "powerIndexUnitName": [
      • "Product List"
        ],
    • "powerIndexUnitEntityId": [
      • "pkg1#ent8"
        ]
        }

}

So in this case I want to add powerIndexUnitName and powerIndexUnitEntityId
to the first Log as new fields.

Would it be possible and what kind of approach would you suggest?

Thank you!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1aaad945-37a4-4764-97b5-49835eb9869e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hello again,

I want to narrow down my question and ask in a more specific way.

I am using Update by Query plug-in to be able to update more than one
document at once.

I have written a query as follows:

{
"query": {
"term": {
"pwuId": "pwu8"
}
},
"script": "{ctx._source.newField = powerIndexUnitName_log}",
"params": {
"powerIndexUnitName_log": "..."
}
}

So, my aim is to add the "powerIndexUnitName_log" field to every document
including "pwuId" field as "pwu8".
The problem is to define the content this "powerIndexUnitName" variable.
With this query I am adding successfully the "newField" to the intended
documents.

My question is can I somehow query and assign a value to this
"powerIndexUnitName_log" variable.
The information is in a document, where powerIndexUnitId is pwu8.

Thank you!

On Monday, October 20, 2014 12:34:36 AM UTC+2, kilsedar wrote:

Hello,

I am asking as a newbie.

I have two kinds of log type. One type gives me ID's of items along with
other information. The other type gives me a particular id and its relevant
information such as its name and so on.

I want to perform a search and perform an update on the first type of log
such that the item's other information (name and so on) can be added to
that log line based on the matching Id.

As an example =>
First Type:
{

  • "_index": "logstash-2014.10.19",
  • "_type": "logs",
  • "_id": "VRh2iqsiRMmgS87BEIuduA",
  • "_version": 1,
  • "_score": 1,
  • "_source": {
    • "@timestamp": "2014-10-19T22:12:30.470Z",
    • "message": "18 Oct 2014 02:21:48,640 DEBUG [http-8080-1]
      (com.rtx.db.Where:45) -
      [2459BC896ECDFF92E85797F84E9DFCB4][page4][pwu2]EmptyPolicyApplicable =
      false ",
    • "@version": "1",
    • "host": "kilsedar-N55SL",
    • "path": [
      • "/home/kilsedar/Downloads/RTX.log",
      • "com.rtx.db.Where:45"
        ],
    • "tags": [
      • "_xmlparsefailure",
      • "LogData"
        ],
    • "day": "18",
    • "month": "Oct",
    • "year": "2014",
    • "time": "02:21:48,640",
    • "mode": "DEBUG",
    • "httpPort": "http-8080-1",
    • "sessionId": "2459BC896ECDFF92E85797F84E9DFCB4",
    • "pageId": "page4",
    • "Info": "EmptyPolicyApplicable = false ",
    • "pwuId": "pwu2"
      }

}
Second Type:
{

  • "_index": "logstash-2014.10.19",
  • "_type": "logs",
  • "_id": "LqDWHES9S1qS2WBYxq9fRA",
  • "_version": 1,
  • "_score": 1,
  • "_source": {
    • "@timestamp": "2014-10-19T22:12:20.322Z",
    • "message": "


      ",
    • "@version": "1",
    • "tags": [
      • "multiline",
      • "_xmlparsefailure",
      • "_grokparsefailure",
      • "WebModel"
        ],
    • "host": "kilsedar-N55SL",
    • "path":"/home/kilsedar/Downloads/WebModel/sv1/area10/page4.wr",
    • "powerIndexUnitId": [
      • "sv1#area10#page4#pwu2"
        ],
    • "powerIndexUnitName": [
      • "Product List"
        ],
    • "powerIndexUnitEntityId": [
      • "pkg1#ent8"
        ]
        }

}

So in this case I want to add powerIndexUnitName and
powerIndexUnitEntityId to the first Log as new fields.

Would it be possible and what kind of approach would you suggest?

Thank you!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6cdd09a0-04b9-4937-bfdc-000718d00cf5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.