# Append data to each line

**URL:** <https://discuss.elastic.co/t/append-data-to-each-line/119936>\
**Category:** Logstash\
**Created:** [February 15, 2018, 7:46am UTC](https://discuss.elastic.co/t/append-data-to-each-line/119936 "2018-02-15T07:46:48Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [February 15, 2018, 7:46am UTC](https://discuss.elastic.co/t/append-data-to-each-line/119936/1 "2018-02-15T07:46:48Z")

</div>

Ingesting XML data which appears like:

```
<statement><field1><field2></field2></field1></statement>
<record><field1><field2></field2></field1></record>
<record><field1><field2></field2></field1></record>
<record><field1><field2></field2></field1></record>

```

Is it possible to configure logstash to add `<statement><field1><field2></field2></field1></statement>` to the beginning or end of `record` line?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 15, 2018, 7:53am UTC](https://discuss.elastic.co/t/append-data-to-each-line/119936/2 "2018-02-15T07:53:12Z")

</div>

Sure, just use a mutate filter and its `replace` option.

```nohighlight
mutate {
  replace => {
    "record" => "<statement><field1><field2></field2></field1></statement>%{record}"
  }
}

```

---

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [February 15, 2018, 8:01am UTC](https://discuss.elastic.co/t/append-data-to-each-line/119936/3 "2018-02-15T08:01:58Z")

</div>

Not sure that would work. My current pipeline ingests the XML document and then uses the xml filter on the message field to parse out the different fields. Since `<statement><field1><field2></field2></field1></statement>` is on it's own line, the XML filter sees it as a single event. I'd gladly rearrange things if it's possible to make this work though.

```
input {
  file {
    id => "Ingest"
    path => "C:/DMARC/*.xml"
    codec => multiline {
      negate => true
      pattern => "<record>"
      what => "previous"
    }
  }
}
filter {
  xml {
    id => "Parse"
    force_array => true
    store_xml => false
    source => "message"
    xpath => [
      "feedback/report_metadata/org_name/text()", "report.org",
      "feedback/report_metadata/email/text()", "report.org_contact",
      "feedback/report_metadata/extra_contact_info/text()", "report.additional_contact",
      "feedback/report_metadata/report_id/text()", "report.id",
      "feedback/report_metadata/date_range/begin/text()", "report.start",
      "feedback/report_metadata/date_range/end/text()", "report.end",
      "feedback/policy_published/domain/text()", "policy.domain",
      "feedback/policy_published/aspf/text()", "policy.spf_mode",
      "feedback/policy_published/adkim/text()", "policy.dkim_mode",
      "feedback/policy_published/p/text()", "policy.dmarc.domain_action",
      "feedback/policy_published/sp/text()", "policy.dmarc.subdomain_action",
      "feedback/policy_published/pct/text()", "policy.percentage",
      "record/row/source_ip/text()", "email.source_ip",
      "record/row/count/text()", "email.count",
      "record/row/policy_evaluated/disposition/text()", "email.dmarc_action",
      "record/row/policy_evaluated/spf/text()", "email.spf_evaluation",
      "record/row/policy_evaluated/dkim/text()", "email.dkim_evaluation",
      "record/row/policy_evaluated/reason/type/text()", "dmarc.override_type",
      "record/row/policy_evaluated/reason/comment/text()", "dmarc.override_comment",
      "record/identifiers/envelope_to/text()", "email.envelope_to",
      "record/identifiers/envelope_from/text()", "email.envelope_from",
      "record/identifiers/header_from/text()", "email.header_from",
      "record/auth_results/dkim/domain/text()", "authresult.dkim_domain",
      "record/auth_results/dkim/result/text()", "authresult.dkim_result",
      "record/auth_results/spf/domain/text()", "authresult.spf_domain",
      "record/auth_results/spf/scope/text()", "authresult.spf_scope",
      "record/auth_results/spf/result/text()", "authresult.spf_result"
    ]
  }
 }
}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 15, 2018, 8:46am UTC](https://discuss.elastic.co/t/append-data-to-each-line/119936/4 "2018-02-15T08:46:45Z")

</div>

Then I don't understand the problem. What does your event currently look like? What would you like it to look like instead?

---

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [February 15, 2018, 1:45pm UTC](https://discuss.elastic.co/t/append-data-to-each-line/119936/5 "2018-02-15T13:45:04Z")

</div>

Each file contains a single policy statement with fields enclosed in the policy\_published XML tags. After the policy statement, there are entries for each IP the remote server communicated with, enclosed in the record XML tag. The policy\_published information identifies how/why actions where taken that are documented in each record.

What it looks like is the two images below, what I am looking for is a way to include the policy\_published fields with each record.

 ![policy](https://us1.discourse-cdn.com/elastic/original/3X/f/6/f6c62beda2135c3f8ef2b86a71fa638223a6d482.JPG)

 ![Record](https://us1.discourse-cdn.com/elastic/original/3X/3/5/3515df2f95caffb81eba2ecc457012b58c3bcd9f.JPG)

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 15, 2018, 1:51pm UTC](https://discuss.elastic.co/t/append-data-to-each-line/119936/6 "2018-02-15T13:51:33Z")

</div>

Oh, so you basically want to merge two events coming from different files. That's not something Logstash is very good at doing.

---

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [February 15, 2018, 1:59pm UTC](https://discuss.elastic.co/t/append-data-to-each-line/119936/7 "2018-02-15T13:59:17Z")

</div>

> [@magnusbaeck](#):
>
> Oh, so you basically want to merge two events coming from different files. That's not something Logstash is very good at doing.

No, I want to copy the policy\_published data for each file into each record event in the same file. The file looks like this:

```
<policy_published><field1><field2></field2></field2></policy_published>
<record><field1><field2></field2></field1></record>
<record><field1><field2></field2></field1></record>
<record><field1><field2></field2></field1></record>
<record><field1><field2></field2></field1></record>

```

I'm wanting this:

```
<policy_published><field1><field2></field2></field2></policy_published><record><field1><field2></field2></field1></record>
<policy_published><field1><field2></field2></field2></policy_published><record><field1><field2></field2></field1></record>
<policy_published><field1><field2></field2></field2></policy_published><record><field1><field2></field2></field1></record>
<policy_published><field1><field2></field2></field2></policy_published><record><field1><field2></field2></field1></record>

```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 15, 2018, 2:50pm UTC](https://discuss.elastic.co/t/append-data-to-each-line/119936/8 "2018-02-15T14:50:59Z")

</div>

You say "_into_ each record" but the example depicts the policy\_pubished elements alongside the record. Anyway, sounds like you'd want to use a ruby filter.

---

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [February 15, 2018, 3:07pm UTC](https://discuss.elastic.co/t/append-data-to-each-line/119936/9 "2018-02-15T15:07:51Z")

</div>

> [@magnusbaeck](#):
>
> You say "_into_ each record" but the example depicts the policy\_pubished elements alongside the record. Anyway, sounds like you'd want to use a ruby filter.

Maybe my example syntax is wrong...I'm just wanting the two example images posted earlier of the parsed events to end up being merged together. One event containing all the policy.\* fields for each record event.

---

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [February 16, 2018, 7:41am UTC](https://discuss.elastic.co/t/append-data-to-each-line/119936/10 "2018-02-16T07:41:56Z")

</div>

> [@magnusbaeck](#):
>
> You say "_into_ each record" but the example depicts the policy\_pubished elements alongside the record. Anyway, sounds like you'd want to use a ruby filter.

I'm no programmer, if you haven't gathered that already, but I'm guessing I would use Ruby to store the data I want copied into a variable and then call that variable using inline Ruby?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 16, 2018, 7:42am UTC](https://discuss.elastic.co/t/append-data-to-each-line/119936/11 "2018-03-16T07:42:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.