# Field merge

**URL:** <https://discuss.elastic.co/t/field-merge/221165>\
**Category:** Logstash\
**Created:** [February 27, 2020, 7:56am UTC](https://discuss.elastic.co/t/field-merge/221165 "2020-02-27T07:56:14Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![111291](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111291/32/62724_2.png) [@111291](https://discuss.elastic.co/u/111291)\
**Post date:** [February 27, 2020, 7:56am UTC](https://discuss.elastic.co/t/field-merge/221165/1 "2020-02-27T07:56:15Z")

</div>

Hello,

Please tell me what is my mistake?

```auto
grok {
      match => { "message" => ["%{INT:Min}:%{BASE10NUM:Sec}-%{INT:Duration},(%{WORD:Event}|%{SPACE:Event}),%{INT:Level}"] }
      match => { "log.file.path" => ["%{INT:TempYYMMDDHH}.log"] }
}

mutate {
      add_field => { "MyTime" => "%{TempYYMMDDHH}%{Min}%{Sec}" }
      remove_field => ["TempYYMMDDHH", "Min", "Sec"]
}
date {
      match => ["MyTime", "yyMMddHHmmss.SSSSSS"]
      target => "@timestamp"
}
```

Trying to combine three fields into one value. At the output, I get the value of the "MyTime" field of the following form: %{TempYYMMDDHH}5955.073000

Input data:  
message =\> 59:55.073000-0,CONN,0  
log.file.path =\> C:\Users\Username\Documents\logs\rmngr\_3496\20021323.log

Best regards!

---

<div class="post-metadata">

**Author:** ![Jeeth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeeth/32/71724_2.png) [@Jeeth](https://discuss.elastic.co/u/Jeeth)\
**Post date:** [February 27, 2020, 8:47am UTC](https://discuss.elastic.co/t/field-merge/221165/2 "2020-02-27T08:47:52Z")

</div>

What is the error here ?

---

<div class="post-metadata">

**Author:** ![111291](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111291/32/62724_2.png) [@111291](https://discuss.elastic.co/u/111291)\
**Post date:** [February 27, 2020, 8:52am UTC](https://discuss.elastic.co/t/field-merge/221165/3 "2020-02-27T08:52:37Z")

</div>

I dont know)  
In field "MyTime" I get this value "%{TempYYMMDDHH}5955.073000"  
but i need to get this "200213235955.073000"  
and this in timestamp "Feb 13, 2020 @ 23:59:55.073"

---

<div class="post-metadata">

**Author:** ![ITIC](https://avatars.discourse-cdn.com/v4/letter/i/90ced4/32.png) [@ITIC](https://discuss.elastic.co/u/ITIC)\
**Post date:** [February 27, 2020, 12:19pm UTC](https://discuss.elastic.co/t/field-merge/221165/4 "2020-02-27T12:19:44Z")

</div>

Hi

Comment out the `remove_field` line, so you can see what those variables contain, so you can debug your config.

My money is on an empty or undefined `%{TempYYMMDDHH}`.

Hope this helps.

---

<div class="post-metadata">

**Author:** ![111291](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111291/32/62724_2.png) [@111291](https://discuss.elastic.co/u/111291)\
**Post date:** [February 27, 2020, 12:33pm UTC](https://discuss.elastic.co/t/field-merge/221165/5 "2020-02-27T12:33:32Z")

</div>

You're right  
`TempYYMMDDHH` is not in the index. What is the reason for this behavior?

`log.file.path` is not empty and in grok debugger there are no line parsing errors

---

<div class="post-metadata">

**Author:** ![ITIC](https://avatars.discourse-cdn.com/v4/letter/i/90ced4/32.png) [@ITIC](https://discuss.elastic.co/u/ITIC)\
**Post date:** [February 27, 2020, 1:04pm UTC](https://discuss.elastic.co/t/field-merge/221165/6 "2020-02-27T13:04:09Z")

</div>

Hi

Maybe `INT` is not the right parser for your file path. I'd try a more generic one, like `DATA`, or even `GREEDYDATA`, just to test it.

Adding a `\` in front, as there is one in your file path, might help too:

```auto
match => { "log.file.path" => ["\%{GREEDYDATA:TempYYMMDDHH}.log"] }

```

You might need to escape that "`\`".

Post the output of `stdout{}`.

Hope this helps

---

<div class="post-metadata">

**Author:** ![111291](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111291/32/62724_2.png) [@111291](https://discuss.elastic.co/u/111291)\
**Post date:** [February 27, 2020, 2:58pm UTC](https://discuss.elastic.co/t/field-merge/221165/7 "2020-02-27T14:58:19Z")

</div>

Unfortunately, but that didn't help either.

I did the same in Elasticsearch without logstash and it worked. But when transferring to logstash, `log.file.path` is not written to the field

Example from elasticsearch:

```auto
PUT _ingest/pipeline/onectj_pipeline 
{
  "description" : "onec tj pipeline",
  "processors": [
    {
      "set": {
        "field": "LogRowsID",
        "value": [
          "{{source}}", "{{_source.offset}}"
        ]
      }
    },
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{INT:_ingest.TempMM}:%{BASE10NUM:_ingest.TempSS}-%{INT:Duration},(%{WORD:Event}|%{SPACE:Event}),%{INT:Level}"
        ]
      }
    },
    {
      "grok": {
        "field": "log.file.path",
        "patterns": [
          "%{INT:_ingest.TempYYMMDDHH}.log"
        ]
      }
    },
    {
      "set": {
        "field": "_ingest.Tempdate",
        "value": "{{_ingest.TempYYMMDDHH}}{{_ingest.TempMM}}{{_ingest.TempSS}}"
      }
    },
    {
      "date": {
        "field": "_ingest.Tempdate",
        "target_field": "@timestamp",
        "formats": [
          "yyMMddHHmmss.SSSSSS"
        ],
        "timezone": "Europe/Moscow"
      }
    },
    {
      "set": {
        "field": "_id",
        "value": "{{_ingest.Tempdate}}-{{_source.offset}}"
      }
    }
  ]
}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 27, 2020, 3:22pm UTC](https://discuss.elastic.co/t/field-merge/221165/8 "2020-02-27T15:22:26Z")

</div>

> [@111291](#):
>
> `log.file.path` is not empty

Are you sure? I have more often seen [log][file][path] (a [log] object containing a [file] object containing a [path] field, rather than a [log.file.path] field which has periods in the object name).

---

<div class="post-metadata">

**Author:** ![ITIC](https://avatars.discourse-cdn.com/v4/letter/i/90ced4/32.png) [@ITIC](https://discuss.elastic.co/u/ITIC)\
**Post date:** [February 28, 2020, 7:18am UTC](https://discuss.elastic.co/t/field-merge/221165/9 "2020-02-28T07:18:33Z")

</div>

Hi

As @Badger says, make sure you have a field called `log.file.path` to begin with. As he suggested, the right way to call nested fields in logstash is `[a][b][c]`, not `a.b.c`.

I'd suggest using underscores, for instance, instead of dots in field names, as a general rule to avoid confusion, so you'll have a\_b\_c and never be confused again.

Hope this helps

---

<div class="post-metadata">

**Author:** ![111291](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111291/32/62724_2.png) [@111291](https://discuss.elastic.co/u/111291)\
**Post date:** [February 28, 2020, 7:24am UTC](https://discuss.elastic.co/t/field-merge/221165/10 "2020-02-28T07:24:24Z")

</div>

Thank you very much for you help! It solved my problem

Decision:

```auto
match => { "[log][file][path]" => ["%{INT:TempYYMMDDHH}.log"] }

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 27, 2020, 7:24am UTC](https://discuss.elastic.co/t/field-merge/221165/11 "2020-03-27T07:24:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.