# Logstash as a parser for data in column

**URL:** <https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352>\
**Category:** Logstash\
**Tags:** docker\
**Created:** [December 17, 2021, 6:04pm UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352 "2021-12-17T18:04:57Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [December 17, 2021, 6:04pm UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/1 "2021-12-17T18:04:57Z")

</div>

Hi  
I'm wonder about approach for this specific data constructed in column. How I can parser only through logstash. Some of fields a specially "ENUM" "HEADER", "BLOCK", "DATE" etc, are changing dynamically.

One files is divided to columns and rows as it was marked on picture.  
Do You have any idea how to parse it?

![Capture_EL_2](https://us1.discourse-cdn.com/elastic/original/3X/c/f/cf362e6cda285e5b50b01a053ae1438146d670cf.png)

---

<div class="post-metadata">

**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\
**Post date:** [December 18, 2021, 11:19pm UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/2 "2021-12-18T23:19:11Z")

</div>

Is this a CSV file?  
Is the file at least always the same format?

You could either define multiple grok patterns (if you know how the fields change) or use a CSV filter and define your delimiter and automatically detect the headers.

> **[Csv filter plugin | Logstash Reference \[7.16\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html)**

```auto
csv {
autodetect_column_names => true
autogenerate_column_names => false
}

```

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [December 20, 2021, 10:54am UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/3 "2021-12-20T10:54:59Z")

</div>

but can I use a CSV template when these values or columns are not separated by semicolons or commas

---

<div class="post-metadata">

**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\
**Post date:** [December 20, 2021, 11:58am UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/4 "2021-12-20T11:58:12Z")

</div>

As long as there is a consistent delimiter you can define what that delimiter is manually.

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [December 20, 2021, 12:57pm UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/5 "2021-12-20T12:57:36Z")

</div>

Can You share an examples?

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [December 20, 2021, 1:10pm UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/6 "2021-12-20T13:10:43Z")

</div>

an the second thing is: How to move a row from (beging "SERVID..") to the first line to be handled as one row through columns.

---

<div class="post-metadata">

**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\
**Post date:** [December 20, 2021, 2:33pm UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/7 "2021-12-20T14:33:55Z")

</div>

Check out the example in the documentation just define a `seperator`.

> **[Csv filter plugin | Logstash Reference \[7.16\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html#plugins-filters-csv-separator)**

---

<div class="post-metadata">

**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\
**Post date:** [December 20, 2021, 2:35pm UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/8 "2021-12-20T14:35:27Z")

</div>

> [@INS](#):
>
> an the second thing is: How to move a row from (beging "SERVID..") to the first line to be handled as one row through columns.

If your output is falling on multiple lines then you first need to use a multiline input to get everything into one document then you can apply the csv filter.

This is definitely not an easy task you have here, but with enough massaging it can be done.

Is there anything you can do to make the output from your app more machine readable?

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [December 20, 2021, 4:08pm UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/9 "2021-12-20T16:08:08Z")

</div>

@Badger You're good at this stuff, can You help with the concept?  
How to rebuild it so I can use `autodetect_column_names`?  
 ![InkedCapture_EL_2_LI_2](https://us1.discourse-cdn.com/elastic/original/3X/6/d/6dbbaa10fc66ad2bfe9114722b899f4439097d89.jpeg)

the row of data:

```auto
ENUM HEADER BLOCK DATE TIME
1184 LDAP FAILURE IN SEARCH RESULT SDABS 211214 134850800

SERVID USERID REASON
0 2 32

```

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [December 20, 2021, 8:29pm UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/10 "2021-12-20T20:29:34Z")

</div>

I've tried to use this method but it doesn't match yet  
In fact, it is non-uniform and there are several tabs between columns.

```auto
input {
 file {
	path => "/opt/data/input/test.txt*"
	start_position => "beginning"
	codec => multiline { "^\d" negate => false what => previous }
	sincedb_path => "/dev/null"
	}
}

filter {

if [message]=~ /^ENUM/ {
	mutate { add_field => { "[@metadata][format]" => "format1" }}
	} else {
		mutate { add_field => { "[@metadata][format]" => "format2" }}
		
}

if [@metadata][format] == "format1" {
    csv { separator => " " 
	autodetect_column_names => true
	autogenerate_column_names => false 
	}
if [ENUM] == "ENUM" { drop {} }
if [EVENT] == "EVENT" { drop {} }
if [MOID] == "MOID" { drop {} }
if [ATTRBID] == "ATTRBID" { drop {} }
	
	
} else {
	csv { separator => " " 
	autodetect_column_names => true
	autogenerate_column_names => false }
    if [SERVID] == "SERVID" { drop {} }
}

output {
stdout {
codec => rubydebug {}
}
}

```

---

<div class="post-metadata">

**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\
**Post date:** [December 21, 2021, 2:23pm UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/11 "2021-12-21T14:23:28Z")

</div>

Ok. I played around with it a bit and I need clarification about your test.txt file.

Does your sample data look like this?

```auto
ENUM HEADER BLOCK DATE TIME
1184 LDAP FAILURE IN SEARCH RESULT SDABS 211214 134850800

SERVID USERID REASON
0 2 32

ENUM HEADER BLOCK DATE TIME
1185 LDAP FAILURE IN SEARCH RESULT SDABS 2113414 1343043800

SERVID USERID REASON
0 2 32

ENUM HEADER BLOCK DATE TIME
1194 LDAP FAILURE IN SEARCH RESULT SDABS 2323214 13483230800

SERVID USERID REASON
0 2 32

```

or does it look like this?

```auto
ENUM HEADER BLOCK DATE TIME
1184 LDAP FAILURE IN SEARCH RESULT SDABS 211214 134850800

SERVID USERID REASON
0 2 32

1185 LDAP FAILURE IN SEARCH RESULT SDABS 2113414 1343043800

0 2 32

1194 LDAP FAILURE IN SEARCH RESULT SDABS 2323214 13483230800

0 2 32

```

Better yet, could you post multiple entries from your sample data?

Because the first one you may be able to massage into something you can use but the second one probably not (especially with column headers).

Is there any way you can get your data output to be better?

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [December 22, 2021, 3:30pm UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/12 "2021-12-22T15:30:28Z")

</div>

> [@AquaX](#):
>
> Better yet, could you post multiple entries from your sample data?
> 
> Because the first one you may be able to massage into something you can use but the second one probably not (especially with column headers).
> 
> Is there any way you can get your data output to be better?

This is the data in the form of printouts from the system, unfortunately they can not be presented otherwise, I post the entire file to have a broad perspective of how it looks like. Maybe someone has an idea how to convert it in logstash to a good form.

link for data input [https://easyupload.io/booeol](https://easyupload.io/booeol)

---

<div class="post-metadata">

**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\
**Post date:** [December 22, 2021, 7:18pm UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/13 "2021-12-22T19:18:04Z")

</div>

Thanks that helps... unfortunately I had to do something dirty and use a [multiline](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html) codec to combine the lines, remove all of the `\n` newline characters using a [gsub](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-gsub) and then write a [grok](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html) filter to get the information out.  
It's not pretty..but I got results.

INPUT

```auto
ENUM HEADER BLOCK DATE TIME 1184 LDAP FAILURE IN SEARCH RESULT SDABS 211214 134851800 SERVID USERID REASON 0 2 32 MOID ATTRBID  

```

GROK

```auto
ENUM%{SPACE}HEADER%{SPACE}BLOCK%{SPACE}DATE%{SPACE}TIME %{NUMBER:ENUM}%{SPACE}%{DATA:HEADER}%{SPACE}%{WORD:BLOCK}%{SPACE}%{NUMBER:DATE}%{SPACE}%{NUMBER:TIME}%{SPACE}SERVID%{SPACE}USERID%{SPACE}REASON%{SPACE}%{NUMBER:SERVID}%{SPACE}%{NUMBER:USERID}%{SPACE}%{NUMBER:REASON}

```

OUTPUT

```auto
{
  "ENUM": [
    [
      "1184"
    ]
  ],
  "HEADER": [
    [
      "LDAP FAILURE IN SEARCH RESULT"
    ]
  ],
  "BLOCK": [
    [
      "SDABS"
    ]
  ],
  "DATE": [
    [
      "211214"
    ]
  ],
  "TIME": [
    [
      "134851800"
    ]
  ],
  "SERVID": [
    [
      "0"
    ]
  ],
  "USERID": [
    [
      "2"
    ]
  ],
  "REASON": [
    [
      "32"
    ]
  ]
}

```

Until you can get some way to better transform the output you are going to have to manually define your grok patterns.

My logstash output:

```auto
{
        "USERID" => "2",
       "message" => "ENUM HEADER BLOCK DATE TIME 1184 LDAP FAILURE IN SEARCH RESULT SDABS 211214 134851800 SERVID USERID REASON 0 2 32 MOID ATTRBID",
      "@version" => "1",
          "tags" => [
        [0] "multiline"
    ],
          "host" => "asp123.myserver.com",
        "HEADER" => "LDAP FAILURE IN SEARCH RESULT",
          "TIME" => "134851800",
          "ENUM" => "1184",
          "DATE" => "211214",
        "SERVID" => "0",
    "@timestamp" => 2021-12-22T19:16:38.162Z,
        "REASON" => "32",
         "BLOCK" => "SDABS"
}
[INFO] 2021-12-22 14:16:40.609 [LogStash::Runner] runner - Logstash shut down.

```

Here's my logstash config:

```auto
input {
  generator {
    message => '
S P 2112141349 AF-7      

EVENT REPORTING RESULT

ENUM HEADER BLOCK DATE TIME
1184 LDAP FAILURE IN SEARCH RESULT SDABS 211214 134851700

SERVID USERID REASON                           
0 2 32

MOID                                                   

ATTRBID

ENUM HEADER BLOCK DATE TIME
1184 LDAP FAILURE IN SEARCH RESULT SDABS 211214 134851800

SERVID USERID REASON                           
0 2 32

MOID                                                   

ATTRBID

ENUM HEADER BLOCK DATE TIME
1184 LDAP FAILURE IN SEARCH RESULT SDABS 211214 134851800

SERVID USERID REASON                           
0 2 32

MOID                                                   

ATTRBID

'
  count => 1
  codec => multiline {
    pattern => 'ENUM HEADER BLOCK DATE TIME'
    what => previous
    negate => true
   }

  }

}

filter {
	mutate {
		gsub => ["message","\n"," "]
	}
	
	grok {
		match => {"message" => "ENUM%{SPACE}HEADER%{SPACE}BLOCK%{SPACE}DATE%{SPACE}TIME %{NUMBER:ENUM}%{SPACE}%{DATA:HEADER}%{SPACE}%{WORD:BLOCK}%{SPACE}%{NUMBER:DATE}%{SPACE}%{NUMBER:TIME}%{SPACE}SERVID%{SPACE}USERID%{SPACE}REASON%{SPACE}%{NUMBER:SERVID}%{SPACE}%{NUMBER:USERID}%{SPACE}%{NUMBER:REASON}"}
	}
}

output {

stdout{}
}

```

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [December 23, 2021, 12:53pm UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/14 "2021-12-23T12:53:43Z")

</div>

Thx a lot then I'll try to attach it to the stream

---

<div class="post-metadata">

**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\
**Post date:** [December 23, 2021, 2:29pm UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/15 "2021-12-23T14:29:34Z")

</div>

If it works please mark the idea that worked best for you as a solution 🙂

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [December 23, 2021, 9:05pm UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/16 "2021-12-23T21:05:07Z")

</div>

> [@AquaX](#):
>
> If it works please mark the idea that worked best for you as a solution

How it should be like file input if I want to parse several files on time  
the root cause was "count =\> 1"

```auto
input {
	file {
    mode => read
    path => "/opt/data/input/BC*"
	file_completed_action => "log"
	file_completed_log_path => "/opt/data/logstash_files/fin_log"
    sincedb_path => "/dev/null"
 
 count => 1
  codec => multiline {
    pattern => 'ENUM HEADER BLOCK DATE TIME'
    what => previous
    negate => true
   }

  }

}

```

```auto
[ERROR] 2021-12-23 21:04:15.194 [Converge PipelineAction::Create<hlr_40>] file - Unknown setting 'count' for file
[ERROR] 2021-12-23 21:04:15.211 [Converge PipelineAction::Create<hlr_40>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:hlr_40, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$ __VARARGS__ (/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:383)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}
warning: thread "Converge PipelineAction::Create<hlr_40>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<hlr_40>`
          create at org/logstash/execution/ConvergeResultExt.java:135
             add at org/logstash/execution/ConvergeResultExt.java:60
  converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:396
[ERROR] 2021-12-23 21:04:15.224 [Agent thread] agent - An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<hlr_40>`"}
[FATAL] 2021-12-23 21:04:15.246 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<hlr_40>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:396:in `block in converge_state'"]}
[FATAL] 2021-12-23 21:04:15.270 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

```

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [December 23, 2021, 9:22pm UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/17 "2021-12-23T21:22:06Z")

</div>

how to make timestamp from fields [DATE] [TIME], hmm messages over grok also doesn't parse as it was expected

 ![Capture](https://us1.discourse-cdn.com/elastic/original/3X/8/a/8a56a3f6c3afe8386ca6081ce59324760d28c603.jpeg)

---

<div class="post-metadata">

**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\
**Post date:** [December 23, 2021, 9:23pm UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/19 "2021-12-23T21:23:15Z")

</div>

> [@INS](#):
>
> `[ERROR] 2021-12-23 21:04:15.194 [Converge PipelineAction::Create<hlr_40>] file - Unknown setting 'count' for file`

You have a count as a setting. That's not a valid setting for the file input. Please read the docs

> **[File input plugin | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html)**

However, if you want to parse multiple files then you can define a pattern of file names:

`path => ["/var/log/alog.*","/var/log/adifferentlog/*.log"]`

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [December 23, 2021, 10:11pm UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/20 "2021-12-23T22:11:09Z")

</div>

it works but for config "file generator" when I use plugin file, multiline codec have a different behavior

```auto
input {
	file {
    mode => read
    path => "/opt/data/input/BC*"
	file_completed_action => "log"
	file_completed_log_path => "/opt/data/logstash_files/fin_log"
    sincedb_path => "/dev/null"
 
 
  codec => multiline {
    pattern => 'ENUM HEADER BLOCK DATE TIME'
    what => previous
    negate => true
   }

  }

}

```

```auto
{
          "path" => "/opt/data/input/BC*",
       "message" => "ENUM HEADER BLOCK DATE TIME\r 1184 LDAP FAILURE IN SEARCH RESULT SDABS 211223 200256700\r SERVID USERID REASON \r 0 2 32\r MOID \r \r ATTRBID\r \r",
          "host" => "3d174836d8ac",
      "@version" => "1",
          "tags" => [
        [0] "multiline",
        [1] "_grokparsefailure"
    ],
    "@timestamp" => 2021-12-24T10:17:20.086Z
}
{
          "path" => "/opt/data/input/BC*",
       "message" => "ENUM HEADER BLOCK DATE TIME\r 1184 LDAP FAILURE IN SEARCH RESULT SDABS 211223 200256700\r SERVID USERID REASON \r 0 2 32\r MOID \r \r ATTRBID\r \r",
          "host" => "3d174836d8ac",
      "@version" => "1",
          "tags" => [
        [0] "multiline",
        [1] "_grokparsefailure"
    ],
    "@timestamp" => 2021-12-24T10:17:20.093Z
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 24, 2021, 6:12pm UTC](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352/21 "2021-12-24T18:12:02Z")

</div>

> [@AquaX](#):
>
> `DATE%{SPACE}TIME %{NUMBER:ENUM}%{SPACE}`

Change that to `DATE%{SPACE}TIME%{SPACE}%{NUMBER:ENUM}%{SPACE}`

You may also want `start_position => beginning` on your file input.

[Next page](https://discuss.elastic.co/t/logstash-as-a-parser-for-data-in-column/292352.md?page=2)
