# Log stash - fetching data from file - not getting the output in the console

**URL:** <https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455>\
**Category:** Logstash\
**Created:** [May 30, 2019, 4:13am UTC](https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455 "2019-05-30T04:13:40Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![ashokchinna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashokchinna/32/47123_2.png) [@ashokchinna](https://discuss.elastic.co/u/ashokchinna)\
**Post date:** [May 30, 2019, 4:13am UTC](https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455/1 "2019-05-30T04:13:40Z")

</div>

I am trying to use the file as input source. But I am not getting the file content in the stdout.

**Command**

./logstash -f ../conf/file.conf

**Configuration File**

```
#sample configuration 1
#file.conf

input
{ file{ 
    path => "C:\Ashok\logstash-7.1.1\files\example.txt"
 }
}
 output 
{ 
 stdout { 
 codec => rubydebug 
 }
}

```

**LogStash Log:-**

```
PS C:\Ashok\logstash-7.1.1\bin> ./logstash -f ../conf/file.conf
Sending Logstash logs to C:/Ashok/logstash-7.1.1/logs which is now configured via log4j2.properties
[2019-05-29T21:11:08,309][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-05-29T21:11:08,328][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.1.1"}
[2019-05-29T21:11:14,294][INFO][logstash.javapipeline] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1500, :thread=>"#<Thread:0x52e14958 run>"}
[2019-05-29T21:11:14,998][INFO][logstash.inputs.file] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"C:/Ashok/logstash-7.1.1/data/plugins/inputs/file/.sincedb_85a3253599fdf570c96274a4a908b138", :path=>["C:\\Ashok\\logstash-7.1.1\\files\\example.txt"]}
[2019-05-29T21:11:15,034][INFO][logstash.javapipeline] Pipeline started {"pipeline.id"=>"main"}
[2019-05-29T21:11:15,121][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-05-29T21:11:15,127][INFO][filewatch.observingtail] START, creating Discoverer, Watch with file and sincedb collections
[2019-05-29T21:11:15,491][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
```

---

<div class="post-metadata">

**Author:** ![logger](https://avatars.discourse-cdn.com/v4/letter/l/34f0e0/32.png) [@logger](https://discuss.elastic.co/u/logger)\
**Post date:** [May 30, 2019, 5:31am UTC](https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455/2 "2019-05-30T05:31:42Z")

</div>

Hello there,

You Need to specify the start\_postion =\> beginning

Look at the fiel Input description and also read about the sincedb. This will help you in the furure

---

<div class="post-metadata">

**Author:** ![ashokchinna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashokchinna/32/47123_2.png) [@ashokchinna](https://discuss.elastic.co/u/ashokchinna)\
**Post date:** [May 30, 2019, 5:47am UTC](https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455/3 "2019-05-30T05:47:35Z")

</div>

I added in conf file. Still not able to see the file content in stdout

```
#sample configuration 1
#file.conf

input
{ file{ 
    path => "C:\Ashok\logstash-7.1.1\files\example.txt"
	start_position => "beginning"  
 }
}
 output 
{ 
 stdout { 
 codec => rubydebug 
 }
}
```

---

<div class="post-metadata">

**Author:** ![logger](https://avatars.discourse-cdn.com/v4/letter/l/34f0e0/32.png) [@logger](https://discuss.elastic.co/u/logger)\
**Post date:** [May 30, 2019, 6:03am UTC](https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455/4 "2019-05-30T06:03:29Z")

</div>

add a line in the file while logstash is running.

That´s why you should read the docs about sincedb.  
Because inside the sincedb file is an information about where logstash offset is set for each file.

Your offset is at the end of the file because you have started it first without the startposition. default is "end". now you have started it at the beginning. BUT logstash looks at the offset inside the sincedb file and recognizes that it is at the end.

With that logstash helps you not to read files/lines twice.

so either add a line at the end of the file.  
or close logstash, delete the sincedb file and restart logstash

---

<div class="post-metadata">

**Author:** ![ashokchinna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashokchinna/32/47123_2.png) [@ashokchinna](https://discuss.elastic.co/u/ashokchinna)\
**Post date:** [May 30, 2019, 6:49am UTC](https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455/5 "2019-05-30T06:49:04Z")

</div>

Thank you for the explanation about sincedb.

I removed the sincedb file, restarted the logstash, still not working (not getting displayed in the console).

Later, I tried to send it to elastic search, not working.

```
input
{ file{ 
    path => "C:\Ashok\logstash-7.1.1\files\example.txt"
	start_position => "beginning"  
 }
}
 output 
{ 
elasticsearch {
      index => "Test"
      document_type => "File"
      hosts => "localhost:9200"
    }
}
```

---

<div class="post-metadata">

**Author:** ![staodd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/staodd/32/24509_2.png) [@staodd](https://discuss.elastic.co/u/staodd)\
**Post date:** [May 30, 2019, 6:56am UTC](https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455/6 "2019-05-30T06:56:48Z")

</div>

Change your file path. Try using / instead of \.

---

<div class="post-metadata">

**Author:** ![ashokchinna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashokchinna/32/47123_2.png) [@ashokchinna](https://discuss.elastic.co/u/ashokchinna)\
**Post date:** [May 30, 2019, 7:08am UTC](https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455/7 "2019-05-30T07:08:58Z")

</div>

Thank you Staale by changing the path, it worked

---

<div class="post-metadata">

**Author:** ![ashokchinna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashokchinna/32/47123_2.png) [@ashokchinna](https://discuss.elastic.co/u/ashokchinna)\
**Post date:** [June 19, 2019, 7:33am UTC](https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455/8 "2019-06-19T07:33:25Z")

</div>

Any thing wrong with the below conf?

I am not seeing the output.

```
input
{ file{ 
    path => "C:/Demo/logstash-7.1.1/files/conf1.txt"
	start_position => "beginning"
	sincedb_path => "NULL"	
 }
}

filter{

	 grok {
     match => ["message", "%{GREEDYDATA:data}"]
   }

}

 output 
{ 
stdout{
		codec => rubydebug
	}
}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 19, 2019, 12:57pm UTC](https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455/9 "2019-06-19T12:57:17Z")

</div>

> [@ashokchinna](#):
>
> sincedb\_path =\> "NULL"

That should be NUL, not NULL.

---

<div class="post-metadata">

**Author:** ![ashokchinna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashokchinna/32/47123_2.png) [@ashokchinna](https://discuss.elastic.co/u/ashokchinna)\
**Post date:** [June 19, 2019, 5:04pm UTC](https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455/10 "2019-06-19T17:04:00Z")

</div>

Tried it. But didn't work.

```
PS C:\Ashok\logstash-7.1.1\bin> ./logstash -f ..\conf\conf1.conf
Sending Logstash logs to C:/Ashok/logstash-7.1.1/logs which is now configured via log4j2.properties
[2019-06-19T10:02:03,600][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-06-19T10:02:03,618][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.1.1"}
[2019-06-19T10:02:10,022][INFO][logstash.javapipeline] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1500, :thread=>"#<Thread:0x7c210d6f run>"}
[2019-06-19T10:02:10,723][INFO][logstash.javapipeline] Pipeline started {"pipeline.id"=>"main"}
[2019-06-19T10:02:10,808][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-06-19T10:02:10,815][INFO][filewatch.observingtail] START, creating Discoverer, Watch with file and sincedb collections
[2019-06-19T10:02:11,276][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 19, 2019, 5:22pm UTC](https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455/11 "2019-06-19T17:22:16Z")

</div>

Enable '--log.level trace' and see what filewatch has to say.

---

<div class="post-metadata">

**Author:** ![ashokchinna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashokchinna/32/47123_2.png) [@ashokchinna](https://discuss.elastic.co/u/ashokchinna)\
**Post date:** [June 19, 2019, 6:32pm UTC](https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455/12 "2019-06-19T18:32:15Z")

</div>

Here you go!

````
   [2019-06-19T11:27:50,498][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
    [2019-06-19T11:27:50,537][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
    [2019-06-19T11:27:50,541][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
    [2019-06-19T11:27:50,593][DEBUG][logstash.outputs.stdout] config LogStash::Outputs::Stdout/@codec = <LogStash::Codecs::RubyDebug id=>"rubydebug_7be1faaa-b97b-4351-a155-569115b6f63b", enable_metric=>true, metadata=>false>
    [2019-06-19T11:27:50,594][DEBUG][logstash.outputs.stdout] config LogStash::Outputs::Stdout/@id = "fa8cc5986c080021fbc9a8d561b69a464cf31e7a288c6ac8eb08606feee109cd"
    [2019-06-19T11:27:50,594][DEBUG][logstash.outputs.stdout] config LogStash::Outputs::Stdout/@enable_metric = true
    [2019-06-19T11:27:50,595][DEBUG][logstash.outputs.stdout] config LogStash::Outputs::Stdout/@workers = 1
    [2019-06-19T11:27:50,631][DEBUG][logstash.javapipeline] Starting pipeline {:pipeline_id=>"main"}
    [2019-06-19T11:27:50,673][INFO][logstash.javapipeline] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1500, :thread=>"#<Thread:0x790907fd run>"}
    [2019-06-19T11:27:50,734][DEBUG][org.logstash.config.ir.CompiledPipeline] Compiled output
     P[output-stdout{"codec"=>"rubydebug"}|[str]pipeline:16:1:```
    stdout{
                    codec => rubydebug
            }
    ```]
     into
     org.logstash.config.ir.compiler.ComputeStepSyntaxElement@c96a6eaa
    [2019-06-19T11:27:50,735][DEBUG][org.logstash.config.ir.CompiledPipeline] Compiled output
     P[output-stdout{"codec"=>"rubydebug"}|[str]pipeline:16:1:```
    stdout{
                    codec => rubydebug
            }
    ```]
     into
````

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 19, 2019, 6:51pm UTC](https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455/13 "2019-06-19T18:51:48Z")

</div>

filewatch logs at trace level, not debug. You are looking for messages like [these](https://discuss.elastic.co/t/logstash-api-successfully-started-but-the-data-doesnot-load/163806/4). If you are unable to interpret them yourself they will be too large to post here, so you can use your favorite internet site that supports posting logs.

---

<div class="post-metadata">

**Author:** ![ashokchinna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashokchinna/32/47123_2.png) [@ashokchinna](https://discuss.elastic.co/u/ashokchinna)\
**Post date:** [June 19, 2019, 7:02pm UTC](https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455/14 "2019-06-19T19:02:51Z")

</div>

I tried to interpret the logs. It's actually processing the file. But wondering why it's not displaying it.

Please check whether the below log helps you in finding the issue.

```
 org.logstash.config.ir.compiler.ComputeStepSyntaxElement@c96a6eaa
[2019-06-19T11:27:51,478][TRACE][logstash.inputs.file] Registering file input {:path=>["C:/Ashok/logstash-7.1.1/files/inputFile1.txt"]}
[2019-06-19T11:27:51,529][INFO][logstash.javapipeline] Pipeline started {"pipeline.id"=>"main"}
[2019-06-19T11:27:51,538][DEBUG][logstash.javapipeline] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x790907fd run>"}
[2019-06-19T11:27:51,551][DEBUG][org.logstash.execution.PeriodicFlush] Pushing flush onto pipeline.
[2019-06-19T11:27:51,576][TRACE][logstash.agent] Converge results {:success=>true, :failed_actions=>[], :successful_actions=>["id: main, action_type: LogStash::PipelineAction::Create"]}
[2019-06-19T11:27:51,621][INFO][filewatch.observingtail] START, creating Discoverer, Watch with file and sincedb collections
[2019-06-19T11:27:51,621][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-06-19T11:27:51,656][DEBUG][logstash.agent] Starting puma
[2019-06-19T11:27:51,673][DEBUG][logstash.agent] Trying to start WebServer {:port=>9600}
[2019-06-19T11:27:51,674][TRACE][filewatch.sincedbcollection] open: reading from NUL
[2019-06-19T11:27:51,682][TRACE][filewatch.sincedbcollection] open: count of keys read: 0
[2019-06-19T11:27:51,738][TRACE][filewatch.discoverer] discover_files {"count"=>1}
[2019-06-19T11:27:51,743][DEBUG][logstash.api.service] [api-service] start
[2019-06-19T11:27:51,839][TRACE][filewatch.discoverer] discover_files handling: {"new discovery"=>true, "watched_file details"=>"<FileWatch::WatchedFile:
@filename='inputFile1.txt', @state='watched', @recent_states='[:watched]', @bytes_read='0', @bytes_unread='0', current_size='24', last_stat_size='24', file_open?='false', @initial=true, @sincedb_key='2350254784-3395-1179648 0 0'>"}
[2019-06-19T11:27:51,844][TRACE][filewatch.sincedbcollection] associate: finding {"inode"=>"2350254784-3395-1179648", "path"=>"C:/Ashok/logstash-7.1.1/files/inputFile1.txt"}
[2019-06-19T11:27:51,847][TRACE][filewatch.sincedbcollection] associate: unmatched
[2019-06-19T11:27:51,909][TRACE][filewatch.tailmode.processor] Delayed Delete processing
[2019-06-19T11:27:51,917][TRACE][filewatch.tailmode.processor] Watched + Active restat processing
[2019-06-19T11:27:51,943][TRACE][filewatch.tailmode.processor] Rotation In Progress processing
[2019-06-19T11:27:51,950][TRACE][filewatch.tailmode.processor] Watched processing
[2019-06-19T11:27:51,964][TRACE][filewatch.tailmode.handlers.createinitial] handling: inputFile1.txt
[2019-06-19T11:27:51,980][TRACE][filewatch.tailmode.handlers.createinitial] opening inputFile1.txt
[2019-06-19T11:27:51,998][TRACE][filewatch.tailmode.handlers.createinitial] handle_specifically opened file handle: 100470, path: inputFile1.txt
[2019-06-19T11:27:52,021][TRACE][filewatch.tailmode.handlers.createinitial] add_new_value_sincedb_collection {"position"=>0, "watched_file details"=>"<FileWatch::WatchedFile: @filename='inputFile1.txt', @state='active', @recent_states='[:watched, :watched]', @bytes_read='0', @bytes_unread='24', current_size='24', last_stat_size='24', file_open?='true', @initial=true, @sincedb_key='2350254784-3395-1179648 0 0'>"}
[2019-06-19T11:27:52,038][TRACE][filewatch.tailmode.processor] Active - file grew: inputFile1.txt: new size is 24, bytes read 0
[2019-06-19T11:27:52,041][TRACE][filewatch.tailmode.handlers.grow] handling: inputFile1.txt
[2019-06-19T11:27:52,077][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
[2019-06-19T11:27:52,095][TRACE][filewatch.tailmode.handlers.grow] reading... {"iterations"=>1, "amount"=>24, "filename"=>"inputFile1.txt"}
[2019-06-19T11:27:52,097][DEBUG][filewatch.tailmode.handlers.grow] read_to_eof: get chunk
[2019-06-19T11:27:52,115][TRACE][filewatch.tailmode.handlers.grow] buffer_extract: a delimiter can't be found in current chunk, maybe there are no more delimiters or the delimiter is incorrect or the text before the delimiter, a 'line', is very large, if this message is logged often try increasing the `file_chunk_size` setting. {"delimiter"=>"\n", "read_position"=>0, "bytes_read_count"=>24, "last_known_file_size"=>24, "file_path"=>"C:/Ashok/logstash-7.1.1/files/inputFile1.txt"}
[2019-06-19T11:27:52,128][DEBUG][filewatch.sincedbcollection] writing sincedb (delta since last write = 1560968872)
[2019-06-19T11:27:52,132][TRACE][filewatch.sincedbcollection] sincedb_write: to: NUL
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 19, 2019, 7:54pm UTC](https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455/15 "2019-06-19T19:54:35Z")

</div>

> [@ashokchinna](#):
>
> [filewatch.tailmode.handlers.grow] buffer\_extract: a delimiter can't be found in current chunk, maybe there are no more delimiters or the delimiter is incorrect or the text before the delimiter, a 'line', is very large, if this message is logged often try increasing the `file_chunk_size` setting. {"delimiter"=\>"\n", "read\_position"=\>0, "bytes\_read\_count"=\>24, "last\_known\_file\_size"=\>24, "file\_path"=\>"C:/Ashok/logstash-7.1.1/files/inputFile1.txt"

This is the important line. It finds the file, reads 24 bytes from it, then leaves them in the buffer, waiting for a line terminator to be added to the file. Your file has no line terminator at the end.

---

<div class="post-metadata">

**Author:** ![ashokchinna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashokchinna/32/47123_2.png) [@ashokchinna](https://discuss.elastic.co/u/ashokchinna)\
**Post date:** [June 19, 2019, 8:57pm UTC](https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455/16 "2019-06-19T20:57:48Z")

</div>

@Badger you are right, I just added a carriage return(\r) in that file and the logstash started to push the data to ES and on console output.

Thank you for your help.

Is it mandate to have a file terminator or delimiter in a file? Is there a pre-defined list of delimiters?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 19, 2019, 10:29pm UTC](https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455/17 "2019-06-19T22:29:03Z")

</div>

> [@ashokchinna](#):
>
> Is it mandate to have a file terminator or delimiter in a file? Is there a pre-defined list of delimiters?

Yes , it is mandatory. The [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-delimiter) says the native line terminator is the default value.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 18, 2019, 9:29pm UTC](https://discuss.elastic.co/t/log-stash-fetching-data-from-file-not-getting-the-output-in-the-console/183455/20 "2019-07-18T21:29:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.