# Logstash input file configuration

**URL:** <https://discuss.elastic.co/t/logstash-input-file-configuration/305081>\
**Category:** Logstash\
**Created:** [May 18, 2022, 2:05pm UTC](https://discuss.elastic.co/t/logstash-input-file-configuration/305081 "2022-05-18T14:05:55Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Iss](https://avatars.discourse-cdn.com/v4/letter/i/58956e/32.png) [@Iss](https://discuss.elastic.co/u/Iss)\
**Post date:** [May 18, 2022, 2:05pm UTC](https://discuss.elastic.co/t/logstash-input-file-configuration/305081/1 "2022-05-18T14:05:55Z")

</div>

Hi!  
I use logstash to import a json file into Elasticsearch.

I use the following configuration but the index is not created in Elasticsearch (or the consol).

```auto
input{
	file{
		codec=>multiline{
			pattern =>"^{header" 
			negate=>true
			what=> previous
			auto_flush_interval=>5
			multiline_tag=>""
		}

		path=> "C:/xxx/xxx/xxx/xxxx/result.json"
		start_position=>"beginning"
		sincedb_path => "NUL"
		sincedb_write_interval =>5
	}
}
filter{

}

output {
    elasticsearch {
        hosts => "localhost:9200"
		index => "pythonfile"
    }
	stdout {codec=>rubydebug}
}

```

I tried with codec =\> "json" or "json\_lines" but get the same problem: Logstash is running but don't import the file.

```auto
[DEBUG] 2022-05-18 16:02:34.800 [pool-9-thread-1] cgroup - One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[DEBUG] 2022-05-18 16:02:35.369 [pool-9-thread-1] jvm - collector name {:name=>"G1 Young Generation"}
[DEBUG] 2022-05-18 16:02:35.369 [pool-9-thread-1] jvm - collector name {:name=>"G1 Old Generation"}
[DEBUG] 2022-05-18 16:02:38.455 [logstash-pipeline-flush] PeriodicFlush - Pushing flush onto pipeline.

```

Can you help me to fix the problem? Thank you!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 18, 2022, 5:00pm UTC](https://discuss.elastic.co/t/logstash-input-file-configuration/305081/2 "2022-05-18T17:00:23Z")

</div>

If you enable log.level TRACE then the filewatch module in the file input will log messages about whether it found the file etc.

---

<div class="post-metadata">

**Author:** ![Iss](https://avatars.discourse-cdn.com/v4/letter/i/58956e/32.png) [@Iss](https://discuss.elastic.co/u/Iss)\
**Post date:** [May 18, 2022, 10:19pm UTC](https://discuss.elastic.co/t/logstash-input-file-configuration/305081/3 "2022-05-18T22:19:22Z")

</div>

Thank you for response.  
When I enable log.level TRACE, I have this :

```auto
[TRACE] 2022-05-19 00:12:45.036 [[main]<file] processor - process_active no change {:path=>"result.txt"}
[DEBUG] 2022-05-19 00:12:46.013 [pool-3-thread-1] cgroup - One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[TRACE] 2022-05-19 00:12:46.039 [[main]<file] processor - process_closed
[TRACE] 2022-05-19 00:12:46.040 [[main]<file] processor - process_ignored
[TRACE] 2022-05-19 00:12:46.040 [[main]<file] processor - process_delayed_delete
[TRACE] 2022-05-19 00:12:46.041 [[main]<file] processor - process_restat_for_watched_and_active
[TRACE] 2022-05-19 00:12:46.041 [[main]<file] processor - process_rotation_in_progress
[TRACE] 2022-05-19 00:12:46.041 [[main]<file] processor - process_watched
[TRACE] 2022-05-19 00:12:46.042 [[main]<file] processor - process_active

```

NB: I have changed the initial file (result.json) to result.txt

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 18, 2022, 10:26pm UTC](https://discuss.elastic.co/t/logstash-input-file-configuration/305081/4 "2022-05-18T22:26:44Z")

</div>

There should be a lot more than that. Enough that you may need to use a file sharing site ([pastebin.com](http://pastebin.com), [gist.github.com](http://gist.github.com) or anywhere similar).

---

<div class="post-metadata">

**Author:** ![Iss](https://avatars.discourse-cdn.com/v4/letter/i/58956e/32.png) [@Iss](https://discuss.elastic.co/u/Iss)\
**Post date:** [May 18, 2022, 11:56pm UTC](https://discuss.elastic.co/t/logstash-input-file-configuration/305081/5 "2022-05-18T23:56:20Z")

</div>

Ok, You are right  
Here is the link [logleveltrace](https://pastebin.com/2LPuXC7p)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 19, 2022, 12:19am UTC](https://discuss.elastic.co/t/logstash-input-file-configuration/305081/6 "2022-05-19T00:19:54Z")

</div>

OK, so it found the file

handling: {:new\_discovery=\>true, :watched\_file=\>"\<FileWatch::WatchedFile: @filename='result.txt', @state=:watched, @recent\_states=[:watched], @bytes\_read=0, @bytes\_unread=0, current\_size=2199147, last\_stat\_size=2199147, file\_open?=false, @initial=true, sincedb\_key='1243747818-162593-11796480 0 0'\>"}

and knows it has to read 2.2 MB from it. So it reads the entire 2.2 MB into memory and is still waiting to see the end of the first line.

buffer\_extract: a delimiter can't be found in current chunk, maybe there are no more delimiters or the delimiter is incorrect or the text before the delimiter, a 'line', is very large, if this message is logged often try increasing the `file_chunk_size` setting. {"delimiter"=\>"\n", "read\_position"=\>2195456, "bytes\_read\_count"=\>3691, "last\_known\_file\_size"=\>2199147, "file\_path"=\>"C:/data/result.txt"}

---

<div class="post-metadata">

**Author:** ![Iss](https://avatars.discourse-cdn.com/v4/letter/i/58956e/32.png) [@Iss](https://discuss.elastic.co/u/Iss)\
**Post date:** [May 19, 2022, 1:09am UTC](https://discuss.elastic.co/t/logstash-input-file-configuration/305081/7 "2022-05-19T01:09:02Z")

</div>

> [@Badger](#):
>
> "delimiter"=\>"\n", "read\_position"=\>2195456, "bytes\_read\_count"=\>3691, "last\_known\_file\_size"=\>2199147

Good news!  
When I add delimiter=\>"\n", the index is created. But I think I missed something: only 3 results are in the index.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 19, 2022, 1:33am UTC](https://discuss.elastic.co/t/logstash-input-file-configuration/305081/8 "2022-05-19T01:33:29Z")

</div>

I am surprised you get 3. If it was a single line then I would have expected a single 2.2 MB event.

---

<div class="post-metadata">

**Author:** ![Iss](https://avatars.discourse-cdn.com/v4/letter/i/58956e/32.png) [@Iss](https://discuss.elastic.co/u/Iss)\
**Post date:** [May 19, 2022, 1:42am UTC](https://discuss.elastic.co/t/logstash-input-file-configuration/305081/9 "2022-05-19T01:42:49Z")

</div>

The index store size is 2.58 MB and it has 3 docs count

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 19, 2022, 1:50am UTC](https://discuss.elastic.co/t/logstash-input-file-configuration/305081/10 "2022-05-19T01:50:34Z")

</div>

The codec logs some messages at log level DEBUG. Taking a look at what those messages say might tell you something.

---

<div class="post-metadata">

**Author:** ![Iss](https://avatars.discourse-cdn.com/v4/letter/i/58956e/32.png) [@Iss](https://discuss.elastic.co/u/Iss)\
**Post date:** [May 19, 2022, 2:07am UTC](https://discuss.elastic.co/t/logstash-input-file-configuration/305081/11 "2022-05-19T02:07:05Z")

</div>

Okay, thank you very much.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 16, 2022, 2:07am UTC](https://discuss.elastic.co/t/logstash-input-file-configuration/305081/12 "2022-06-16T02:07:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.