File paths must be absolute, relative path specified

[Cyphy]

hey there,
hope you can help me

I started my ELK setup with docker-compose a few days ago. It looked fine as all 3 containers where running without any data. Today i wanted to import some xml files with logstash and xpath, but it already errors when he wants an absolute path. My path "C:/Users/username/Desktop/DockerComposeFile/data/data_1.xml" is absolute indeed (i replaced my username with username ). I first tried with *.xml at the end, and the same error occured. Then i tried this version, but also the same error. I also tried with normal slashes, but didn't work eather. I am working on a Windows machine, just for notice. The full Error is:

Pipeline error {:pipeline_id=>"main", :exception=>#<ArgumentError: File paths must be absolute, relative path specified: C:/Users/username/Desktop/DockerComposeFile/data/data_1.xml>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/logstash/inputs/file.rb:283:in `block in register'", "org/jruby/RubyArray.java:1809:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/logstash/inputs/file.rb:281:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in `block in register_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:386:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:311:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x55796994@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:54 run>"}

EDIT:

my .config is:

input {
  file{
	path => "C:\Users\username\Desktop\DockerComposeFile\data\data_1.xml"
	type => "xml"
	start_position => "beginning"
	sincedb_path => "/dev/null"
  }
}
filter {
	xml {
        source => "message"
        store_xml => false
        xpath => [
				"//GetRecordByIdResponse/MD_Metadata/@fileIdentifier", "identifier",
                "//GetRecordByIdResponse/MD_Metadata/@dateStamp", "date",
                "//GetRecordByIdResponse/MD_Metadata/identificationInfo/SV_ServiceIdentification/citation/CI_Citation/@title", "title"
        ]
	}
}
output {
  elasticsearch { 
	hosts => "elasticsearch:9200"
	index => "searchData"
  }
}

Hope you can help me.

Cheers

[stephenb (Stephen Brown)]

Try using forward slashes / for the path

[warkolm (Mark Walkom)]

@Cyphy

	sincedb_path => "/dev/null"

This is also nul on Windows, so sincedb_path => "nul"

[Cyphy]

i already tried this before and doesn't worked either.

[Cyphy]

I tried this now, but it doesn't seem to help me with the main Problem. Got again the same Error message :

[2021-01-03T16:03:16,414][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<ArgumentError: File paths must be absolute, relative path specified: C:\Users\username\Desktop\DockerComposeFile\data\data_1.xml>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/logstash/inputs/file.rb:283:in `block in register'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/logstash/inputs/file.rb:281:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in `block in register_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:386:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:311:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x18ba32e2@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:54 run>"}
```

[stephenb (Stephen Brown)]

@Cyphy

/usr/share/logstash/pipeline/logstash.conf

You do need to use forward slashes, I know you tried but that is the proper syntax.

Windows logstash setting issue!

I have tried this code. my template is creating properly but index are not generating from the file. this is logstash.conf file. i am guessing it is 'sincedb' issue but I am confused. OS : windows 10, ELK version : 6.5.4 input { file { path => "C:\elk\logstash-6.5.4\logstash-6.5.4\config\review_data.json" start_position => "beginning" sincedb_path => "null" codec => "json" } } output { stdout { codec => rubydebug } elasticsearch { hosts => ["http://localhost:9200/…

and technically I think it is sincedb_path => "NUL"

[Cyphy]

I tried what u mentioned, but it doesn't work eather

That is my .conf file:

input {
 file{
   path => "C:/Users/username/Desktop/DockerComposeFile/data/data_1.xml"
   type => "xml"
   start_position => "beginning"
   sincedb_path => "NUL"
 }
}
filter {
   xml {
       source => "message"
       store_xml => false
       xpath => [
   			"//GetRecordByIdResponse/MD_Metadata/@fileIdentifier", "identifier",
               "//GetRecordByIdResponse/MD_Metadata/@dateStamp", "date",
               "//GetRecordByIdResponse/MD_Metadata/identificationInfo/SV_ServiceIdentification/citation/CI_Citation/@title", "title"
       ]
   }
}
output {
 elasticsearch { 
   hosts => "elasticsearch:9200"
   index => "searchData"
 }
}

and that is my Error Message:

[2021-01-03T17:02:45,689][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<ArgumentError: File paths must be absolute, relative path specified: C:/Users/username/Desktop/DockerComposeFile/data/data_1.xml>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/logstash/inputs/file.rb:283:in `block in register'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/logstash/inputs/file.rb:281:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in `block in register_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:386:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:311:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x172ee530@/usr/share/logstash/logstash-core/lib/logstash/pipelines_registry.rb:140 run>"}

The Error message is pretty much the same, only the last part is a little bit different.
I also tried to use double slashes after C: but it won't worked eather. I also tried sincedb_path => "null" or sincedb_path => "NULL", both doesn't worked.

[stephenb (Stephen Brown)]

It appears you are running logstash inside a docker container (I missed that), did you make your windows filesystem available to the docker container?

The docker images are Unix Based, it will work but you will need to mount the files system and then the sincedb_path => "/dev/null" again because it is inside the container. This can be a bit confusing.

Perhaps download and logstash outside of docker and try again. This is pretty common if you are just learning.
https://artifacts.elastic.co/downloads/logstash/logstash-7.10.1-windows-x86_64.zip

[Cyphy]

@stephenb

/dev/null

Okay i think this may do the trick. but i have not really an idea how i mount the Windows File system to the Linux based. I found a lot that i could change the allowed Files in the settings, but as i am using WSL 2 there is no option for me in the settings. I think about that i have to configure this in the docker-compose.yml under volumes, but not sure about that.
That is my File:

version: '2.2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.1
    container_name: elasticsearch
    hostname: elasticsearch
    environment:
      - node.name=elasticsearch
      - discovery.seed_hosts=elasticsearch
      - cluster.initial_master_nodes=elasticsearch
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elknetwork
      
  kibana:
    image: docker.elastic.co/kibana/kibana:7.10.1
    container_name: kibana
    hostname: kibana
    environment:
        ELASTICSEARCH_URL: "http://elasticsearch:9200"
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
    networks:
      - elknetwork
      
  logstash:
    image: docker.elastic.co/logstash/logstash:7.10.1
    container_name: logstash
    hostname: logstash
    ports:
      - 9600:9600
      - 8089:8089
    volumes:
    - .\logstash:/usr/share/logstash/pipeline/
    depends_on:
      - elasticsearch
    networks:
      - elknetwork
      
networks:
    elknetwork:
        driver: bridge
volumes:
  data01:
    driver: local

I assume that my volumes are wrong configurated. Inside logstash part but inside Elasticsearch as well. I think that the error doesn't occured cause i wasn't searching anything. but not sure about it.
But thank u for ur help anyways. i think there is more i have to investigate about the docker system :).

[Cyphy]

Nevermind, i switched - .\logstash:/usr/share/logstash/pipeline/ to C:/Users/username/Desktop/DockerComposeFile/data/, and at least the Error message is not showing anymore, and the container is running. I think there is still a problem because the index isn't showing up in kibana, but i think i can handle that. At least thanks for your help and the tipps. If u see any shit in my configurations feel free to tell me :D. I have a lot to learn

Cheers