Can't find elasticsearch.yml

i installed elk using docker compose my kibana works well but elasticsarch and logstash doesn't work and i cant find Elasticsearch.yml and logstash.yml to chande configuration and see what's the probleme

Can you provide more configuration? With this info users will give just general information.

  • How is the configuration of the docker-compose.yml?

This is the default location of elasticsearch installed as a service in Linux:

root@elastic:~# find / -name elasticsearch.yml
/etc/elasticsearch/elasticsearch.yml

I've prepared a docker-compose.yml file using the same image:version that you're using in your ELK:

version: '3.3'
services:
  Elasticsearch:
    image: elasticsearch:7.4.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
    ports:
      - 9200:9200
    restart: unless-stopped
    networks:
      - netElk
  Kibana:
    image: kibana:7.4.0
    container_name: kibana
    ports:
      - 5601:5601
    restart: unless-stopped
    networks:
      - netElk
    depends_on:
      - Elasticsearch
  Logstash:
    image: logstash:7.4.0
    container_name: logstash
    ports:
      - 5044:5044
    restart: unless-stopped
    networks:
      - netElk

networks:
  netElk:
    driver: bridge

Running okay:

CONTAINER ID   IMAGE                 COMMAND                  CREATED              STATUS              PORTS                              NAMES
b4f5cf7f2c24   kibana:7.4.0          "/usr/local/bin/dumb…"   15 seconds ago       Up 14 seconds       0.0.0.0:5601->5601/tcp             kibana
1045f9f0f40c   elasticsearch:7.4.0   "/usr/local/bin/dock…"   16 seconds ago       Up 14 seconds       0.0.0.0:9200->9200/tcp, 9300/tcp   elasticsearch
6bbb5a1ce07d   logstash:7.4.0        "/usr/local/bin/dock…"   About a minute ago   Up About a minute   0.0.0.0:5044->5044/tcp, 9600/tcp   logstash

The default configuration used for Elasticsearch in this docker images is:

[root@1045f9f0f40c elasticsearch]# find / -name elasticsearch.yml
/usr/share/elasticsearch/config/elasticsearch.yml

The logstash docker the same situation:

/usr/share/logstash/config/logstash.yml

And for kibana the same:

/usr/share/kibana/config/kibana.yml

AndI can see Elasticsearch is running okay:

[2022-05-07 19:18:27] {afuscoar@afuscoar} (~)$ -> curl --insecure -X GET http://localhost:9200/_cat/health
1651943909 17:18:29 docker-cluster green 1 1 3 3 0 0 0 0 - 100.0%

So I think the problem is the configuration files you're trying to do in the mapping.

  • Check your config files are right.
  • Check in your mappings.

as you see my docker-compose is running ok

CONTAINER ID        IMAGE                                                     COMMAND                  CREATED             STATUS              PORTS                              NAMES
a4785c523425        logstash:7.4.0        "/usr/local/bin/do..."   4 seconds ago       Up 3 seconds        0.0.0.0:5044->5044/tcp, 9600/tcp   logstash
2ee119b2cf2b        kibana:7.4.0          "/usr/local/bin/du..."   4 seconds ago       Up 3 seconds        0.0.0.0:8080->5601/tcp             kibana
964e20d72349        elasticsearch:7.4.0   "/usr/local/bin/do..."   4 seconds ago       Up 3 seconds        9300/tcp, 0.0.0.0:8081->9200/tcp   elasticsearch

and connection to kibana working well

[root@tcta4ws00b00000 elk]# curl -I  http://10.118.98.40:8080
HTTP/1.1 302 Found
location: /app/kibana
kbn-name: kibana
kbn-xpack-sig: 438c83a9ba6d6306923674be83f5f336
content-type: text/html; charset=utf-8
cache-control: no-cache
content-length: 0
Date: Sat, 07 May 2022 17:36:15 GMT
Connection: keep-alive

connection to Elasticsearch is also working

[root@tcta4ws00b00000 elk]#  curl -I  http://10.118.98.40:8081
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 541

logstash container is runing well

[root@tcta4ws00b00000 config]# docker inspect a4785c523425
[
    {
        "Id": "a4785c5234251a08bee2e8f39510a4636ce95620bd873e43c3d3ea45bfaf7a60",
        "Created": "2022-05-07T17:33:03.924495121Z",
        "Path": "/usr/local/bin/docker-entrypoint",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 49301,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2022-05-07T17:33:04.528785571Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:c2c1ac6b995bef47f6adb7966b814be41cd06f1ba5b18c9fb586fd360fc94837",
        "ResolvConfPath": "/var/lib/docker/containers/a4785c5234251a08bee2e8f39510a4636ce95620bd873e43c3d3ea45bfaf7a60/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/a4785c5234251a08bee2e8f39510a4636ce95620bd873e43c3d3ea45bfaf7a60/hostname",
        "HostsPath": "/var/lib/docker/containers/a4785c5234251a08bee2e8f39510a4636ce95620bd873e43c3d3ea45bfaf7a60/hosts",
        "LogPath": "",
        "Name": "/logstash",

but connection to logstash is not working

[root@tcta4ws00b00000 elk]#  curl -I  http://10.118.98.40:5044
curl: (56) Recv failure: Connection reset by peer

i figured that it doesn't create the file logstash.yaml under this directory /usr/share/logstash/config/ even if i mentioned that in volume as y can see

logstash:
    restart: unless-stopped
    container_name: logstash
    image: logstash:7.4.0
    volumes:
      - /opt/application/devops-elk/elk/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
      - /opt/application/devops-elk/elk/logstash/pipeline:/usr/share/logstash/pipeline

do ihave to create the file(logstash.yml) by my self if yes can you give me what configuration shoul i use in that file

In the docker images there are already configuration files up:

  • Where are stored the pipelines:
bash-4.2$ cat config/pipelines.yml 
# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
#   https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

- pipeline.id: main
  path.config: "/usr/share/logstash/pipeline"
  • And the simple default pipeline configuration:
bash-4.2$ cat pipeline/logstash.conf 
input {
  beats {
    port => 5044
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Of course, it just has one input (usually we receive data from any beat app -filebeat, metricbeat-). And in the output the codec used for the output data.

If you wanna send the information to the Elasticsearch instance you also need to add another directive in the output area.

For example:

# I have the input:
input {
  # In this case instead of using beats, I'm processing the changes of the
  # /var/log/kern.log file located in the same machine
  file {
    path => "/var/log/kern.log"
    start_position => "beginning"
  }
}

# I use some filtering to parse the data of this file
# And have a better representative data in the fields
# of the index
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
    }
  }
}

# Now in the output 
output {
  # I should add the elasticsearch instance where I wanna send the output
  elasticsearch { 
	hosts => ["localhost:9200"]
	index => "kernlogs" 
  }
  stdout { codec => rubydebug }
}

So in this case you should have an output with the Elasticsearch instance. In this case I have it locally but since you added them in the same network and they resolve, you can use the hostname.