PHP-FPM (or nginx) isn't able to index to elasticsearch

Hi, it's me again, and still having the same issue I was having in this topic: File not found when attempting to index . However, I have made some progress. I now know that when I attempt to, I get the following:

2023-12-08 16:02:26 172.19.0.4 -  08/Dec/2023:19:02:26 +0000 "POST /admin.php" 404
2023-12-08 16:02:28 172.19.0.4 -  08/Dec/2023:19:02:26 +0000 "GET /index.php" 200
2023-12-08 16:02:29 172.19.0.4 -  08/Dec/2023:19:02:27 +0000 "GET /index.php" 200

I have researched and it seems like it's an issue with nginx's connection, like the path being wrong, but I have checked that plenty of times to no avail. The following are nginx's config:

server {
  listen 80;
  root /var/www/html;
  index index.php index.html index.htm;
  server_name 127.0.0.1;
  location ~\.php$ {
    fastcgi_pass App:9000;
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    fastcgi_index index.php;
    fastcgi_param REQUEST_METHOD $request_method;
    fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_read_timeout 500;
    include fastcgi_params;
  }	

  location / {
    try_files $uri $uri/ /index.php$query_string;
    gzip_static on;
  }
 
}

...And then the docker compose:

version: '2.2'
services:
   web:
     image: nginx:latest 
     ports: 
      - "127.0.0.1:80:80" 
     networks:
      - elastic
     volumes:
      - ./nginx/conf.d/default.conf:/etc/nginx/conf.d/defaul.conf 
      - ./php/App:/var/www/html
     links:
      - app 
   
   app:
     image: php:8-fpm
     build: 
       dockerfile: ./Dockerfile
     networks:
      - elastic
     volumes: 
      - ./php/App:/var/www/html

Anyone can see the issue here? It can do searches just fine, but posting stuff just gets me file not found. Thank you in advance for your time.

There is nothing about Elasticsearch in the logs and docker compose you shared, it is not clear what is the issue here.

Also, in the other post you shared you didn't provide any error log.

Without any logs is pretty hard to see what is the issue.

Issue is, I'm pretty sure the index request isn't even getting to elasticsearch for it to have logs about in the first place. From my research, nginx can't find elasticsearch to make the indexing request for some reason, hence the 404 code I posted here. I think the issue is the nginx configurations, and was hoping to find someone who went through this already.

I'm not sure how this is related to Elasticsearch, you have a 404 to a admin.php file, this seems unrelated to Elasticsearch.

Where is the request to Elasticsearch? You have a PHP application that will then try to write on Elasticsearch and you can reach your PHP application, is that?

Also, is this your entire docker compose?

Where is the elasticsearch stack? What is listening on port 9000? didn't say anything exposing this on your compose.

From what you shared it looks like that you have a PHP application running on the app container and listening on port 9000 and this application should index some data to Elasticsearch?

I have an application in PHP that should index things to elasticsearch. I'm using nginx and php-fpm, so nginx is handling the connection. The php file is in the link to the other post I made.

As for the docker compose, I can post it. Here:

version: '2.2'
services:
   web:
     image: nginx:latest 
     ports: 
      - "127.0.0.1:8000:80" 
     networks:
      - elastic
     volumes:
      - ./nginx/conf.d/default.conf:/etc/nginx/conf.d/defaul.conf 
      - ./php/App:/var/www/html/
     links:
      - app 
   
   app:
     image: php:8-fpm
     ports: 
      - "127.0.0.1:9000:80" 
     build: 
       dockerfile: ./Dockerfile
     mem_limit: 512m 
     mem_reservation: 512M 
     working_dir: /var/www/
     networks:
      - elastic
     volumes: 
      - ./php/App:/var/www/html/

   es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.1
    container_name: es01
    environment:
      - node.name=es01
      - xpack.security.enabled=false
      - discovery.type=single-node
      - bootstrap.memory_lock=true
    volumes:
      - ./php/App/prog_book.csv:/usr/share/prog_book.csv      
    ports:
      - "127.0.0.1:9200:9200"
    networks:
      - elastic

   kib01:
    image: docker.elastic.co/kibana/kibana:7.17.1
    container_name: kib01
    ports:
      - "127.0.0.1:5601:5601"
    volumes:
      - ./www:/usr/share/tcc/kibana.yml
      - ./php/App/prog_book.csv:/usr/share/prog_book.csv
    environment:
      ELASTICSEARCH_HOSTS: http://es01:9200
    depends_on: 
      - es01 
    networks:
      - elastic

   logstash:
    image: docker.elastic.co/logstash/logstash:7.17.1
    environment:
      - xpack.monitoring.enabled=false
    command: logstash -f /usr/share/logstash/pipeline/logstash.conf
    volumes:
      - ./logstash/logconf.d:/usr/share/logstash/pipeline/logstash.conf
      - ./logstash/patterns:/opt/logstash/patterns
      - ./php/App/prog_book.csv:/var/www/html/prog_book.csv
    depends_on:
      - es01
    networks:
      - elastic     
   
volumes:
  kibana.yml:
    driver: local  

networks:
  elastic:
    driver: bridge

Frankly, app doesn't need the port bit, it's handled by nginx, but it was something my teacher thought was the issue. I can cut that and it works the same. The dockerfile installs and updates php-fpm, composer, and helps logstash dump the contents of a csv into elasticsearch on startup.

Some stuff there might be redundant, since I have been trying a ton of stuff to make this work, and I am fully aware the code is frankensteined out of a lot of other stuff. Can't find good examples for what I want to do, so I'm using bits and pieces. Keep in mind that it is fully capable of searching, just indexing that becomes a pain. Kibana can also index through it's own terminal, so the issue is somewhere between the php and elasticsearch. I think it's missing some sort of permission.

Also! I did look for the elasticsearch logs in docker desktop, both the real time stuff and the file. It doesn't seem like it's reaching elasticsearch. I'll post an excerpt from the nginx container logs:

2023-12-08 18:52:52 172.26.0.1 - - [08/Dec/2023:21:52:52 +0000] "GET /index.php?q=ai HTTP/1.1" 200 1103 "http://127.0.0.1:8000/index.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0" "-"
2023-12-08 18:52:55 172.26.0.1 - - [08/Dec/2023:21:52:55 +0000] "GET /favicon.ico HTTP/1.1" 200 547 "http://127.0.0.1:8000/index.php?q=ai" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0" "-"

This is what happens when I get a successful search;

2023-12-08 18:53:02 172.26.0.1 - - [08/Dec/2023:21:53:02 +0000] "POST /admin.php HTTP/1.1" 404 27 "http://127.0.0.1:8000/inserir.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0" "-"
2023-12-08 18:53:04 172.26.0.1 - - [08/Dec/2023:21:53:04 +0000] "GET /favicon.ico HTTP/1.1" 200 547 "http://127.0.0.1:8000/admin.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0" "-"

This is what happens when I get an unsuccessful index. It looks like meaningless wall of text to me, but I'm clearly not very smart.

Update! It seems like nginx can't simply connect to elasticsearch, so I had to do some stuff on the nginx config:

upstream elasticsearch {
    server 127.0.0.1:9200;
    keepalive 15;
  }
server {
    listen 8080;
    location / {
        proxy_pass "http://es01";
        proxy_http_version 1.1;
        proxy_set_header Connection "Keep-Alive";
        proxy_set_header Proxy-Connection "Keep-Alive";
        proxy_read_timeout 10s;
    }
}

server {
  listen 80;
  root /var/www/html;
  index index.php index.html index.htm;
  server_name 127.0.0.1;
  location ~\.php$ {
    fastcgi_pass App:9000;
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    fastcgi_index index.php;
    fastcgi_param REQUEST_METHOD $request_method;
    fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_read_timeout 500;
    include fastcgi_params;
  }	

  location / {
    try_files $uri $uri/ /index.php$query_string;
    gzip_static on;
  }
  
}

And now, I get an extra line of error:

2023-12-08 21:06:08 172.28.0.1 - - [09/Dec/2023:00:06:08 +0000] "POST /admin.php HTTP/1.1" 404 27 "http://127.0.0.1:8000/inserir.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0" "-"
2023-12-08 21:06:09 172.28.0.1 - - [09/Dec/2023:00:06:09 +0000] "GET /favicon.ico HTTP/1.1" 499 0 "http://127.0.0.1:8000/admin.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0" "-"
2023-12-08 21:06:12 172.28.0.1 - - [09/Dec/2023:00:06:12 +0000] "GET /favicon.ico HTTP/1.1" 200 547 "http://127.0.0.1:8000/inserir.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0" "-"

499 is the code for the connection dying mid request. New info is welcome, so I'm sharing it!

From everything you shared it seems that you have some issues on both your docker-compose and nginx configuration.

First, I'm not sure your Elasticsearch is even working, on your docker-compose you are not binding any persistent volume for it, which means that if it works, you will lose everything when it restarts.

I would suggest that first you make sure that your Elasticsearch and Kibana are working, the following docker compose will spin-up a single-node cluster without authentication on version 7.17.15.

version: '2.2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.15
    container_name: elasticsearch
    environment:
      - node.name=elasticsearch
      - cluster.name=es-docker
      - discovery.type=single-node
      - xpack.security.enabled=false
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - elasticdata:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic

  kibana:
    image: docker.elastic.co/kibana/kibana:7.17.15
    container_name: kibana
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    volumes:
      - kibanadata:/usr/share/kibana/data
    ports:
      - 5601:5601
    networks:
      - elastic

volumes:
  elasticdata:
    driver: local
  kibanadata:
    driver: local

networks:
  elastic:
    driver: bridge

Then your Nginx configuration looks wrong, first you are declaring a upstream but you are not using it:

upstream elasticsearch {
    server 127.0.0.1:9200;
    keepalive 15;
  }

Also, using server 127.0.0.1:9200 means that Elasticsearch should be running on the same container as Nginx, which is not the case.

This here will also not work:

proxy_pass "http://es01";

Your container es01 does not have port 80 exposed.

I would suggest that you leave nginx to run only your php application, there is no need to use it to talk to elasticsearch.

You should split your problem in 2 parts, first make sure that Elasticsearch is running, the docker-compose that I shared above you solve this, then you make sure that your PHP application and Nginx are working, create a simple endpoint that would render something in the browser, like print the phpinfo() when you make a request to port 8080, when you have this working then you can start troubleshooting the connection between your PHP container and Elasticsearch.

Validate that you can get information from Elasticsearch using PHP, you can use this example in the documentation.

If everything works, then you can try to index your data and see if you will get any error or not.

At the moment is not possible to know if you have any issue with elasticsearch as everything seems to be related to docker or nginx, which are out of the scope of this forum.

Well, the thing is the elasticsearch and kibana both do work. I have indexed stuff with logstash, I have done searches through php/nginx, I have indexed stuff through kibana (though that wasn't kept, admittedly; I was gonna solve that next). That's how I know everything is working except specifically the indexing of new data. I'm all in for not messing with nginx, but I ran out of options, so...

Pictured: successful search in php using that mess.


..

If you can index in your Elasticsearch cluster using Logstash and can even query using your php application, but you cannot index, then you need to validate your code and troubleshoot it.

It is not possible to know what is happening without seeing what you are receiving from Elasticsearch when you try to index some data.

When you make a request to index something in Elasticsearch you will always get a response, it can be a success response or some error, you need to capture that response and share it.

The documentation has some examples about it.

In your other post you said you are using this code:

<?php
require_once 'init.php';
if(!empty($_POST)){
  if(isset($_POST['Book_title'], $_POST['Description'], $_POST['Number_Of_Pages'], $_POST['Price'], $_POST['Rating'], $_POST['Reviews'], $_POST['Type'])){
  	$Book_title = $_POST['Book_title'];
  	$Description = $_POST['Description'];
  	$Number_Of_Pages = $_POST['Number_Of_Pages'];
  	$Price = $_POST['Price'];
  	$Rating = $_POST['Rating'];
  	$Reviews = $_POST['Reviews'];
  	$Type = $_POST['Type'];
  	$postData = $es->index([
      'index' => 'books',
      'type' => '_doc',
      'body' => [
         'Book_title' => $Book_title,
         'Description' => $Description,
  	     'Number_Of_Pages' => $Number_Of_Pages,
  	     'Price' => $Price,
  	     'Rating' => $Rating,
  	     'Reviews' => $Reviews,
  	     'Type' => $Type
      ]  
  	]);
    if($postData){
      echo '<div class="alert" role="alert"> Documento inserido com sucesso. </div>';
    }
  }
}
?>
<!DOCTYPE html>
<html>
<head>
  <title>Inserir dados:</title>
  <link rel="stylesheet" href="estilo.css">
</head>
<body>
 <form action="admin.php" method="post" autocomplete="off">
  <label> Insira dados do livro: <br><br>
    <input type="text" name="Book_title" placeholder="Título do livro:"><br><br>
    <textarea name="Description" rows="8" placeholder="Descreva o livro:"></textarea> <br><br>
    <input type="int" name="Number_Of_Pages" placeholder="Número de paginas:"><br><br>
    <input type="float" name="Price" placeholder="Preço:"><br><br>
    <input type="float" name="Rating" placeholder="Pontuação:"><br><br>
    <input type="int" name="Reviews" placeholder="Número de reviews:"><br><br>
    <input type="text" name="Type" placeholder="Encadernamento"><br><br>
  </label>
  <input type="submit" value="Inserir">
  </form> 
</body>
</html>

And are getting a File not Found message, but as mentioned this is not from Elasticsearch, it does not throw any File not found error, so it is something in your code.

Also, you can remove this line 'type' => '_doc', there are no types in elasticsearch anymore.

I would suggest that you change this:

    if($postData){
      echo '<div class="alert" role="alert"> Documento inserido com sucesso. </div>';
    }

And print the response you have in $postData to help understand what is happening.

1 Like

Managed to solve it, though I did have to update ELK to the newest versions to access the resources you linked (which weren't available in 7.17; last time I tried to run 8 it didn't work). Thank you very much for the help, feels good to finally be done with this.

And how did you solve it? Share what was the issue and how you solved, it may help other people in the future.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.