Field 'tags' is backed by an array but the associated Spark Schema does not reflect this

Elastic Stack Elasticsearch
1

Hi,
I am trying to read the data from index in ES using pyspark. I am able to load the data into the dataframe. Used this code
reader = spark.read.format("es").option("es.read.metadata", True).option("es.port","9200").option("es.nodes", "http://xx.x.x.xxx").option("es.read.field.as.array.include","XRAM,XCPU,count,message")

df = reader.load("webserverstats")

df.printSchema()

image
image.png1104×375 8.97 KB
. Its schema is there. Later on this error occured.
new_df=df.toPandas()
new_df.show()
image
image.png1334×617 34.6 KB
.

Could anybody help resolve this issue.

Thanks in advance.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.