-Xmx sets the maximum heap size, but there's other things that use memory apart from the heap. From the docs (emphasis mine):
Set
XmxandXmsto no more than 50% of your physical RAM. Elasticsearch requires memory for purposes other than the JVM heap and it is important to leave space for this. For instance, Elasticsearch uses off-heap buffers for efficient network communication, relies on the operating system’s filesystem cache for efficient access to files, and the JVM itself requires some memory too. It is normal to observe the Elasticsearch process using more memory than the limit configured with theXmxsetting.