[Elasticsearch] The Server Is Not A Supported Distribution Of Elasticsearch

Summary

The recent upgrade of the Python Elasticsearch client to version 7.14.0 has caused compatibility issues with AWS Elasticsearch Service due to Elastic's decision to restrict support to its official distribution, necessitating a choice between downgrading or migrating to Elastic Cloud or AWS OpenSearch.

Abstract

Users who have updated their Python Elasticsearch client to version 7.14.0 are encountering an error when connecting to AWS Elasticsearch Service, which is running an older version v7.9. This is because Elastic has modified its client to only support its official Elasticsearch distribution, effectively blocking connections to forked versions like AWS's Open Distro for Elasticsearch. The error message states that the server is not a supported distribution of Elasticsearch. Elastic's response to the issue is to direct users to their Elastic Cloud service for a supported version. This change is part of Elastic's broader shift in licensing and business strategy, which has led to the creation of OpenSearch by AWS. Users are now faced with the decision of either reverting to an older client version as a short-term fix or considering a strategic migration to either Elasticsearch or OpenSearch in the long term. The community is also questioning the trustworthiness of Elastic's products following this move.

Opinions

The author expresses skepticism about Elastic's decision to restrict client support, questioning the necessity and impact on trust.
There is a preference indicated for AWS OpenSearch, with the belief that great open-source software should be built collaboratively with a diverse community.
The author suggests that the long-term solution involves a strategic decision and possible migration, considering the ongoing development of OpenSearch clients by AWS.
The community, including open-source developers who previously contributed to Elasticsearch, feels let down by Elastic's licensing changes and the resulting corporate battle.

Python Elasticsearch v7.14.0 Error

If you recently upgraded your Python elasticsearch client version to 7.14.0 , you may find yourself running into the following error when trying to connect to your existing AWS Elasticsearch Service (My version is v7.9 )

20210824 08:06:40.249330 [MainThread:INFO] run_app: es_log_level=WARNING
20210824 08:06:40.884733 [MainThread:CRITICAL] __init__: Uncaught exception
Traceback (most recent call last):
  File "/home/xxxx/src/run.py", line 26, in <module>
    root()
  File "/home/xxxx/venv/lib/python3.7/site-packages/click/core.py",  
...
File "/home/xxxx/venv/lib/python3.7/site-packages/elasticsearch/transport.py", line 413, in perform_request
    _ProductChecker.raise_error(self._verified_elasticsearch)
  File "/xxxx/venv/lib/python3.7/site-packages/elasticsearch/transport.py", line 630, in raise_error
    raise UnsupportedProductError(message)
elasticsearch.exceptions.UnsupportedProductError: The client noticed that the server is not a supported distribution of Elasticsearch

Error highlight:

“The client noticed that the server is not a supported distribution of Elasticsearch"

What Happened?

Elastic has modified the official Python client (Almost all its clients) for its Elasticsearch database not to work with forked versions and closed the GitHub issue to comments. Their office comment for Python clients is

The Elasticsearch clients go through rigorous testing and validation against the official Elasticsearch distribution coming from Elastic. Since the AWS Elasticsearch Service has incompatible APIs with Elasticsearch itself, either by missing APIs or has incompatible format, we do not support it. Please consider using Elastic Cloud for an official and formally supported hosted Elasticsearch distribution.

Elasticsearch was an open-source product, but in January changed its licensing in response to what the company saw as unfair usage by public cloud platforms, AWS in particular. AWS forked Elasticsearch, before the license change, to create the Open Distro for Elasticsearch, which later evolved into OpenSearch, which had its 1.0 release last month. Casualties of this corporate battle include open-source developers who contributed to Elasticsearch but now feel let down.

So when you upgrade your Elasticsearch client version to v7.14.0 , you picked up on this quiet change to the source code, which checks that the database is Elastic’s product and not a fork.

If there is no X-Elastic-Product HTTP header in the response or if the X-Elastic-Product HTTP header value isn’t Elasticsearch raise an error

, is one of the changes.

How to Resolve This Issue?

Elasticsearch official answer/solution is:

Please consider using Elastic Cloud for an official and formally supported hosted Elasticsearch distribution.

I don’t have much confidence in it, they don’t really have to make this move, and because of this move, how much trust should the community put into their product going forward??

Anyway, I have the following two solutions for you, if you run into this issue like me:

Short term solution: You don’t have other choices but revert back your Elasticsearch client version to <v7.14.0 , unfortunately. So your current production environment can still work.

Long-term solution: Have a strategic conversation with your manager and dev team, make a decision on what Elasticsearch product you should use, basically Elasitcsearch vs OpenSearch (AWS). Then make medium/long-term plans for migration if you do decide to go with Elasticsearch. But I do believe that great open source software is built together, with a diverse community of contributors. So my personal preference is to stick with AWS OpenSearch and wait for their release of the OpenSearch clients. They already forked the main client branch and developing new clients. The new clients will offer the same APIs and functionality they use today.