Elastic Cloud Enterprise 1.1.0 upgrade issues and workaround

Last week we released ECE 1.1.0 with new features and bug fixes. Since then, we’ve discovered a critical bug that leads to Kibana being unavailable post-upgrade. This bug only affects users who upgraded from 1.0.x versions of ECE to 1.1.0. 

Note: NEW installs of ECE 1.1.0 are not affected

Issue

If you have upgraded to ECE 1.1.0 and any of the following are true:

  • You have tried to restart allocators
  • You have tried to increase capacity by adding hosts.,

If you have done those things, you will see Kibana authentication issues and unavailability. Monitoring will also be affected.

Immediate Workaround

As an immediate workaround, we’ve created a script that can fix this issue on your deployment, which is available for download. You can run this script anywhere that has access to the coordinator node on port 12400.

The script requires two pieces of information:

  1. NODE_IP which is the coordinator’s IP address
  2. AUTH which is the password for the admin console root user

Note: The software package jq is required on the host running this script for it to run properly.

Here’s what it looks like:

[user@localhost ~]$ curl -O https://download.elastic.co/cloud/fix-ece-1.1.0-upgrade.sh
[user@localhost ~]$ chmod a+x ./fix-ece-1.1.0-upgrade.sh
[user@localhost ~]$ ./fix-ece-1.1.0-upgrade.sh
Please enter IP address of the coordinator:<your ip here>
Please enter root password to admin console:<your password here>

The script will test connectivity, and if successful, make the changes. If not, it will let you know. After that has happened, it will prompt you to remove frc-services-forwarders-services-forwarder from every host. 

This is a manual step, which you can do with this command on every host running ECE:

docker rm -f frc-services-forwarders-services-forwarder

Once this is removed, ECE will automatically restart the above service with the patched configuration.

Official patched version

We are working on a 1.1.1 release that addresses this situation. We are targeting a 1.1.1 release early next week which will allow you to upgrade from 1.1.0 or any older versions.

If you are planning to upgrade your existing deployment to 1.1.0, we strongly recommend to hold off until we release 1.1.1.

We apologize for the inconvenience this may have caused and look forward to releasing 1.1.1 ASAP. Please contact us if you have any questions about this.