Skip to main content

Migrating a single-node to a multi-node deployment

warning

You should upgrade your single-node deployment to the latest Restate version before adding more nodes.

This guide shows how to scale an existing single-node deployment to a multi-node cluster. It assumes you have a running single-node Restate server that is running the replicated loglet and replicated metadata server, which are enabled by default in Restate >= 1.4.0.

Older versions of Restate (<= 1.3.2) use the local loglet and local metadata server by default. The local loglet and local metadata server are suitable for development and single-node deployments. We recommend using the replicated loglet and replicated metadata server to ensure high availability and durability. They are also required for multi-node clusters. Starting with version 1.4.0, existing logs and metadata will be automatically migrated to the replicated equivalents.

1
Verify that node is running the replicated metadata server

Check that the metadata service is running using the restatectl tool.

restatectl metadata-server list-servers

You should see a single member node providing metadata service:

Metadata service

 NODE  STATUS  VERSION  LEADER  MEMBERS  APPLIED  COMMITTED  TERM  LOG-LENGTH  SNAP-INDEX  SNAP-SIZE

 N1    Member  v1       N1      [N1]     2        2          2     1           1           6.7 KiB

If you see the node as unreachable with an error reason of "Unimplemented", verify that you are running the latest version. The older local metadata server is no longer supported in Restate 1.4.0 and newer.

2
Verify that node is running the replicated loglet

The default configuration does not require adjustments to be cluster-ready. However, if you have explicitly specified which roles to run, you should make sure the configuration includes the log-server role. Similarly, make sure that you do not explicitly set the loglet provider type to local. While the local logelt is still supported, the default type is replicated starting from 1.4.0. Without these settings, the automatic loglet type migration will not take place.

restate.toml
roles = [

    "worker",

    "admin",

    "metadata-server",

    "log-server"

]



[bifrost]

default-provider = "replicated"

Confirm that cluster configuration uses the replicated loglet as the default log provider.

restatectl config get

In the default configuration you should expect to see:

⚙️ Cluster Configuration

├ Number of partitions: 24

├ Partition replication: {node: 1}

└ Logs Provider: replicated

 ├ Log replication: {node: 1}

 └ Nodeset size: 1

You can confirm that the type of logs in use by the server using the command:

restatectl logs list

If you have enabled the log-server role and left the default provider unset (or set it to replicated), and still do not see the cluster configuration you can change the cluster log configuration using:

restatectl config set --log-provider replicated --log-replication 1

This command sets the default log provider to replicated with a default replication of 1. As long as you have a single-node deployment, you must set the replication to 1. Otherwise, the server will become unavailable because it cannot provision the new log segments.

3
Configure snapshot repository

If you plan to extend your single-node deployment to a multi-node deployment, you also need to configure the snapshot repository. This allows new nodes to join the cluster by restoring the latest snapshot.

restate.toml
[worker.snapshots]

destination = "s3://snapshots-bucket/cluster-prefix"

4
Create snapshots to allow other nodes to join

For other nodes to join, you need to snapshot every partition because the local loglet is not accessible from other nodes. Run the following command to create a snapshot for each partition.

restatectl snapshots create --trim-log

Note that this also instructs Restate to trim the logs after partition state has been successfully published to the snapshot repository. This ensures that the logs no longer reference historic local loglets that may have existed on the original node.

5
Turn a single-node into a multi-node deployment

To add more nodes to your cluster, you need to start new Restate servers with the same cluster-name and a metadata client with the address of the existing node.

restate.toml
cluster-name = "my-cluster"



[metadata-client]

addresses = ["ADVERTISED_ADDRESS_OF_EXISTING_NODE"]

If you want the new nodes to participate in the replicated metadata service, then they should run the metadata-server role. To make nodes resilient to individual metadata service failures, specify the full list of addresses of all the nodes running the metadata service on your Restate cluster nodes.

restate.toml
[metadata-client]

addresses = ["NODE_1_ADVERTISED_ADDRESS", "NODE_2_ADVERTISED_ADDRESS", "NODE_3_ADVERTISED_ADDRESS"]

6
Verify that your cluster consists of multiple nodes

If everything is set up correctly, you should see the new nodes in the cluster status.

restatectl status

🎉
Congratulations, you migrated your single-node deployment to a multi-node deployment!

Here are some next step for you to try: