Moving Firestore Data between Different Projects (GCP)

Have you ever wanted to move Firestore data between two projects? maybe to create a clone and run tests or to migrate from one project to another?

The backstory

Recently we had a situation where we needed to backtrack some operations done on data in production. We needed to make sure the backtrack will run smoothly and correctly and so the need to have a cloned copy of the production state in dev and use the data copied to run tests before moving to production.

Our set-up had two firebase projects. One used as our production instance the other for development. A Firebase project is the top-level entity for Firebase.

When you create a new Firebase project in the Firebase console, you’re actually creating a Google Cloud project behind the scenes. You can think of a Google Cloud project as a virtual container for data, code, configuration, and services. Basically, a Firebase project is a Google Cloud project.

The example in this article demonstrates how to export Firestore data from a source project(production)and then import that data into a destination project(development) Firestore.

So how did we do it?

Before you can use the managed export and import service, you must complete the following tasks:

1. Enable billing for both your source project and destination project. Only Google Cloud projects with billing enabled can use the export and import functionality.
2. Make sure your account has the necessary Cloud IAM permissions in your source project and destination project. i.e Owner, Cloud Datastore Owner, or Cloud Datastore Import Export Admin
3. Set up the gcloud command-line tool and connect to your project in one of the following ways:

• Access gcloud from the Google Cloud console using Cloud Shell.
• Start Cloud Shell
• Make sure gcloud is configured for the correct project:gcloud config set project [DESTINATION_PROJECT_ID]

1) Export Data from source project.

Steps

1. Create a cloud storage bucket in the source project.
2. Use the gcloud firestore export command to export data from your source project. Replace [SOURCE_BUCKET] with the name of your Cloud Storage bucket:

• Export all data: gcloud firestore export gs://[SOURCE_BUCKET] — -async
• Export specific collections: gcloud firestore export gs://[SOURCE_BUCKET] — -collection-ids = [COLLECTION_ID_1],[COLLECTION_ID_2] — -async

3. Take note of your export operation’s outputURIPrefix as you use this next to check on the progress of the export with the command: gcloud firestore operations describe [outputURIPrefix]

Wait till operationState is successful

Here’s the catch, If your source bucket location is different from the Cloud Firestore location of your destination project, you must move your data files to a Cloud Storage bucket in the same location as your destination project. Before importing the data to the destination firestore location, do a transfer from the source storage bucket to the destination storage bucket.

2) Transfer data From one bucket to another

Steps

1. Create a cloud storage bucket in the destination project.(must have a unique name)to receive the data being transfered.
2. Open the Transfer page in the Google Cloud Console.
3. Click Create transfer job.
4. Follow the step-by-step walkthrough, clicking Continue as you complete each step:

• Select Source: Use Google Cloud Storage Bucket as your selected source, and click Browse to find and select the bucket you want to move your objects out of.(the source bucket in production)

Browsing for the source bucket

• Select Destination: Click Browse to find and select the bucket you want to move your objects into. In most cases browse won’t show the destination’s bucket, you can just enter its name manually.

Type in the destination bucket

• Additionally, select the checkbox Delete source objects after the transfer completes.
• Configure Transfer: You can ignore this section.

5. After you complete the step-by-step walkthrough, click Create. This begins the process of copying objects from your source bucket into your destination one. This process may take some time. You can view the transfer progress on the transfer page.

Transfer progress-completed

3) Import data from source project

Steps

Before you can begin an import operation, you must make sure your destination project can access your Cloud Firestore data files.

1. Create a cloud storage bucket in the destination project.(must have a unique name).
2. Make sure gcloud is configured for the correct project: gcloud config set project [DESTINATION_PROJECT_ID]
3. Use the gcloud firestore import command to import the data in your destination bucket into your destination project where [EXPORT_PREFIX]matches the pre-fix in your export operation's outputUriPrefixe.g gcloud firestore import gs://[SOURCE_BUCKET]/2019-03–05T20:58:23_56418 — -async
4. Check on the progress of the import with the command: gcloud firestore operations describe [outputURIPrefix]

You did it! You have successfully moved data from one project to another.

Thank you for reading!