Move Data from Dev/Test to Prod

[Noureldin]

Introduction

It is a best practice to test configuration in the development/test environment(s) before moving it to production.  In some cases, production data is loaded into the Dev/Test environments for validation purposes as well.  This article discusses how to move just your data from Dev/Test to Prod if going through the process of reloading it in the standard way is too time consuming or too disruptive.

Applicable To

The content of this article is valid for Incorta on premises versions 5.1.x and 5.2.x.

Let's Go

Moving all data to a new production environment

Here we want to move the whole Tenant plus data from Dev/Test to a newly provisioned Prod server.

1. Install Incorta on the appropriately provisioned new production server
2. Configure the Incorta loader and analytics services
3. Take a backup from the Dev/Test tenant
4. Import the tenant to the new production server.  You can change the name and path.

Note: Please be sure there is no running schema update or load while copying the data.  We highly recommend shutting down the services and taking a backup while the services are down.

     5. Copy the content of the tenant from Dev/Test into the new tenant directory created in step 4

     6. Run load from staging for the schemas keeping in mind any dependencies between schemas

     7. Run incremental loads

Moving One schema to new production

1. Take a schema export from Dev/Test
2. Take a backup of the schema parquet folder under <Dev_tenants>/source/<schema_name>
3. Import the new schema taken from step 1 to the production server
4. Copy the Schema folder from step 2 onto the production tenant under <Prod_tenants>/source/
5. Run staging load for the schema
6. Run incremental load

Moving One schema to new production (Run Incremental load with no staging)

1. Take Schema export from Dev/Test
2. Take a backup of the schema parquet folder under <Dev_tenants>/source/<schema_name>
3. Import the new schema taken from step 1 to the production server
4. Copy the Schema folders from step 2 onto the production tenant under <Prod_tenants>/source/
5. Select all columns from Dev/Test related to the schema

Spoiler

connect to metadata DB of Dev/Test
select * from FILES_VERSIONS where NAME like '<schema_name>%';

     6. Update the Production Metadata

Spoiler

connect to metadata DB of Dev/Test
update FILES_VERSIONS with the same output from step C.5

     7. Run incremental load

 

Sync One schema to existing production.

1. Take a schema export from Dev/Test
2. Take a backup from the schema folders under: 
   • <Dev_tenants>/source/<schema_name>
   • <Dev_tenants>/ddm/schemas/<schema_name
   • <Dev_tenants>/ddm/joins/<schema_name>*
3. Import the new schema taken from step 1 to the production server
4. Remove the existing schema folders on the existing production tenant under:
   • <Prod_tenants>/source/<schema_name
   • <Prod_tenants>/ddm/schemas/<schema_name
   • <Prod_tenants>/ddm/joins/<schema_name>*
5. Copy the Schema folders from step 2 onto the production tenant under 
   • <Prod_tenants>/source/<schema_name
   • <Prod_tenants>/ddm/schemas/<schema_name
   • <Prod_tenants>/ddm/joins/<schema_name>*
6. Remove all columns from the Production Metadata schema

Spoiler

connect to metadata DB of Prod
DELETE FROM FILES_VERSIONS WHERE NAME like '<schema_name>%';

     7. Run Stage load

Related Material

• Migrating Content Between Environments

[JoeM]

Great article @Noureldin!