Document - contributions welcome - for CLONE or UPGRADE to clusters

[RADSr]

I'd like to get feedback/additions/improvements on the below:

I'm working on capturing some notes as I'm in the midst of an upgrade adventure and think I may make a document for myself/Community/support to use when doing this kind of work.

Text from my initial plan below, but note that I'm combining two ideas because I've used them in conjunction - 1) upgrade and 2) cluster cloning

My initial ask is for things to include ( pitfalls, gotchas, tick lists, etc. ) when doing something like this.

1.  * not sure where this belongs
   1. Inspector Tool
      1. Support had me run this before upgrading PROD, but not the DEV clone.   
      2. Documentation says Sev 1 issues can interfere w/ the upgrade
      3. So probably resolve all Sev 1 issues before upgrading the clone
      4. Why the ease of ignoring this when updating from the cloud admin console?
          
2. Cluster cloning
   1. Turn off schedules!
      1. This is tremendously important because...
         1. Clones maintain not only the schedule but also the email settings so you'll get two of everything so 
   2. Change email settings
      1. It *looks* like the *default* tenant configurations update w/ the target cluster, but the existing tenant does not 
         1. I'm past the point where I can confirm this w/ my current project
   3. Re-add users in the cloud admin console
      1. Apparently if you have access to a cluster in cloud admin you do *not* automatically have access to it's clone - even if the clone target is an existing cluster to which you did have access.
   4. Data agent
      1. For a clone only I think you just need to regenerate the .auth file
      2. For an upgrade 
         1. Install new data agent 
         2. Generate new .auth file  ( do we need to create a new data agent data source in the IA UI? ) 
         3. Swear a lot when it doesn't work
         4. TBD
         5. If creating a new data source data agent update the data source data agent drop down 
   5. Kind of a cool note - it looks like the clone keeps the "last run" metadata so incremental loads will run from the last run of the clone source.
      1. Needs confirmation
         
         

=================== UPGRADE PLAN SAMPLE ==========================

 Upgrade Plan

1. Back up current DEV
   1. Tenant
   2. Parquet files for Incremental_Only_Snapshot_Tables schema
   3. Local data files
      
      
2. Delete all objects from DEV
   
   
3. B/U and Restore PROD to DEV
   1. Parquet files for Incremental_Only_Snapshot_Tables schema
   2. Local data files
      1.      At this point can we continue to run incremental loads or do we need an initial full load?
         1. How will this affect snapshots?
      2. 
         
         
   3. Upgrade DEV to 2023.4.0
      1. Test Incremental loads
      2. Test full loads
      3. A/B testing DEV to PROD
         
         
   4. Upgrade PROD
      
      
   5. Restore DEV  from B/U in step 1
      
      

Celebratory drinks and dinner!

[Tristan]

For the Inspector question, you should resolve all Sev 1 issues before upgrading for every instance that you upgrade.  I'm not sure how you are defining "clone" here.  Once a clone is created and you start to work with it in any way, it is no longer a clone.  If you truly want a clone, then you should wait to create it until after you have done the work of upgrading the environment from which it is to be cloned.

[Tristan]

Now that I read further, it looks like you are doing a manual cloning process by copying configuration down from Prod.  I assume this is so that you can upgrade a cluster that looks as much like prod as possible.

You could use Incorta Cloud's cloning capability to create a separate copy of the prod cluster that is not dev.  You would want to adjust any settings, like email, that you do not want to be active on that cluster.  At that point, you could clear the sev 1 issues on the new cluster if they were not already cleared in Prod and then test the upgrade.  Once you see that it works, you could drop this clone and proceed with upgrading Prod which should behave the same way.  I think you would still want to upgrade your Dev cluster first before taking this approach.

I may be missing the point here but I thought I would share this idea as an alternative.

[RADSr]

You've got the goal - what I did was request support clone the PROD environment into the DEV cluster ( retained the "-dev" cluster name ) and then asked that the "-dev" cluster be upgraded so I could compare DEV to PROD as identical code, different version clusters.  

I am kind of surprised that I can upgrade directly from cloud admin if the sev 1 issues are that much of a potential problem.   I'm not opposed to it, just surprised  😉   

It looks like creating the clone on my own is pretty straightforward but only for clusters I own ( makes sense kind of ) - do you know how creating a clone for upgrading/testing/etc. plays with licensing for customers?   

[Tristan]

Many licenses allow for up to 3 cluster so if a new cluster is meant to be permanent and stays within the 3 cluster limit, then all is good.  If it exceeds, then the cluster should be deleted as soon as its purpose is complete.  Basically, if a clone makes you exceed your licensing, you should delete it as soon as you can.  If not, it is OK to have it hang around.