Upgrades

Guidance for low downtime updates.

1. The non-active tools & bots servers should be updated first if possible. These can be rebooted freely.
2. Decide whether to do a DC failover (only needed for larger upgrades with likely long downtime (Major Version Changes).
3. If not doing a DC failover, decide whether to failover just traffic, this will make services a bit faster as there is a lot of latency between the DB.
4. Databases should only be restarted carefully and will come back in read only

Standard Procedure

1. Upgrade & Reboot the passive bots & tools server
2. Upgrade & reboot the primary bots server
3. depool the passive DB from Phab
4. stop mysql on the pasive DB, upgrade & reboot
5. confirm the passive DB is replicating fine
6. stop phd on the primary toolserver
7. place phab & active DB in read only, repool the passive DB
8. confirm replication is up to date and mark the active DB as down in Phab
9. reboot the primary DB & Toolserver
10. confirm replication is connected
11. remove DB read only
12. restore phab to RW with a master-replica setup.
13. start phd on the primary toolserver

Traffic Failover

1. lower the load balancing session affinity
2. Upgrade & Reboot the passive bots & tools server
3. disable the primary bots server in the LB pool
4. Upgrade & reboot the primary bots server
5. enable the primary bots server in the LB pool
6. depool the passive DB from Phab
7. stop mysql on the pasive DB, upgrade & reboot
8. confirm the passive DB is replicating fine
9. stop phd on the primary toolserver
10. place phab & active DB in read only, repool the passive DB
11. disable the primary toolserver in the LB pool
12. confirm replication is up to date and mark the active DB as down in Phab
13. reboot the primary DB & Toolserver
14. confirm replication is connected
15. remove DB read only
16. enable the primary toolserver in the LB pool
17. restore phab to RW with a master-replica setup.
18. start phd on both toolservers
19. raise the load balancing session affinity

DC Failover

1. Upgrade & Reboot the passive bots & tools server
2. depool the passive DB from Phab
3. stop mysql on the pasive DB, upgrade & reboot
4. confirm the passive DB is replicating fine
5. repool the passive DB
6. switch DNS to the passive DC
7. disable services in the active DC
8. run the backup scripts in the active DC
9. stop phd on the primary toolserver
10. restore backups for non replicated systems
11. confirm replication is up to date and mark the active DB as down in Phab
12. reverse the direction of replication
13. promote the replica DB to master
14. confirm replication status
15. reboot the primary DB & Toolserver
16. confirm replication is connected
17. remove new master DB & Phab read only
18. repeat with roles swapped to failback