Upgrades

    From FOSSBotsWiki

    Guidance for low downtime updates.

    1. The non-active tools & bots servers should be updated first if possible. These can be rebooted freely.
    2. Decide whether to do a DC failover (only needed for larger upgrades with likely long downtime (Major Version Changes).
    3. If not doing a DC failover, decide whether to failover just traffic, this will make services a bit faster as there is a lot of latency between the DB.
    4. Databases should only be restarted carefully and will come back in read only

    Standard Procedure[edit | edit source]

    1. Upgrade & Reboot the passive bots & tools server
    2. Upgrade & reboot the primary bots server
    3. depool the passive DB from Phab
    4. stop mysql on the pasive DB, upgrade & reboot
    5. confirm the passive DB is replicating fine
    6. stop phd on the primary toolserver
    7. place phab & active DB in read only, repool the passive DB
    8. confirm replication is up to date and mark the active DB as down in Phab
    9. reboot the primary DB & Toolserver
    10. confirm replication is connected
    11. remove DB read only
    12. restore phab to RW with a master-replica setup.
    13. start phd on the primary toolserver

    Traffic Failover[edit | edit source]

    1. lower the load balancing session affinity
    2. Upgrade & Reboot the passive bots & tools server
    3. disable the primary bots server in the LB pool
    4. Upgrade & reboot the primary bots server
    5. enable the primary bots server in the LB pool
    6. depool the passive DB from Phab
    7. stop mysql on the pasive DB, upgrade & reboot
    8. confirm the passive DB is replicating fine
    9. stop phd on the primary toolserver
    10. place phab & active DB in read only, repool the passive DB
    11. disable the primary toolserver in the LB pool
    12. confirm replication is up to date and mark the active DB as down in Phab
    13. reboot the primary DB & Toolserver
    14. confirm replication is connected
    15. remove DB read only
    16. enable the primary toolserver in the LB pool
    17. restore phab to RW with a master-replica setup.
    18. start phd on both toolservers
    19. raise the load balancing session affinity

    DC Failover[edit | edit source]

    1. Upgrade & Reboot the passive bots & tools server
    2. depool the passive DB from Phab
    3. stop mysql on the pasive DB, upgrade & reboot
    4. confirm the passive DB is replicating fine
    5. repool the passive DB
    6. switch DNS to the passive DC
    7. disable services in the active DC
    8. run the backup scripts in the active DC
    9. stop phd on the primary toolserver
    10. restore backups for non replicated systems
    11. confirm replication is up to date and mark the active DB as down in Phab
    12. reverse the direction of replication
    13. promote the replica DB to master
    14. confirm replication status
    15. reboot the primary DB & Toolserver
    16. confirm replication is connected
    17. remove new master DB & Phab read only
    18. repeat with roles swapped to failback