FOSSBotsWiki:Server admin log

    From FOSSBotsWiki

    2022-11-24

    • <Sario> restarted matterbridge --16:47, 24 November 2022 (UTC)

    2022-11-07

    • <Sario> server restart on tools2002, bots 3001, db3001 --12:58, 7 November 2022 (UTC)
    • <Sario> apt upgrade on tools2002, tools 3001, db2002, db3001, bots3001 --12:57, 7 November 2022 (UTC)

    2022-11-03

    • <RhinosF1> take broken relay as an opportunity to reboot bots2002 --19:28, 3 November 2022 (UTC)

    2022-10-18

    • <Sariboo> update tzdata package on all hosts --14:02, 18 October 2022 (UTC)

    2022-10-10

    • <Sariboo> restart 'dbus' and 'unattended-upgrades' services on tools2001 --12:47, 10 October 2022 (UTC)
    • <Sariboo> restart 'dbus' and 'unattended-upgrades' services on db2002 and db3001 --12:45, 10 October 2022 (UTC)
    • <Sariboo> restart 'dbus' and 'unattended-upgrades' services on bots2002 --12:40, 10 October 2022 (UTC)
    • <Sariboo> restart 'dbus' and 'unattended-upgrades' services on bots3001 --12:37, 10 October 2022 (UTC)

    2022-10-03

    • <Sario> Update apt tzdata package on all hosts --12:41, 3 October 2022 (UTC)

    2022-09-25

    • <RhinosF1> moved phab & icinga from zoho to zepto by zoho --16:19, 25 September 2022 (UTC)
    • <Sario> apt upgrade (grub packages) on bots3001, tools3001, and db3001 --13:46, 25 September 2022 (UTC)
    • <Sario> apt upgrade (grub packages) on bots2002 and tools2002 --13:27, 25 September 2022 (UTC)

    2022-09-22

    • <Sario> spt upgrade on bots2002 and db2002 --20:33, 22 September 2022 (UTC)

    2022-09-20

    • <Sario> update db2002 and db3001 --21:09, 20 September 2022 (UTC)

    2022-09-18

    • <RhinosF1> upgrade phabricator to phorge --13:15, 18 September 2022 (UTC)
    • <RhinosF1> sudo -u www-data /var/phorge-deploy/phorge/bin/storage upgrade --force --13:09, 18 September 2022 (UTC)

    2022-09-17

    • <RhinosF1> all systems ok --13:11, 17 September 2022 (UTC)
    • <RhinosF1> empty the daemon_logevent table, restart phd, bleh --13:11, 17 September 2022 (UTC)
    • <RhinosF1> maint complete --13:04, 17 September 2022 (UTC)
    • <RhinosF1> mysql will be shut down for a bit, expect phab to flap --11:39, 17 September 2022 (UTC)
    • <RhinosF1> starting phab + db maint --11:39, 17 September 2022 (UTC)
    • <RhinosF1> rebooting bots2002 --11:36, 17 September 2022 (UTC)
    • <RhinosF1> reboot db3001 --11:33, 17 September 2022 (UTC)
    • <RhinosF1> upgrade all packages needing it and restart various services --11:32, 17 September 2022 (UTC)

    2022-09-09

    • <RhinosF1> deployments suspended following yesterday's news --06:54, 9 September 2022 (UTC)

    2022-09-06

    • <RhinosF1> world blown up (hopefully not - mariadb restarted to increase log retention) --18:52, 6 September 2022 (UTC)
    • <RhinosF1> quickly breaking the world --18:50, 6 September 2022 (UTC)
    • <Sario> restart bridgebot --15:33, 6 September 2022 (UTC)

    2022-08-26

    • <RhinosF1> add a MC for AS51559 NETINTERNET Netinternet Bilisim Teknolojileri AS --21:40, 26 August 2022 (UTC)
    • <RhinosF1> mysql to RO, restart mysql and put back RW. i hate phab. please be better. --09:30, 26 August 2022 (UTC)
    • <RhinosF1> stop slave; restart mysql on db3001, now OK for restarts --08:43, 26 August 2022 (UTC)
    • <RhinosF1> apt upgrade and then systemctl restart apache2.service rsyslog.service ssh.service unattended-upgrades.service on bots3001 --08:38, 26 August 2022 (UTC)
    • <RhinosF1> apt upgrade and then systemctl restart rsyslog.service ssh.service unattended-upgrades.service --08:38, 26 August 2022 (UTC)
    • <RhinosF1> apt upgrade and then systemctl restart apache2.service rsyslog.service ssh.service unattended-upgrades.service on tools3001 --08:37, 26 August 2022 (UTC)
    • <RhinosF1> apt upgrade and then systemctl restart rsyslog.service ssh.service unattended-upgrades.service on db3001 --08:37, 26 August 2022 (UTC)
    • <RhinosF1> systemctl restart apache2.service mirahezebotprodlibera.service mirahezebottestlibera.service rsyslog.service ssh.service streambot.service unattended-upgrades.service [bots2002] --08:34, 26 August 2022 (UTC)
    • <RhinosF1> rhinosf1@tools2002:~$ sudo -u www-data /var/phab-deploy/phabricator/bin/phd restart --08:34, 26 August 2022 (UTC)
    • <RhinosF1> systemctl restart apache2.service icingabot.service icinga2.service rsyslog.service ssh.service unattended-upgrades.service [tools2002] --08:32, 26 August 2022 (UTC)

    2022-08-25

    • <Sario> restart matterbridge --22:49, 25 August 2022 (UTC)
    • <RhinosF1> upgraded libxslt1.1:arm64 (1.1.34-4, 1.1.34-4+deb11u1) before that on tools2002, not showing anywhere else and needsrestart is clear --10:58, 25 August 2022 (UTC)
    • <RhinosF1> restart phd and apache on tools2002 --10:54, 25 August 2022 (UTC)

    2022-08-24

    • <RhinosF1> start full backups of / in a screen --13:59, 24 August 2022 (UTC)
    • <RhinosF1> transfer fossbots.org to CF account --08:08, 24 August 2022 (UTC)

    2022-08-23

    • <RhinosF1> tweaked some caching rules and added image resizing --15:00, 23 August 2022 (UTC)

    2022-08-22

    • <RhinosF1> remove macfan from alertops --10:31, 22 August 2022 (UTC)
    • <RhinosF1> completed a lot of maintenance to setup NeedsRestart and upgrade stuff, db2002 will be back shortly --09:09, 22 August 2022 (UTC)

    2022-08-21

    • <RhinosF1> restart 4 services on tools2002 --08:47, 21 August 2022 (UTC)

    2022-08-19

    • <RhinosF1> fix a few postfix permissions --15:08, 19 August 2022 (UTC)
    • <RhinosF1> restart postfix on tools2002 --14:57, 19 August 2022 (UTC)
    • <RhinosF1> fix nagios uid/gid to be consistent between bots2002 and tools2002 --14:55, 19 August 2022 (UTC)
    • <RhinosF1> reboot db3001, upgrade tzdata everywhere, upgrade goloang on bots2002 --14:14, 19 August 2022 (UTC)
    • <RhinosF1> configure nagios for boston nodes --10:18, 19 August 2022 (UTC)

    2022-08-17

    • <RhinosF1> db3001 upgrades complete --12:48, 17 August 2022 (UTC)
    • <RhinosF1> upgrade db3001 to match db2002 mariadb version --12:39, 17 August 2022 (UTC)
    • <RhinosF1> maint done --12:36, 17 August 2022 (UTC)
    • <RhinosF1> starting mariadb maint --12:32, 17 August 2022 (UTC)

    2022-08-12

    • <RhinosF1> rollback phab upgrades --13:37, 12 August 2022 (UTC)
    • <RhinosF1> upgrade phabricatorrr --13:02, 12 August 2022 (UTC)
    • <RhinosF1> add 3x VPS in OVH CA with equivalent spec to BOS --09:10, 12 August 2022 (UTC)

    2022-08-11

    • <RhinosF1> stop pt-heartbeat while i finish for the day, will conduct a better test next week when i've got time to monitor disk space most of the day. filing a task too --14:51, 11 August 2022 (UTC)
    • <RhinosF1> added pt-heartbeat running as root to test out, created it's tables & a user. password stored in /root. will add to puppet assuming it causes no mess. --13:37, 11 August 2022 (UTC)

    2022-08-09

    • <RhinosF1> update libgnutls30 unzip on tools2002 --22:05, 9 August 2022 (UTC)

    2022-08-08

    • <RhinosF1> same on tools2002 --14:20, 8 August 2022 (UTC)
    • <RhinosF1> update libtirpc on db2002 --10:14, 8 August 2022 (UTC)
    • <MacFan4000> update bot-admins group in cf access to use new org rather then old one --00:44, 8 August 2022 (UTC)

    2022-08-06

    • <RhinosF1> start some networking monitoring on db2002 that doesn't work yet --18:06, 6 August 2022 (UTC)

    2022-08-05

    • <RhinosF1> flip back to tools3001 with errors fixed, will try bots3001 after lunch --12:58, 5 August 2022 (UTC)
    • <RhinosF1> and flip back to web-london after tools3001 done - many errors --12:44, 5 August 2022 (UTC)
    • <RhinosF1> flip fossbots.org from web-london to web-boston lb and test each node --12:35, 5 August 2022 (UTC)

    2022-08-04

    • <RhinosF1> renumber sopel from id 1010 to 995 --08:58, 4 August 2022 (UTC)

    2022-08-02

    • <RhinosF1> reboot tools2002, db2002, repool tools and re-enable london, depool bots and then reboot bots-lon --14:27, 2 August 2022 (UTC)
    • <RhinosF1> disable london in phab config and depool tools2002 --14:17, 2 August 2022 (UTC)

    2022-08-01

    • <MacFan4000> updated cf access config for ops to use FOSSBots gh team rather then MirahezeBots --14:00, 1 August 2022 (UTC)
    • <MacFan4000> created icinga account for peer --13:52, 1 August 2022 (UTC)
    • <RhinosF1> invite peer to cf console and mail --12:56, 1 August 2022 (UTC)
    • <RhinosF1> invited peerau to github --12:49, 1 August 2022 (UTC)
    • <RhinosF1> renumbered streambot user on bots2002 --12:47, 1 August 2022 (UTC)

    2022-07-27

    • <Sario> sudo apt upgrade on bots2002 and db2002 --23:14, 27 July 2022 (UTC)

    2022-07-25

    • <Sario> sudo apt upgrade on tools2002 --15:54, 25 July 2022 (UTC)

    2022-07-11

    • <RhinosF1> and finally bots2002 --09:40, 11 July 2022 (UTC)
    • <RhinosF1> reboot tools2 & db2 --09:31, 11 July 2022 (UTC)
    • <RhinosF1> reboot and then restore --09:17, 11 July 2022 (UTC)
    • <RhinosF1> stop mysql on db3001 for reboot --09:16, 11 July 2022 (UTC)

    2022-07-10

    • <MacFan4000> update packages + kernal on tools3001 and reboot --22:58, 10 July 2022 (UTC)
    • <MacFan4000> update packages + kernal on bots3001 and reboot --22:47, 10 July 2022 (UTC)
    • <MacFan4000> updated matterbridge to v1.25.2 --22:31, 10 July 2022 (UTC)
    • <RhinosF1> also for a cc --18:39, 10 July 2022 (UTC)
    • <RhinosF1> tweaked WAF for an ASN --18:17, 10 July 2022 (UTC)

    2022-06-28

    • <MacFan4000> deleted doug's icinga account --15:57, 28 June 2022 (UTC)
    • <RhinosF1> delete Dougs forwarding rule in gmail --07:47, 28 June 2022 (UTC)
    • <RhinosF1> delete Reception123 & Doug from allowed forwarders --07:46, 28 June 2022 (UTC)
    • <RhinosF1> drop Doug mail account --07:44, 28 June 2022 (UTC)
    • <RhinosF1> remove dmehus from twitter --07:42, 28 June 2022 (UTC)
    • <RhinosF1> remove dmehus from @mirahezebots and @fossbots github --07:41, 28 June 2022 (UTC)

    2022-06-15

    • <RhinosF1> repooled *3* so CF detects when they're online - still some network issues --17:28, 15 June 2022 (UTC)
    • <RhinosF1> force rebooted bos2 VMs --17:00, 15 June 2022 (UTC)
    • <RhinosF1> depooled *3* from LB config --16:11, 15 June 2022 (UTC)

    2022-06-11

    • <RhinosF1> all services back to intended states --10:11, 11 June 2022 (UTC)
    • <RhinosF1> reboot db2002 & tools2002 --10:05, 11 June 2022 (UTC)
    • <RhinosF1> reboot db3001 --09:58, 11 June 2022 (UTC)
    • <RhinosF1> depool and stop mysql on db3001; update both dbs and tools2002 --09:57, 11 June 2022 (UTC)
    • <RhinosF1> upgrade and reboot bots2002 --09:49, 11 June 2022 (UTC)
    • <RhinosF1> upgrade and reboot tools3001 --09:48, 11 June 2022 (UTC)
    • <RhinosF1> reboot bots3001 for kernel updates --09:44, 11 June 2022 (UTC)
    • <RhinosF1> end production failover --09:41, 11 June 2022 (UTC)
    • <RhinosF1> (START) failover sopel-prod boston -> london --09:02, 11 June 2022 (UTC)

    2022-06-10

    • <RhinosF1> ignore, meant tools3 --20:44, 10 June 2022 (UTC)
    • <RhinosF1> reboot bots3 for kernel upgrade --20:43, 10 June 2022 (UTC)

    2022-06-01

    • <RhinosF1> db2002 is now replicating to db3001 --23:07, 1 June 2022 (UTC)
    • <MacFan4000> update golang on bots2002 --17:17, 1 June 2022 (UTC)

    2022-05-31

    • <MacFan4000> start bots2002 --16:09, 31 May 2022 (UTC)
    • <RhinosF1> failed web traffic to tools2002, removed bots2002 from rotation --15:29, 31 May 2022 (UTC)
    • <RhinosF1> failed Phabricator back to london. DB & phab now in london. Backups only run in london. Digests set in Boston. --15:27, 31 May 2022 (UTC)

    2022-05-30

    • <MacFan4000> switch phab dns to tools3001 --22:27, 30 May 2022 (UTC)

    2022-05-23

    • <RhinosF1> update libxml2 on db2002 & bots2002 --14:54, 23 May 2022 (UTC)

    2022-05-22

    • <RhinosF1> update tools2002 and restart apache2/phd --21:56, 22 May 2022 (UTC)

    2022-05-21

    • <RhinosF1> check /etc/mysql/mariadb.conf.d/custom.cnf before restart --13:23, 21 May 2022 (UTC)
    • <RhinosF1> move restart in RO to custom.cnf file as it didn't persist for some reason --13:23, 21 May 2022 (UTC)
    • <RhinosF1> update mariadb --13:05, 21 May 2022 (UTC)
    • <RhinosF1> enable IMUA on old domain --12:39, 21 May 2022 (UTC)
    • <RhinosF1> block python requests --07:44, 21 May 2022 (UTC)

    2022-05-20

    • <RhinosF1> tweak firewall on fossbots.org --05:59, 20 May 2022 (UTC)

    2022-05-19

    • <RhinosF1> blocked an IP & re-enabled OVH screening --22:19, 19 May 2022 (UTC)
    • <MacFan4000> restart phd --21:47, 19 May 2022 (UTC)

    2022-05-18

    • <MacFan4000> restart icingabot --22:19, 18 May 2022 (UTC)

    2022-05-15

    • <RhinosF1> pausing phd for maintenance --14:44, 15 May 2022 (UTC)
    • <RhinosF1> upgrade s3cmd via PyPi to 2.2.0 --14:32, 15 May 2022 (UTC)

    2022-05-14

    • <RhinosF1> that's for dev --09:54, 14 May 2022 (UTC)
    • <RhinosF1> install importlib-metadata in 3.9 venv --09:53, 14 May 2022 (UTC)

    2022-05-13

    • <RhinosF1> rebuilding dev packages due to permissions issues --15:11, 13 May 2022 (UTC)

    2022-05-10

    • <RhinosF1> T276 - mariadb now starts in read only --15:16, 10 May 2022 (UTC)
    • <RhinosF1> restarting maridb to pick up new config --15:15, 10 May 2022 (UTC)
    • <RhinosF1> start phd on tools2002 --15:12, 10 May 2022 (UTC)
    • <RhinosF1> reboot tools 2002 --15:05, 10 May 2022 (UTC)
    • <RhinosF1> reboot db2002 --15:02, 10 May 2022 (UTC)
    • <RhinosF1> phab+mysql to RO --15:02, 10 May 2022 (UTC)
    • <RhinosF1> reboot bots2002 --14:59, 10 May 2022 (UTC)

    2022-05-09

    • <MacFan4000> added mail alias for zppix --14:54, 9 May 2022 (UTC)
    • <MacFan4000> allowed zppix emails in cf access --14:39, 9 May 2022 (UTC)
    • <MacFan4000> invited Zppix to GH org granting only read access --14:35, 9 May 2022 (UTC)
    • <MacFan4000> update icingabot config and restat --14:23, 9 May 2022 (UTC)

    2022-05-08

    • <RhinosF1> backing up db2002 --15:31, 8 May 2022 (UTC)

    2022-05-06

    • <MacFan4000> <RhinosF1> updated domain to Sario's info rather than Owen --01:39, 6 May 2022 (UTC)
    • <MacFan4000> add page rule for phab to CF --01:39, 6 May 2022 (UTC)
    • <Voidwalker> test --01:38, 6 May 2022 (UTC)

    2022-05-05

    • <MacFan4000> notifico switch channels --12:16, 5 May 2022 (UTC)

    2022-04-27

    • <RhinosF1> dropped 'reception' on db/tools/bots2002 --21:42, 27 April 2022 (UTC)
    • <RhinosF1> drop Reception's ssh key from sudo rm /etc/ssh/userkeys/reception --21:38, 27 April 2022 (UTC)

    2022-04-03

    • <MacFan4000> sudo shutdown -h now on bots1 --00:25, 3 April 2022 (UTC)

    2022-04-01

    • <RhinosF1> upgrade zlib1g zlib1g-dev on db2002 --15:30, 1 April 2022 (UTC)

    2022-03-31

    • <RhinosF1> drop some 3.7 builds - I wish this was managed better by CI --21:11, 31 March 2022 (UTC)

    2022-03-28

    • <RhinosF1> added sario@ as an email for fossbots.org --18:19, 28 March 2022 (UTC)
    • <Sario> run sudo apt update && sudo apt upgrade -y on all servers --10:56, 28 March 2022 (UTC)

    2022-03-26

    • <Sario> sudo apt update && sudo apt upgrade -y on bot2002, tools2002, and db2002 --17:22, 26 March 2022 (UTC)

    2022-03-22

    • <MacFan4000> restart bridgebot to fix nick --15:27, 22 March 2022 (UTC)
    • <Sario> sudo apt update && sudo apt upgrade -y on bots2002 --15:13, 22 March 2022 (UTC)

    2022-03-20

    • <MacFan4000> verified ownership for fossbots.org for google search console --02:47, 20 March 2022 (UTC)

    2022-03-19

    • <RhinosF1> install fossbots.org origin keys --23:13, 19 March 2022 (UTC)

    2022-03-08

    • <Sario> apt upgrade on db2002 --18:47, 8 March 2022 (UTC)
    • <MacFan4000> restart phd --12:55, 8 March 2022 (UTC)

    2022-03-07

    • <Sario> apt upgrade on bots2002 --20:13, 7 March 2022 (UTC)
    • <Sario> apt upgrade on tools2002 --20:08, 7 March 2022 (UTC)

    2022-03-05

    • <RhinosF1> fixed CORS issue on phab-storage by changing "*" to * --23:18, 5 March 2022 (UTC)
    • <RhinosF1> move MX records to *.mx.cloudflare.net --18:43, 5 March 2022 (UTC)
    • <RhinosF1> add easydmarc to dmarc report urls --10:16, 5 March 2022 (UTC)
    • <MacFan4000> restart phd --04:00, 5 March 2022 (UTC)

    2022-03-03

    • <RhinosF1> tools2002 is down --18:40, 3 March 2022 (UTC)

    2022-03-02

    • <MacFan4000> update prod sopel and restart --22:52, 2 March 2022 (UTC)

    2022-02-28

    • <MacFan4000> macfan@bots2002:/srv/sopelbots$ sudo chown -R sopel:sopel prod --16:32, 28 February 2022 (UTC)
    • <MacFan4000> macfan@bots2002:/srv/sopelbots$ sudo chown -R sopel:sopel test --16:32, 28 February 2022 (UTC)

    2022-02-26

    • <Sario> apply security apt updates on tools2002 --18:37, 26 February 2022 (UTC)
    • <Sario> apply security apt updates on bots2002 --18:35, 26 February 2022 (UTC)

    2022-02-21

    • <RhinosF1> deleted bots1 from hund, switched cloudflare & closed maint incident --17:10, 21 February 2022 (UTC)
    • <MacFan4000> switch dns for sopel --16:42, 21 February 2022 (UTC)
    • <RhinosF1> on tools2002 --14:57, 21 February 2022 (UTC)
    • <RhinosF1> sudo systemctl reset-failed --14:56, 21 February 2022 (UTC)
    • <RhinosF1> added new VMs to CF --14:51, 21 February 2022 (UTC)
    • <RhinosF1> shutdown tools1 & db1 --14:44, 21 February 2022 (UTC)

    2022-02-20

    • <RhinosF1> purging cloudflare cache for icinga --19:31, 20 February 2022 (UTC)
    • <MacFan4000> switch dns for icinga --19:11, 20 February 2022 (UTC)
    • <MacFan4000> switched dns for phab --17:03, 20 February 2022 (UTC)
    • <MacFan4000> switch dns for webmail to tools2002 --16:26, 20 February 2022 (UTC)
    • <MacFan4000> phab readonly and SET GLOBAL read_only = 1; --15:44, 20 February 2022 (UTC)
    • <MacFan4000> switched phabdigests dns to point to tools2002 --13:32, 20 February 2022 (UTC)
    • <MacFan4000> copied phabdigests config files from tools1 to tools2002 --13:31, 20 February 2022 (UTC)

    2022-02-12

    • <MacFan4000> apt upgrade on tools2002 --17:21, 12 February 2022 (UTC)

    2022-02-06

    • <MacFan4000> intentionally messed up permissions for streambot db on bots1 to stop it from running --18:25, 6 February 2022 (UTC)

    2022-02-04

    • <MacFan4000> apt upgrade on bots2002 and db2002 --12:46, 4 February 2022 (UTC)

    2022-01-30

    • <Sario> restart Streambot --21:51, 30 January 2022 (UTC)
    • <RhinosF1> disable https://phab.mirahezebots.org/people/manage/25/#54 --21:37, 30 January 2022 (UTC)
    • <dmehus> Restarted StreamBotMH --19:06, 30 January 2022 (UTC)
    • <MacFan4000> added new servers to hund --15:48, 30 January 2022 (UTC)
    • <MacFan4000> added bots2002 to the pool and verified that everything is working --15:12, 30 January 2022 (UTC)
    • <MacFan4000> undo rollback --14:41, 30 January 2022 (UTC)
    • <MacFan4000> rolled back --14:06, 30 January 2022 (UTC)
    • <MacFan4000> forcing main site through tools2002 again for more testing --13:49, 30 January 2022 (UTC)

    2022-01-29

    • <MacFan4000> remove tools1 from rotation --23:01, 29 January 2022 (UTC)
    • <MacFan4000> added bots2002 and tools2002 to web rotation --23:00, 29 January 2022 (UTC)
    • <MacFan4000> restart icingabot --18:00, 29 January 2022 (UTC)
    • <MacFan4000> installing puppet on new servers --17:05, 29 January 2022 (UTC)
    • <MacFan4000> added dns records for the new servers --16:33, 29 January 2022 (UTC)
    • <MacFan4000> created 3 new v1.medium aarch vms --13:58, 29 January 2022 (UTC)

    2022-01-16

    • <MacFan4000> updated sygnal contacts to remove ndk --01:53, 16 January 2022 (UTC)

    2022-01-15

    • <MacFan4000> tweak a firewall rule --16:28, 15 January 2022 (UTC)
    • <MacFan4000> disabled browser integrity check --03:23, 15 January 2022 (UTC)

    2022-01-11

    • <RhinosF1> sudo ufw reject from 5.197.246.6 to any && sudo ufw reject from 176.111.173.226 to any --17:12, 11 January 2022 (UTC)
    • <MacFan4000> updated sygnal chan contacts --15:45, 11 January 2022 (UTC)

    2022-01-09

    • <dmehus> Restarted prod and test MirahezeBot --05:37, 9 January 2022 (UTC)

    2022-01-06

    • <MacFan4000> restart -test --22:59, 6 January 2022 (UTC)
    • <MacFan4000> tweaked release branch protection rule to require conversation resolution and two approving reviews per the deployment policy --22:26, 6 January 2022 (UTC)
    • <MacFan4000> added protection rule on dev branch requiring build checks to pass --21:26, 6 January 2022 (UTC)

    2022-01-04

    • <RhinosF1> cleaned conflicts in /var/flask on tools1 to fix puppet --15:44, 4 January 2022 (UTC)
    • <RhinosF1> updated all hosts --15:40, 4 January 2022 (UTC)

    2022-01-03

    • <MacFan4000> restart both bots --18:59, 3 January 2022 (UTC)
    • <MacFan4000> restart icingabot --18:53, 3 January 2022 (UTC)

    2022-01-02

    • <Sario> restart StreamBot --20:25, 2 January 2022 (UTC)

    2022-01-01

    • <RhinosF1> killed snoonet new year party bot running in my screen --12:43, 1 January 2022 (UTC)

    2021-12-18

    • <RhinosF1> start snoonet new year celebration bot in screen --14:48, 18 December 2021 (UTC)

    2021-12-16

    • <RhinosF1> depool 82.145.63.172 (tools1) from core web dns --20:46, 16 December 2021 (UTC)
    • <MacFan4000> restart mh-discord --00:46, 16 December 2021 (UTC)

    2021-12-11

    • <RhinosF1> enable page shield --18:07, 11 December 2021 (UTC)
    • <RhinosF1> enable cloudflare security insights --18:04, 11 December 2021 (UTC)

    2021-12-06

    • <Sario> Restart prod --19:21, 6 December 2021 (UTC)

    2021-12-01

    • <RhinosF1> apt-get update && upgrade across cluster --22:51, 1 December 2021 (UTC)

    2021-11-20

    2021-11-15

    • <Sario> deleted fh freenode relay bot --22:09, 15 November 2021 (UTC)

    2021-11-14

    • <dmehus> Restarted StreamBotMH --19:36, 14 November 2021 (UTC)
    • <dmehus> Restarted prod and test MirahezeBot --18:42, 14 November 2021 (UTC)

    2021-11-09

    • <MacFan4000> restart beta bot --15:15, 9 November 2021 (UTC)
    • <MacFan4000> restart prod bot for update --15:06, 9 November 2021 (UTC)

    2021-11-04

    • <MacFan4000> restart icingabot --16:20, 4 November 2021 (UTC)

    2021-10-25

    • <Spookreeeno> raise security level on bots domains --09:40, 25 October 2021 (UTC)
    • <Sario> restart prod --02:08, 25 October 2021 (UTC)

    2021-10-18

    • <MacFan4000> restart icingabot --13:01, 18 October 2021 (UTC)

    2021-10-11

    • <MacFan4000> created icinga account for dmehus --15:22, 11 October 2021 (UTC)
    • <RhinosF1> all hosts upgraded --14:16, 11 October 2021 (UTC)
    • <RhinosF1> reboot bots1 --14:16, 11 October 2021 (UTC)
    • <RhinosF1> updating hosts to 10.11 --14:08, 11 October 2021 (UTC)
    • <MacFan4000> invited dmehus to GitHub org --02:38, 11 October 2021 (UTC)
    • <MacFan4000> Notice: /Stage[main]/Users/Users::Hashuser[dmehus]/Users::User[dmehus]/User[dmehus]/ensure: created --02:36, 11 October 2021 (UTC)

    2021-10-10

    • <RhinosF1> reduce firewall to normal --21:24, 10 October 2021 (UTC)
    • <RhinosF1> tweak firewall settings --13:02, 10 October 2021 (UTC)

    2021-10-06

    • <MacFan4000> removed Dougs icinga account --19:01, 6 October 2021 (UTC)
    • <RhinosF1> drop dmehus email --18:21, 6 October 2021 (UTC)
    • <RhinosF1> remove doug from git --18:13, 6 October 2021 (UTC)
    • <RhinosF1> disable shell for dmehus --17:57, 6 October 2021 (UTC)

    2021-09-30

    • <RhinosF1> and. again more updates --16:24, 30 September 2021 (UTC)
    • <RhinosF1> reboot relays for updates --16:12, 30 September 2021 (UTC)

    2021-09-12

    • <RhinosF1> restart prod for sopel update --09:09, 12 September 2021 (UTC)

    2021-09-10

    • <RhinosF1> made MirahezeBot cycle memipedia to reset modes so everyone wasn't locked out --06:36, 10 September 2021 (UTC)

    2021-09-07

    • <RhinosF1> added dmehus to bot-roots on GitHub --17:01, 7 September 2021 (UTC)
    • <RhinosF1> apt-get upgrade && sudo apt autoremove --14:32, 7 September 2021 (UTC)
    • <RhinosF1> apt-get update on all hosts --14:30, 7 September 2021 (UTC)
    • <MacFan4000> update channel contacts for -sre --01:02, 7 September 2021 (UTC)

    2021-08-27

    • <RhinosF1> shift web traffic towards tools --07:50, 27 August 2021 (UTC)

    2021-08-26

    • <MacFan4000> restart dev bot --15:23, 26 August 2021 (UTC)
    • <MacFan4000> restart prod bot --13:14, 26 August 2021 (UTC)

    2021-08-24

    • <RhinosF1> pool tools1 back into web --13:37, 24 August 2021 (UTC)

    2021-08-21

    • <MacFan4000> added db1 to Hund as a ping check --12:46, 21 August 2021 (UTC)
    • <RhinosF1> convert hund checks to ICMP --12:27, 21 August 2021 (UTC)

    2021-08-20

    • <RhinosF1> depool tools1 for web rotation --07:08, 20 August 2021 (UTC)

    2021-08-19

    • <RhinosF1> remove test from hund - MacFan --12:27, 19 August 2021 (UTC)

    2021-08-18

    • <RhinosF1> actually delete test2001 --13:04, 18 August 2021 (UTC)
    • <RhinosF1> 2001* --12:57, 18 August 2021 (UTC)
    • <RhinosF1> downtime test2002 --12:56, 18 August 2021 (UTC)
    • <RhinosF1> remove test2001 from Argo --12:53, 18 August 2021 (UTC)
    • <RhinosF1> delete test2001 from DNS --12:52, 18 August 2021 (UTC)

    2021-08-06

    • <Sario> restarting discord-IRC relays --15:32, 6 August 2021 (UTC)

    2021-08-04

    • <RhinosF1> require TLS 1.3+ for Cloudflare requests, virtually no use of TLS 1.2 --10:32, 4 August 2021 (UTC)

    2021-08-02

    • <RhinosF1> restart sshd on bots1 --15:22, 2 August 2021 (UTC)
    • <MacFan4000> restart icingabot --01:04, 2 August 2021 (UTC)

    2021-07-30

    • <MacFan4000> restarted mysql --14:22, 30 July 2021 (UTC)

    2021-07-28

    • <RhinosF1> creating backup of mysql all dbs --17:20, 28 July 2021 (UTC)
    • <RhinosF1> restarted mysql on db1 --17:19, 28 July 2021 (UTC)

    2021-07-26

    • <MacFan4000> restart beta again --15:30, 26 July 2021 (UTC)
    • <MacFan4000> restart beta --15:26, 26 July 2021 (UTC)

    2021-07-23

    • <MacFan4000> restart beta again --16:46, 23 July 2021 (UTC)
    • <Sario> restarting beta --16:44, 23 July 2021 (UTC)
    • <Sario> restarting prod --16:42, 23 July 2021 (UTC)

    2021-07-21

    • <RhinosF1> clean my tests up --14:27, 21 July 2021 (UTC)
    • <RhinosF1> testing setup of a toolserver --14:07, 21 July 2021 (UTC)

    2021-07-20

    • <RhinosF1> upgrade test2001, run puppet everywhere, ssh is happy --21:54, 20 July 2021 (UTC)
    • <RhinosF1> rhinosf1@bots1:~$ sudo -u sopel /srv/sopelbots/devvenv/bin/pip3 install -U "idna<4,>=3.2" --21:35, 20 July 2021 (UTC)
    • <Sario> set phab read/write --14:47, 20 July 2021 (UTC)
    • <Sario> set MariaDB read/write --14:46, 20 July 2021 (UTC)
    • <Sario> reboot db1 --14:45, 20 July 2021 (UTC)
    • <Sario> set MariaDB read only --14:35, 20 July 2021 (UTC)
    • <Sario> Set phab read only --14:29, 20 July 2021 (UTC)
    • <RhinosF1> moved Sario to github admins group --14:05, 20 July 2021 (UTC)
    • <MacFan4000> made Sario a GH org owner --14:03, 20 July 2021 (UTC)
    • <MacFan4000> removed Sario from icinga sopel-admins and added to admins --14:03, 20 July 2021 (UTC)

    2021-07-16

    • <MacFan4000> restart streambot --13:23, 16 July 2021 (UTC)

    2021-07-15

    • <MacFan4000> rolled back icinga/icingaweb update --22:04, 15 July 2021 (UTC)
    • <MacFan4000> upgrading icinga2 and icingaweb2 on tools1 --21:14, 15 July 2021 (UTC)
    • <RhinosF1> audited icinga CVEs and confirmed no risk --21:11, 15 July 2021 (UTC)

    2021-07-14

    • <RhinosF1> phab+mysql to RW --21:55, 14 July 2021 (UTC)
    • <RhinosF1> upgrade db1, no reboot required --21:53, 14 July 2021 (UTC)
    • <RhinosF1> mysql to RO --21:51, 14 July 2021 (UTC)
    • <RhinosF1> upgrade tools1, no reboot required --21:48, 14 July 2021 (UTC)
    • <RhinosF1> phab RO --21:44, 14 July 2021 (UTC)
    • <RhinosF1> upgrade test2001, no reboot needed --21:41, 14 July 2021 (UTC)
    • <RhinosF1> reboot bots1 --21:39, 14 July 2021 (UTC)
    • <RhinosF1> update a lot on bots1 --14:28, 14 July 2021 (UTC)
    • <RhinosF1> update prod --14:26, 14 July 2021 (UTC)

    2021-07-12

    • <RhinosF1> rhinosf1@bots1:~$ sudo service 'mirahezebot*libera' restart --12:46, 12 July 2021 (UTC)
    • <RhinosF1> update sopel on both venv's --12:45, 12 July 2021 (UTC)

    2021-07-11

    • <RhinosF1> restart beta --20:14, 11 July 2021 (UTC)

    2021-07-08

    • <MacFan4000> cleared /var/log/daemon.log --15:12, 8 July 2021 (UTC)
    • <MacFan4000> removed old freenode sopel logs --14:59, 8 July 2021 (UTC)
    • <MacFan4000> cleared syslog to regain space --12:30, 8 July 2021 (UTC)

    2021-07-07

    • <MacFan4000> removing sopel prod logs from July 2020 --23:38, 7 July 2021 (UTC)

    2021-07-03

    • <RhinosF1> disabled CNAME flattening outside of root on non proxies --20:12, 3 July 2021 (UTC)

    2021-06-23

    • <MacFan4000> restart -test for update --19:10, 23 June 2021 (UTC)
    • <MacFan4000> restart -test for update --18:44, 23 June 2021 (UTC)
    • <MacFan4000> restart prod for update --18:40, 23 June 2021 (UTC)

    2021-06-22

    • <RhinosF1> added moderation change (see audit log) due to trust & safety concerns. Direct all enquires to me. --22:27, 22 June 2021 (UTC)
    • <MacFan4000> reboot test2001 to see if hostname will stick --12:04, 22 June 2021 (UTC)

    2021-06-21

    • <MacFan4000> update rules set channel='##Sario' where channel='##sario'; --15:41, 21 June 2021 (UTC)

    2021-06-20

    • <RhinosF1> updating sopel to 7.1.1 --12:04, 20 June 2021 (UTC)
    • <RhinosF1> destory archived repos --11:56, 20 June 2021 (UTC)

    2021-06-17

    • <RhinosF1> updated health check data --21:48, 17 June 2021 (UTC)
    • <RhinosF1> added test2001 to main rotation --21:40, 17 June 2021 (UTC)

    2021-06-16

    • <MacFan4000> testing upgrading puppet on test2001 --23:27, 16 June 2021 (UTC)
    • <MacFan4000> updated branch protection rule on core repo to account for new ci check names --17:02, 16 June 2021 (UTC)
    • <MacFan4000> archived the pingpong repo on github --16:49, 16 June 2021 (UTC)
    • <MacFan4000> restart prod --16:46, 16 June 2021 (UTC)

    2021-06-15

    • <RhinosF1> restart icinga2 --13:53, 15 June 2021 (UTC)

    2021-06-14

    • <RhinosF1> apt-get upgrade on tools1 --15:29, 14 June 2021 (UTC)

    2021-06-13

    • <MacFan4000> dropped freenode account for FosshostRelay --21:45, 13 June 2021 (UTC)
    • <RhinosF1> add spelling mistake to fhfreenode config to crash it at Fosshost request due to issue --11:46, 13 June 2021 (UTC)

    2021-06-05

    • <RhinosF1> test2001.mhbots.org exists with dodgy /etc/hostname --16:54, 5 June 2021 (UTC)
    • <RhinosF1> create mirahezebots-fhc-test2001 --15:57, 5 June 2021 (UTC)

    2021-06-03

    • <Sario> stop discordircfh --19:39, 3 June 2021 (UTC)
    • <Sario> restart discordircfhfreenode relay --19:07, 3 June 2021 (UTC)
    • <RhinosF1> sudo systemctl reset-failed --16:59, 3 June 2021 (UTC)

    2021-06-02

    • <MacFan4000> restart icingabot --01:50, 2 June 2021 (UTC)
    • <MacFan4000> updated phap --00:59, 2 June 2021 (UTC)

    2021-05-31

    • <MacFan4000> deploying sopel 7.1 on prod --21:40, 31 May 2021 (UTC)

    2021-05-30

    • <MacFan4000> restart icingabot --16:06, 30 May 2021 (UTC)
    • <RhinosF1> clear phab cache, update done --16:01, 30 May 2021 (UTC)
    • <RhinosF1> manually deploy apache changes --15:57, 30 May 2021 (UTC)
    • <RhinosF1> reboot tools1 --15:38, 30 May 2021 (UTC)
    • <RhinosF1> restart apache2 on tools1 --15:21, 30 May 2021 (UTC)
    • <RhinosF1> stopped phd --15:19, 30 May 2021 (UTC)

    2021-05-28

    • <MacFan4000> restart icingabot --00:19, 28 May 2021 (UTC)

    2021-05-26

    • <MacFan4000> restart mhdiscordlibera --13:12, 26 May 2021 (UTC)
    • <MacFan4000> dropped freenode accounts for MirahezeBot and mh-discord --04:18, 26 May 2021 (UTC)

    2021-05-24

    • <Sario> restart test to update chanops --01:16, 24 May 2021 (UTC)
    • <Sario> Restart prod to update chanops --01:16, 24 May 2021 (UTC)

    2021-05-22

    • <Sario> Restart streambot --21:22, 22 May 2021 (UTC)
    • <Sario> restart mhdiscordlibera --18:52, 22 May 2021 (UTC)
    • <Sario> restart streambot --18:44, 22 May 2021 (UTC)
    • <RhinosF1> <reception> restarted StreamBot a while back due to net issues --18:40, 22 May 2021 (UTC)

    2021-05-21

    • <RhinosF1> Sygnal had conflicts (See rhinosf1@bots1:/srv/sopelbots$ sudo cp -r AntiSpamSopel AntiSpamSopel-202105211858UK) --18:00, 21 May 2021 (UTC)
    • <RhinosF1> created 15328 -rw-r--r-- 1 root root 15692766 May 21 17:39 /srv/sopelbots/testfreenode.tar.gz --17:41, 21 May 2021 (UTC)
    • <RhinosF1> back default.* and logs/default.* / freenode stuff up to test/freenode --17:36, 21 May 2021 (UTC)
    • <RhinosF1> rhinosf1@bots1:~$ unalias puppet-run --17:21, 21 May 2021 (UTC)
    • <RhinosF1> rhinosf1@bots1:~$ sudo rm -f /home/macfan/puppet-run.sh --17:20, 21 May 2021 (UTC)
    • <RhinosF1> reinstall python --17:13, 21 May 2021 (UTC)

    2021-05-20

    • <MacFan4000> got the bot accounts cloaked under me on libera, and reenabled sygnal --13:09, 20 May 2021 (UTC)

    2021-05-12

    • <MacFan4000> updated ping lists for #miraheze and #miraheze-overflow, added ping lists for #miraheze-sre, #miraheze-ops and #miraheze-offtopic --15:57, 12 May 2021 (UTC)
    • <MacFan4000> archived bots-pip-auto on GitHub --15:26, 12 May 2021 (UTC)
    • <MacFan4000> removed more unneeded sopel packages from tools1 --15:01, 12 May 2021 (UTC)
    • <MacFan4000> removing channelmgnt from tools1 as it shouldn't be there --14:55, 12 May 2021 (UTC)

    2021-05-10

    • <RhinosF1> git reset /var/flask --21:24, 10 May 2021 (UTC)
    • <RhinosF1> various apt upgrades everywhere --21:01, 10 May 2021 (UTC)

    2021-05-09

    • <Sario> restart -test --15:22, 9 May 2021 (UTC)

    2021-05-05

    • <RhinosF1> reduce firewall rules --20:29, 5 May 2021 (UTC)
    • <MacFan4000> whitelisted Hund ipv6 addresses in cloudflare --19:01, 5 May 2021 (UTC)

    2021-05-04

    • <MacFan4000> whitelisting bots1/db1 ip addresses in cloudflare --14:59, 4 May 2021 (UTC)

    2021-05-03

    • <MacFan4000> manually run git pull in /var/flask on tools1 --11:43, 3 May 2021 (UTC)

    2021-05-01

    • <MacFan4000> restart icingabot --22:32, 1 May 2021 (UTC)
    • <RhinosF1> allow semrush --21:52, 1 May 2021 (UTC)
    • <RhinosF1> fix bingbot exemption --21:45, 1 May 2021 (UTC)
    • <RhinosF1> allow google + bingbot access --21:17, 1 May 2021 (UTC)
    • <RhinosF1> tighten firewall to target attack more closely --20:33, 1 May 2021 (UTC)
    • <RhinosF1> challenge automated traffic --19:55, 1 May 2021 (UTC)

    2021-04-30

    • <RhinosF1> JS Challenge 2611 BELNET -- known use for activity in violation of privacy standards --21:38, 30 April 2021 (UTC)

    2021-04-20

    • <RhinosF1> delete rsyslog and restart it's service --15:38, 20 April 2021 (UTC)

    2021-04-19

    • <MacFan4000> after much frustration, postfix is now working to relay mail sent on tools1 to fosshost mail --18:04, 19 April 2021 (UTC)

    2021-04-18

    • <MacFan4000> updated the hostnames last night to use server.mirahezebots.org format --12:21, 18 April 2021 (UTC)

    2021-04-17

    • <MacFan4000> added Sario to bots-pip-auto (write access) --20:08, 17 April 2021 (UTC)
    • <MacFan4000> granted Sario write access to the mirahezebots.org repo --20:01, 17 April 2021 (UTC)
    • <MacFan4000> added icinga account for dmehus --16:25, 17 April 2021 (UTC)
    • <RhinosF1> rebuild streambot as system user --15:21, 17 April 2021 (UTC)
    • <RhinosF1> updates done --13:42, 17 April 2021 (UTC)
    • <RhinosF1> kernal update + reboot db1/tools1 - mysql/phab RO --13:33, 17 April 2021 (UTC)
    • <RhinosF1> reboot bots1 --13:27, 17 April 2021 (UTC)
    • <RhinosF1> updating kernel on bots1 --13:25, 17 April 2021 (UTC)

    2021-04-16

    • <RhinosF1> beta up to date --15:50, 16 April 2021 (UTC)
    • <RhinosF1> reclone devcode/* --15:38, 16 April 2021 (UTC)
    • <RhinosF1> add permissions policy header worker and challenge some traffic due to ongoing attack --07:24, 16 April 2021 (UTC)

    2021-04-09

    • <RhinosF1> switched MirahezeB twitter to black in response to the death of HRH Prince Philip --11:35, 9 April 2021 (UTC)

    2021-04-08

    2021-04-07

    • <RhinosF1> disable puppet cron --12:31, 7 April 2021 (UTC)

    2021-04-04

    • <RhinosF1> rhinosf1@100:/$ sudo useradd --system --home-dir /discord-irc relays --14:09, 4 April 2021 (UTC)

    2021-04-03

    • <RhinosF1> chown+chmod /srv/sopelbots --12:13, 3 April 2021 (UTC)
    • <RhinosF1> rhinosf1@100:~$ sudo useradd --system --home-dir /srv/sopelbots sopel --12:07, 3 April 2021 (UTC)
    • <RhinosF1> restart to update sopel --11:53, 3 April 2021 (UTC)

    2021-04-01

    • <MacFan4000> SigmaBot has been shut down for good --17:49, 1 April 2021 (UTC)
    • <MacFan4000> restart prod --17:28, 1 April 2021 (UTC)
    • <MacFan4000> stop prod for config changes --17:26, 1 April 2021 (UTC)
    • <MacFan4000> restarting both bots --17:14, 1 April 2021 (UTC)

    2021-03-31

    • <MacFan4000> update curl on bots1/tools1 --18:33, 31 March 2021 (UTC)
    • <RhinosF1> subscribed staff@ to DSA --17:24, 31 March 2021 (UTC)

    2021-03-29

    • <Sario> Restart test --18:08, 29 March 2021 (UTC)
    • <RhinosF1> move sopelprod under venv --13:56, 29 March 2021 (UTC)
    • <RhinosF1> rhinosf1@100:~$ python3 -m venv /srv/sopelbots/prodvenv/ --13:45, 29 March 2021 (UTC)

    2021-03-28

    • <RhinosF1> publish 'dev' branch rather than 'Testing' for sopel --21:35, 28 March 2021 (UTC)
    • <RhinosF1> update MirahezeBotBeta --09:37, 28 March 2021 (UTC)

    2021-03-27

    • <MacFan4000> restart icingabot --13:56, 27 March 2021 (UTC)
    • <MacFan4000> stop icingabot in order to login and update email --13:39, 27 March 2021 (UTC)
    • <MacFan4000> restart -test --13:07, 27 March 2021 (UTC)
    • <MacFan4000> added staff@ alias with same recipients as bots@miraheze.org --00:35, 27 March 2021 (UTC)
    • <MacFan4000> added aliases for all team members --00:28, 27 March 2021 (UTC)

    2021-03-26

    • <MacFan4000> generated ssl for mail.mirahezebots.org --23:32, 26 March 2021 (UTC)
    • <MacFan4000> adding dns records for mail sub domain as well as spf, dmarc, and dkim --23:30, 26 March 2021 (UTC)
    • <MacFan4000> restart phab daemons --18:03, 26 March 2021 (UTC)
    • <MacFan4000> restarted -test --17:22, 26 March 2021 (UTC)
    • <MacFan4000> restarted -test --17:04, 26 March 2021 (UTC)

    2021-03-25

    • <RhinosF1> phab was out of RO 20 minutes ago --22:32, 25 March 2021 (UTC)
    • <RhinosF1> bots1 updated & rebooted, all done --22:32, 25 March 2021 (UTC)
    • <RhinosF1> update+reboot tools --22:11, 25 March 2021 (UTC)
    • <RhinosF1> mysql back to rw --22:08, 25 March 2021 (UTC)
    • <RhinosF1> update+reboot db1 --22:03, 25 March 2021 (UTC)
    • <RhinosF1> phab+mysql to read only --22:00, 25 March 2021 (UTC)
    • <MacFan4000> restart icingabot --18:04, 25 March 2021 (UTC)
    • <MacFan4000> restart streambot --18:04, 25 March 2021 (UTC)

    2021-03-24

    • <MacFan4000> enabled notifico on sopel repo --02:00, 24 March 2021 (UTC)

    2021-03-23

    • <Sario> Stopping prod for config change --18:09, 23 March 2021 (UTC)
    • <Sario> restart prod --18:03, 23 March 2021 (UTC)
    • <Sario> Set phab highpri channel to #miraheze-sre --17:56, 23 March 2021 (UTC)

    2021-03-21

    • <RhinosF1> maint done --21:28, 21 March 2021 (UTC)
    • <RhinosF1> world to read only --21:15, 21 March 2021 (UTC)

    2021-03-20

    • <RhinosF1> blocked MJ12Bot - Exceeding Crawl delay --15:09, 20 March 2021 (UTC)

    2021-03-16

    • <RhinosF1> challenge all requests from OVH IPs --18:06, 16 March 2021 (UTC)
    • <RhinosF1> challenging all requests to phab due to ongoing abuse --17:59, 16 March 2021 (UTC)

    2021-03-15

    2021-03-14

    • <RhinosF1> deployed custom robots.txt for phab --22:08, 14 March 2021 (UTC)
    • <RhinosF1> cache stats now much better after phab rule changes at WAF level. We're now caching about 2/3rds of all requests for content by us. --11:40, 14 March 2021 (UTC)

    2021-03-13

    • <RhinosF1> MirahezeBotBeta is now running sopel-7.1.0.dev0 --14:15, 13 March 2021 (UTC)

    2021-03-11

    • <MacFan4000> updated phabricator --23:48, 11 March 2021 (UTC)

    2021-03-10

    • <RhinosF1> challenge requests to / and /auth/ as they're uncacheable and wasting backend resources --19:39, 10 March 2021 (UTC)

    2021-03-09

    • <Sario> restart StreamBot --04:33, 9 March 2021 (UTC)

    2021-03-08

    • <RhinosF1> fixed icinga db resource --14:38, 8 March 2021 (UTC)
    • <MacFan4000> revoked and deleted all certs for tools1 --14:29, 8 March 2021 (UTC)
    • <RhinosF1> bots1: restarting apache --13:45, 8 March 2021 (UTC)
    • <MacFan4000> revoked and deleted all certs for bots1 --13:39, 8 March 2021 (UTC)
    • <MacFan4000> disabled and deleted all miraheze.wiki vhost configurations on tools1 and bots1 --13:16, 8 March 2021 (UTC)
    • <RhinosF1> latency has jumped for some reason. Looks like requests also higher but that's expected. Could be a dodgy bot. Activated more intense firewall rules. --11:17, 8 March 2021 (UTC)
    • <RhinosF1> phab is back --10:45, 8 March 2021 (UTC)
    • <RhinosF1> killed PID 27238 ("php ./phd-daemon") & phab -> read only --10:27, 8 March 2021 (UTC)

    2021-03-06

    • <MacFan4000> restart streambot for config change --20:35, 6 March 2021 (UTC)
    • <RhinosF1> created GitHub ticket #1049671 regarding tum.de codeql emails and previous log entry --16:46, 6 March 2021 (UTC)
    • <RhinosF1> blocked ASN = 12816 for sending spam emails to me and scraping with the only purpose of collecting data for their spam --16:40, 6 March 2021 (UTC)

    2021-03-05

    • <RhinosF1> block empty User Agents --23:10, 5 March 2021 (UTC)
    • <RhinosF1> switch Phabricator hund alerts to paging --13:18, 5 March 2021 (UTC)
    • <RhinosF1> increase HSTS Max-Age to 1 year and submit to preload --08:08, 5 March 2021 (UTC)
    • <RhinosF1> change hund frequency to 4 minutes (1 higher than a #page) --07:24, 5 March 2021 (UTC)
    • <MacFan4000> restart icingabot --01:49, 5 March 2021 (UTC)

    2021-03-04

    • <MacFan4000> restart prod --22:14, 4 March 2021 (UTC)
    • <MacFan4000> restart apache --18:48, 4 March 2021 (UTC)
    • <MacFan4000> stop apache on bots1 to test alerts --18:40, 4 March 2021 (UTC)
    • <MacFan4000> stop icingabot as a test --17:28, 4 March 2021 (UTC)
    • <MacFan4000> restart apache --13:43, 4 March 2021 (UTC)
    • <MacFan4000> stopped apache on bots1 for testing new services --13:37, 4 March 2021 (UTC)
    • <MacFan4000> setting up Hund custom domain as status.mirahezebots.org --13:25, 4 March 2021 (UTC)

    2021-03-03

    • <RhinosF1> test done --14:02, 3 March 2021 (UTC)
    • <RhinosF1> testing downtime emails --13:56, 3 March 2021 (UTC)

    2021-03-02

    • <RhinosF1> fixed phab digest emails --22:45, 2 March 2021 (UTC)

    2021-03-01

    • <RhinosF1> updated a lot --22:26, 1 March 2021 (UTC)

    2021-02-27

    • <MacFan4000> restart both bots --19:01, 27 February 2021 (UTC)

    2021-02-26

    • <MacFan4000> restart -test --14:12, 26 February 2021 (UTC)
    • <MacFan4000> -prod --14:01, 26 February 2021 (UTC)
    • <MacFan4000> restart --14:01, 26 February 2021 (UTC)

    2021-02-25

    • <MacFan4000> restart streambot --22:05, 25 February 2021 (UTC)
    • <MacFan4000> updated sygnal ping lists --13:24, 25 February 2021 (UTC)
    • <MacFan4000> restart sigmabot and streambot following netsplit --13:03, 25 February 2021 (UTC)

    2021-02-24

    • <MacFan4000> restart prod --19:35, 24 February 2021 (UTC)
    • <MacFan4000> stop prod for config change --19:34, 24 February 2021 (UTC)
    • <MacFan4000> restart -test for config change --19:33, 24 February 2021 (UTC)
    • <MacFan4000> add icinga account for Sario --18:42, 24 February 2021 (UTC)

    2021-02-23

    • <RhinosF1> restart beta to apply 418 --16:30, 23 February 2021 (UTC)

    2021-02-22

    • <RhinosF1> apt-update/upgrade the world --21:53, 22 February 2021 (UTC)
    • <RhinosF1> MariaDB [phabricator_user]> UPDATE user_profile SET title = WHERE id = 5; --21:43, 22 February 2021 (UTC)

    2021-02-20

    • <MacFan4000> restart both bots --14:17, 20 February 2021 (UTC)
    • <MacFan4000> sudo pip3 install --upgrade sopel-plugins.channelmgnt --14:17, 20 February 2021 (UTC)
    • <MacFan4000> new version of channelmgnt released --14:14, 20 February 2021 (UTC)
    • <MacFan4000> updating channelmgnt branch protection rule to not require windows builds --13:11, 20 February 2021 (UTC)

    2021-02-19

    • <MacFan4000> restart both bots --19:09, 19 February 2021 (UTC)
    • <MacFan4000> restarting sigmabot --13:19, 19 February 2021 (UTC)
    • <MacFan4000> restart both bots --12:46, 19 February 2021 (UTC)
    • <RhinosF1> deployed fix to Sygnal for darkmatterman450 and other Comcast users --10:56, 19 February 2021 (UTC)
    • <RhinosF1> phab complete --07:54, 19 February 2021 (UTC)
    • <RhinosF1> all other repos recloned, phab ongoing --07:48, 19 February 2021 (UTC)

    2021-02-18

    • <RhinosF1> after a few fixes, deployed all new packages and confirmed installed. Ran puppet again. MirahezeBotBeta started up again. --11:10, 18 February 2021 (UTC)
    • <RhinosF1> new post-merge hooks synced --10:55, 18 February 2021 (UTC)
    • <RhinosF1> puppet-run x2 to deploy #57 --10:54, 18 February 2021 (UTC)
    • <RhinosF1> puppet-run (x2) / clear current pip installs of plugins in devvenv --10:54, 18 February 2021 (UTC)
    • <RhinosF1> stop MirahezeBotBeta for maintenance --10:47, 18 February 2021 (UTC)
    • <RhinosF1> started cloudflared everywhere --09:49, 18 February 2021 (UTC)
    • <RhinosF1> update db1 --09:49, 18 February 2021 (UTC)
    • <RhinosF1> update tools1 --09:45, 18 February 2021 (UTC)
    • <RhinosF1> update bots1 to puppet7 --09:41, 18 February 2021 (UTC)

    2021-02-16

    • <MacFan4000> restart -test for config change --21:06, 16 February 2021 (UTC)
    • <MacFan4000> verification was through dns --01:41, 16 February 2021 (UTC)
    • <MacFan4000> verified my ownership to mirahezebots.org on google search console and updated acl - also updated acl for phab.bots.miraheze.wiki earlier as well as I had updated the Twitter acl --01:41, 16 February 2021 (UTC)
    • <MacFan4000> updated GitHub profile for @MirahezeBots and verified mirahezebots.org via dns --01:14, 16 February 2021 (UTC)

    2021-02-15

    • <RhinosF1|NotHere> Notice: /Stage[main]/Profile::Discordirc/File[/discord-irc/fhconfig.json]/content: content changed --21:39, 15 February 2021 (UTC)
    • <MacFan4000> restart prod --19:50, 15 February 2021 (UTC)
    • <MacFan4000> stop prod for config change --19:48, 15 February 2021 (UTC)
    • <MacFan4000> restart -test for config change --19:47, 15 February 2021 (UTC)
    • <MacFan4000> removed Zppix' icinga account (unless when you can't login to Cloudflare access) --18:10, 15 February 2021 (UTC)
    • <MacFan4000> updated acls for all MHBots pypi projects --18:06, 15 February 2021 (UTC)
    • <MacFan4000> all bots should be online now --18:05, 15 February 2021 (UTC)
    • <MacFan4000> stop mh discord and sigmabot to reset password --17:42, 15 February 2021 (UTC)
    • <MacFan4000> restart prod bot --17:30, 15 February 2021 (UTC)
    • <MacFan4000> restart icingabot --17:23, 15 February 2021 (UTC)
    • <MacFan4000> restart streambot --17:22, 15 February 2021 (UTC)
    • <MacFan4000> restart test bot, leaving prod down for config changes --17:19, 15 February 2021 (UTC)
    • <MacFan4000> stopping all bots on the MirahezeBot account --17:10, 15 February 2021 (UTC)
    • <MacFan4000> temporarily stopping steamboat to login and change password --17:05, 15 February 2021 (UTC)
    • <MacFan4000> create new openweathermap account for bots[at]miraheze.org --16:17, 15 February 2021 (UTC)
    • <MacFan4000> deleted openweathermap account for tools.zppixbot[at]tools.wmflabs.org --16:15, 15 February 2021 (UTC)
    • <MacFan4000> restart streambot for config change --13:16, 15 February 2021 (UTC)
    • <RhinosF1|NotHere> restarting bots for config change --08:45, 15 February 2021 (UTC)
    • <RhinosF1|NotHere> remove Zppix access from all but Icinga, support of 2 online members of Miraheze SRE and high likely support of another 2 --07:42, 15 February 2021 (UTC)
    • <Zppix> stop mh relay until further notice --01:06, 15 February 2021 (UTC)

    2021-02-13

    • <MacFan4000> update apt packages and kernal on db1 and reboot --03:23, 13 February 2021 (UTC)
    • <MacFan4000> update apt packages and kernal on tools1 and reboot --03:11, 13 February 2021 (UTC)
    • <MacFan4000> updating kernal version on bots1 and rebooting --02:57, 13 February 2021 (UTC)

    2021-02-12

    • <MacFan4000> restart both relay instances --22:36, 12 February 2021 (UTC)
    • <MacFan4000> stopping both discord relay instances for config changes --22:07, 12 February 2021 (UTC)
    • <MacFan4000> restart discordircmh --19:58, 12 February 2021 (UTC)
    • <MacFan4000> restart discordircfh --19:51, 12 February 2021 (UTC)
    • <MacFan4000> restart discordircmh --19:42, 12 February 2021 (UTC)
    • <MacFan4000> reboot db1 --13:58, 12 February 2021 (UTC)
    • <MacFan4000> restart phd and icingabot --13:57, 12 February 2021 (UTC)
    • <MacFan4000> reboot tools1 --13:56, 12 February 2021 (UTC)
    • <MacFan4000> restart discord relay --13:55, 12 February 2021 (UTC)
    • <MacFan4000> restart prod sopel and sigmabot --13:53, 12 February 2021 (UTC)
    • <MacFan4000> reboot bots1 --13:51, 12 February 2021 (UTC)

    2021-02-10

    • <MacFan4000> restart icingabot --13:38, 10 February 2021 (UTC)

    2021-02-09

    • <MacFan4000> restart both bots --02:03, 9 February 2021 (UTC)

    2021-02-07

    • <Voidwalker> restart relay after fix "#sre":"miraheze-sre" -> "#sre":"#miraheze-sre" --20:36, 7 February 2021 (UTC)
    • <MacFan4000> restart mh discord relay to add channel --20:23, 7 February 2021 (UTC)

    2021-02-05

    • <RhinosF1|NotHere> made mirahezebots.cloudflareaccess.com SSO portal look nicer --18:34, 5 February 2021 (UTC)
    • <MacFan4000> added custom puppet-run command on all servers for all users (this makes running puppet a bit quicker) --16:49, 5 February 2021 (UTC)
    • <MacFan4000> related* --16:12, 5 February 2021 (UTC)
    • <MacFan4000> removing osticket install and released vhost config and dropping osticket database --16:11, 5 February 2021 (UTC)

    2021-02-04

    • <MacFan4000> update phab icinga check with new url --20:50, 4 February 2021 (UTC)
    • <MacFan4000> ./bin/config set phabricator.base-uri https://phab.mirahezebots.org --20:42, 4 February 2021 (UTC)
    • <MacFan4000> setup https://phab.mirahezebots.org --20:40, 4 February 2021 (UTC)
    • <MacFan4000> macfan@112:/var/phabricator$ sudo ./bin/config set security.alternate-file-domain https://phab-storage.mirahezebots.org --20:33, 4 February 2021 (UTC)
    • <MacFan4000> setup https://phab-storage.mirahezebots.org --20:31, 4 February 2021 (UTC)
    • <MacFan4000> restart prod --14:51, 4 February 2021 (UTC)
    • <MacFan4000> stop prod for config change (since rss is still on prod) --14:49, 4 February 2021 (UTC)
    • <MacFan4000> restart -test for config change --14:47, 4 February 2021 (UTC)
    • <MacFan4000> setup https://sopel.mirahezebots.org --14:44, 4 February 2021 (UTC)
    • <MacFan4000> configured https://phabdigests.mirahezebots.org --13:05, 4 February 2021 (UTC)
    • <RhinosF1|NotHere> use ssh- instead of ssh. --11:02, 4 February 2021 (UTC)
    • <RhinosF1|NotHere> add CF PKG repo to bots1 --10:57, 4 February 2021 (UTC)
    • <RhinosF1|NotHere> mirahezebots.cloudflareaccess.com now creates SSO to ssh.*.mirahezebots.org --10:53, 4 February 2021 (UTC)
    • <RhinosF1|NotHere> add GitHub sso to CF --10:45, 4 February 2021 (UTC)
    • <RhinosF1|NotHere> add icinga dns record --10:22, 4 February 2021 (UTC)
    • <RhinosF1|NotHere> add ssh.*.mirahezebots.org to proxy --10:20, 4 February 2021 (UTC)
    • <RhinosF1|NotHere> add phab (do not failover), phabdigests and phab-static (safe) to DNS --10:17, 4 February 2021 (UTC)

    2021-02-03

    • <MacFan4000> configure and enable authicated origin pull --23:25, 3 February 2021 (UTC)
    • <MacFan4000> config www.mirahezebots.org on bots1/tools1 --23:16, 3 February 2021 (UTC)
    • <RhinosF1|NotHere> added www.org records --23:01, 3 February 2021 (UTC)
    • <RhinosF1|NotHere> revert that, impossible --22:58, 3 February 2021 (UTC)
    • <RhinosF1|NotHere> making @.org a CNAME to each server --22:56, 3 February 2021 (UTC)
    • <RhinosF1|NotHere> added DNS records to .org for all our servers --22:54, 3 February 2021 (UTC)
    • <MacFan4000> add CNAME for www.wiki.mirahezebots.org as well --22:49, 3 February 2021 (UTC)
    • <MacFan4000> added CNAME for wiki.mirahezebots.org --22:46, 3 February 2021 (UTC)

    2021-02-01

    • <Zppix> run apt update and full-upgrade on bots1 as root --18:56, 1 February 2021 (UTC)
    • <MacFan4000> reenabled Cloudflare universal ssl --16:04, 1 February 2021 (UTC)
    • <MacFan4000> disabled Cloudflare universal ssl --15:36, 1 February 2021 (UTC)
    • <MacFan4000> configured bots1/tools1 to use the new cert and restarted apache --15:36, 1 February 2021 (UTC)
    • <MacFan4000> generated Cloudflare origin cert for mirahezebots.org --15:32, 1 February 2021 (UTC)
    • <Zppix> deploy new discord relay config change --14:39, 1 February 2021 (UTC)

    2021-01-29

    • <MacFan4000> cleaning up invalid tell messages and restarting prod --22:06, 29 January 2021 (UTC)
    • <MacFan4000> cleaning up invalid tell messages and restarting -test --21:56, 29 January 2021 (UTC)
    • <MacFan4000> remove mmdb files for both instances --18:05, 29 January 2021 (UTC)
    • <RhinosF1|NotHere> stopping cache for mhbots.org --16:26, 29 January 2021 (UTC)
    • <MacFan4000> configured mirahezebots.org vhost on tools1 for flask --16:09, 29 January 2021 (UTC)
    • <MacFan4000> configured mirahezebots.org vhost on bots1 for flask --16:04, 29 January 2021 (UTC)
    • <RhinosF1|NotHere> install Flask on bots1/tools1 and update npm on bots1 --15:06, 29 January 2021 (UTC)
    • <RhinosF1|NotHere> init'd new web on bots1/tools1 --13:56, 29 January 2021 (UTC)
    • <RhinosF1|NotHere> sudo apt-get install libapache2-mod-wsgi-py3 python-dev on bots1/tools1 --13:52, 29 January 2021 (UTC)

    2021-01-28

    • <RhinosF1|NotHere> add mirahezebots.org to Google Search --11:46, 28 January 2021 (UTC)
    • <RhinosF1|NotHere> issue SSL for mirahezebots.org on bots/tools*, enable Strict Proxy mode --09:26, 28 January 2021 (UTC)

    2021-01-27

    • <Voidwalker> upgraded all packages on all servers --21:40, 27 January 2021 (UTC)
    • <MacFan4000> setup vhost for mirahezebots.org on tools1 and enable --19:43, 27 January 2021 (UTC)
    • <MacFan4000> setup vhost for mirahezebots.org on bots1 and enable --19:38, 27 January 2021 (UTC)
    • <RhinosF1|NotHere> applied for Fosshost domain for cdn --18:58, 27 January 2021 (UTC)
    • <MacFan4000> restart icingabot --16:57, 27 January 2021 (UTC)

    2021-01-25

    • <MacFan4000> restart icingabot --12:58, 25 January 2021 (UTC)

    2021-01-24

    • <MacFan4000> renewed bots1 SSL --17:44, 24 January 2021 (UTC)

    2021-01-23

    • <MacFan4000> restart -test --18:43, 23 January 2021 (UTC)

    2021-01-22

    • <RhinosF1> that's on bots1 and I mean the install list --22:04, 22 January 2021 (UTC)
    • <RhinosF1> updating production pip --22:03, 22 January 2021 (UTC)

    2021-01-20

    • <MacFan4000> restart -test --20:07, 20 January 2021 (UTC)

    2021-01-17

    • <MacFan4000> restart icingabot --18:34, 17 January 2021 (UTC)

    2021-01-16

    • <RhinosF1> enabled org wide moderation limit --18:24, 16 January 2021 (UTC)

    2021-01-10

    • <MacFan4000> restart icingabot + phd --22:03, 10 January 2021 (UTC)
    • <MacFan4000> restart sigmabot --22:01, 10 January 2021 (UTC)

    MirahezeBotsWiki:Server admin log/Archive index

    2021-01-06

    • <RhinosF1> issues seem fine at least for now, I should be sleeping --07:05, 6 January 2021 (UTC)
    • <RhinosF1> upstream packet loss issues, mtr installed everywhere, restarted mysql which seems to have helped and stopped apache2 because db1 has no web portals --06:48, 6 January 2021 (UTC)
    • <MacFan4000> restart -test --02:10, 6 January 2021 (UTC)

    2021-01-05

    • <RhinosF1> add deepsource --22:07, 5 January 2021 (UTC)
    • <Zppix> update nodejs to latest and npm to 7.3.0; restart discord relay to ensure update applies --18:56, 5 January 2021 (UTC)

    2021-01-02

    • <RhinosF1> update libp11-kit0 --10:12, 2 January 2021 (UTC)
    • <RhinosF1> fix feedparser and chardet on prod --10:10, 2 January 2021 (UTC)
    • <RhinosF1> and a lot more --10:06, 2 January 2021 (UTC)
    • <RhinosF1> Successfully installed aiohttp-3.7.3 attrs-20.3.0 cachetools-4.2.0 --10:02, 2 January 2021 (UTC)
    • <RhinosF1> upgrade pip on devvenv --10:01, 2 January 2021 (UTC)

    2021-01-01

    • <RhinosF1> that's on beta --23:00, 1 January 2021 (UTC)
    • <RhinosF1> install requests-cache --23:00, 1 January 2021 (UTC)
    • <RhinosF1> deploying fix for botsphab:T114 --22:48, 1 January 2021 (UTC)