Page MenuHomePhab@Tower
Create Task
Maniphest T99

Restore prod
Closed, ResolvedPublic
Actions

Description

During T98 an issue was found with the FirewallD component and during unbreak operation, the server fully isolated itself and rescue is required.
Due of no backups, server is not considered a cattle and rescue operation is in progress.

Event Timeline

thunderysteak changed the task status from Open to Work In Progress.Jan 4 2020, 4:20 PM
thunderysteak triaged this task as Unbreak Now! priority.
thunderysteak created this task.
thunderysteak created this object in space S3 Public.
thunderysteak created this object with visibility "Public (No Login Required)".
thunderysteak updated the task description. (Show Details)
thunderysteak added a comment.Jan 4 2020, 4:31 PM

Booted into rescue mode and system partition mounted
Attempting fix

thunderysteak added a comment.Jan 4 2020, 4:42 PM

Seems like all issues are caused by OVH's custom kernel
Attempting to replace the kernel for a stock LTS one

thunderysteak added a comment.Jan 4 2020, 4:50 PM

Kernel replaced, switching to boot from hard-drive
Waiting for server to ping

thunderysteak added a comment.Jan 4 2020, 5:00 PM

SSH ports returned to stock port to not cut connection again
Firewall configs nuked from orbit
Proceeding with new rule configuration

thunderysteak added a comment.Jan 4 2020, 5:04 PM

Firewall is up.
MySQL database has entered failed state, attempting to recover.

thunderysteak added a comment.Jan 4 2020, 5:24 PM

MySQL folder in /var/run/ is missing and is not being recreated on boot
Creating it and assigning correct permissions makes the server function but gets removed on boot. Possible side-effect of kernel change.
Applying fix from here

thunderysteak added a comment.Jan 4 2020, 6:21 PM

Continuing verification of functions
Firewall is functional and core services are up.

thunderysteak closed this task as Resolved.Jan 4 2020, 6:43 PM

Server functions with no errors visible. Closing

thunderysteak moved this task from Current Queue to Completed on the Steak Commissions board.Jan 4 2020, 6:43 PM