Downtime 2/2/2024-2/3/2024

[Susan]

Dear Forum Family,

I'm writing to apologize for the extended downtime of our peer support website tonight. Around 6pm, our internet connection unexpectedly dropped. I promptly contacted our ISP's business support line to investigate, but did not receive a call back by 8pm, so I reached out to their residential support team, who discovered the ISP's engineers had mistakenly reallocated our IP range to a new stadium opening in town this weekend.

We had no choice but to accept a new IP range from the ISP, I immediately updated our server configuration. Once the server was ready, I updated the DNS records to point to the new IP address. Through persistent troubleshooting and reconfiguration, I was able to restore service by midnight.

I should have been notified well in advance if they planned on reallocating our IP range. This lack of notice is unacceptable, and I intend to raise these issues with the head of the municipal Fiber to the Home ISP to prevent this from happening again. They know the importance of the service we provide the Trans community, and as a support website we rely on having stable internet connectivity. 

We are dedicated to providing a reliable online community, and I know this disruption cut off access to important peer support at a difficult time. Thank you for your patience and understanding. I sincerely apologize for the unexpected disruption and will be following up to ensure we are notified in advance of any changes that could impact our connectivity going forward.

Sincerely,

[Devlyn]

I just woke up and missed all the excitement. Glad we're back.

Hugs, Devlyn

[Sarah B]

I felt the effects down under. Not happy Jan.

Seriously? mistakenly reassigned by engineers?  No checks taken care of before going ahead?

Something is not right. Have you had the same IP range, since you started Susan's Place, Susan?

Love and hugs
Sarah B

[Susan]

I had it since there has been fiber to the home in this town. I attribute it to faulty record keeping. 

We are going to be asking the ISP for

•     Formal incident response procedures for unplanned network changes impacting your services
•     Advanced notification SLAs (ex. 30 days notice) before maintenance or alterations
•     Faster support response SLAs (ex. 30 min critical incident, 2 hrs high priority)
•     Increased Priority routing and QoS configurations to guarantee our traffic enjoys priority status, as one form of compensation.
•     Review change management policies to prevent unauthorized/uncommunicated network changes
•     Documentation of your server IPs, subnets, and MACs to avoid accidental alterations
•     Regular reports or insights into network operations, capacity, and planning
•     Minimum 30 day advance notification for any network changes that could impact your services
•     Immediate escalation contact in the event of a sudden outage or change, I now have their engineer's direct cellphone number.
•     Maximum 30 minute response SLA for critical incidents like this
•     Regular status reports on network operations and capacity planning
•     Compensation for outages caused by unnotified changes

We may not get all of it. But it never hurts to ask, the worst thing they can do is say no, and if they do we are no worse off than we are now.