Friday Night Database Recovery Crash Course (A True Story)

Comments · 456

• SK

  @Skoro_bog · 12 days ago

  - our change management allows changing prod DB parameters without any peer review <br>- our db recovery documentation is out of date, so basically useless <br>- our alerting is equivalent to a blank page<br>What do we do? Right, fire the junior engineer!

  555

• KW

  @kwak0 · 12 days ago (edited)

  I will never EVER run a rm -rf command that does not include the literal directory path in the command. That stuff is way to dangerous to mess around with.

  330

• AR

  @ara0n · 12 days ago

  Chatgpt starts reasoning - man this cracked me up

  456

• TI

  @timop6340 · 12 days ago

  I realised what is about to happen the moment temporary variable was used and the session timing out.

  286

• MA

  @matthewaparte7423 · 12 days ago

  I yelled ‘NO!’ when the shell session ended😭

  94

• KK

  @KK-jo2uf · 12 days ago

  Recovering data for data gap from analytics and stripe or 3rd party logs would be an absolute nightmare

  115

• OF

  @ofadiman · 12 days ago

  The backup job which is running every night but the backup is not working anyway is so classic 🧑‍🚒

  79

• MA

  @marc-hg · 6 days ago

  As a software engineer this was nerve-racking to watch. True horror

  16

• SA

  @SamikBiswas-y8i · 12 days ago (edited)

  Dude I&apos;ve not watched any youtube video with this much interest for months<br><br>Edit: I always tend to skip videos or make it faster while watching but this one kept me engaged from start to end. The visuals are not generic diagrams or flowcharts but all the interfaces are realistic enough and also a good storytelling. Hope this blows up more

  221

• OV

  @OverG88 · 12 days ago (edited)

  One of my colleagues dropped data from one (luckily, one) table after 11 pm. Friday night… The whole DB was 20 TB in size. I swear I will never forget her Slack messages. Horror and desperation in the same time.

  82

• BU

  @BusterDoggyDogg · 12 days ago (edited)

  As an Incident Manager I&apos;ve seen this happen in real life more than once...<br>People running Terraform commands without knowing what they&apos;re doing are the worst.

  56

• LK

  @LKRaider · 10 days ago

  Happened here, twice in 5 years, with junior engineers I hired. With my help we got the systems restored. They stayed hired both times per my call, and made to senior engineers within our company. I found these kinds of situations build character and a real commitment to the team afterwards, it conjures an equivalent effect of fighting a battle and winning together.

  11