Contact
Subscribe

Git Saved our Bacon

Oct 6, 2011
work safe

My current work project is a mobile app for a medical device company. The software is, depending on the eventual feature set, FDA regulated. This means we will eventually have to undergo a HUGE FDA audit on our codebase. Not a problem if we have out entire history with comments and tags tying commits to work items, right? Well, our source control server died.

Yesterday morning, our gitorious server decided to eat its virtual disk. IT is trying to get it pulled from a recent backup, but it’s a many terabyte off-site backup. But we have work to do. So we fired up a new git instance, pushed master from the dev who had the latest, and we started working again. Once gitorious is stood back up, we will push latest to that and our code reviews in crucible will work like a champ again. Yay!

Here is how we did it:

  1. create a bare git repository on a developer’s machine (mine). 
    git init --bare /c/GimmeTehCodez/project.git
  2. Share the directory with windows sharing. I limited access to only my team members, but had to set them all to Read/Write access.
  3. add a new remote for the mirror. Note the four /s in the remote URL. Two are a relative file URL, three are an absolute URL. Four is another computer. In this case it is Wallace who shares my GimmeTehCodez directory for coworkers to upload stuff that can’t be put in email. Kind of a neat hack, huh? No drive mapping needed! 
    git remote add mirror 'file:////Wallace/GimmeTehCodez/project.git'
  4. start by pushing the last good master back in to the repo 
    git push mirror master

And we’re off and running again! We could get a network share from IT, but that would take too long and distract them from getting my server back.

If we were using TFS and had a server issue like this, how long would we be out? If we’d lost our code history, how much worse would the FDA audit be with all that lost history?

Next step is to start thinking about a ‘hot standby.’. Probably not needed, but it would be cool. Then I can think about using git-tfs to make our tfs projects stand-by as well…

4 Comments
Tags

4 Comments

  1. Anthony Panozzo  •  Oct 6, 2011 @12:50 pm

    Ah, a great feature of distributed VCSes. Underrated until the day when things go wrong. I’m hoping I will never need to use this, but I’m sure the day will come… 🙂 Appreciate the writeup!

  2. Alex Conner  •  Oct 7, 2011 @9:29 am

    Git is an amazing SCM solution all around. To spin up a new server you don’t even need the latest code, since as developers push the code will get updates. And, Git protects the code such that it’s cryptographically secure and will complain loudly if your new server doesn’t have the same “heritage” as your copy.

    Earlier this week our Git host was acting flaky (turned out to be a hardware problem, I love that VM’s can still suffer from those) and instead of getting worried and trying to back it up I just checked the last CI build and made sure it was up-to-date.

    As far as a hot standby goes, that’s super easy as well. It can be added as a step in CI (everyone should be doing CI), or even a post-recieve hook on the “canonical repo server” that forks and pushes to a backup server. (similar to how Heroku does deployments). Then everyone can add that as a remote and use that whenever the primary is unavailable. If you’ve got a ton of repos you can also just rsync them periodically though that’s likely going to be slower.

    Also Gitolite has Mirroring as a design feature so that’s worth reading up on here.

  3. Alex Conner  •  Oct 7, 2011 @9:34 am

    Or if you want something cooler (who doesn’t) this looks pretty spiffy.

  4. Ball  •  Oct 7, 2011 @10:18 am

    Thanks for the tips, Alex!

Leave a Reply

You must be logged in to post a comment.