My family shoots a lot of video. We’re on our second Flip camera, we used a mini-DV camcorder until it would no longer record cleanly, my wife and I both have iPhones, and we have two young sons who we find very photogenic. Add that all up, and you get what can seem like an unmanageable amount of digital video: 560GB divided between 1,500 video clips since 2004, 80% of that since our first was born in 2009. At last count, those clips totaled 80+ hours, and that number is increasing every day.
A couple of years ago, I digitized all of my wife’s family’s home videos from the 80s and 90s, and it was heartbreaking to see how much of the videotape had degraded and was no longer watchable. I didn’t want anything like that to happen to our family memories, whether it be due to hard drive crash, accidental deletion, or just disorganization, so over the last few months, I’ve formalized the system that I’ll use to ensure that all of our videos stay organized and backed up.
On the first of every month, I ensure that all videos from our Flip camera (which we use rarely, and usually only for long recordings) and our iPhones have been transferred to a computer. On that computer, there’s a daily cron job that looks for new video files in the Flip and iPhoto libraries and copies them to the “Home Videos” folder on our Drobo, which is network-accessible via our Time Capsule.
Using exiftool, all of these new files are automatically renamed using the format “2012-01-02 03 04 05.MOV”, where the filename is the date and time the video was taken:
exiftool "-FileName<CreateDate" -d "%Y-%m-%d %H %M %S.%%e" [files_to_rename]
I then add captions to the videos by adding a description of the video to the filename (after the timestamp). If multiple clips are from the same event, I can leave them without a description, and a cron job will automatically add the description from the previous clip, appending ” – 2.MOV”, ” – 3.MOV”, etc. (It only does this for clips taken within a 15 minute window of one another.) It takes me about 10 minutes to caption a month’s worth of video.
I chose to organize the video files directly, without using any video library management software, since I want to have as much control over the portability of the clips as possible. I’ll want to be able to stream them to myriad devices over my home network or even the Internet, and I don’t want to have to deal with whatever folder and file structure was chosen by the management software.
If all I wanted to do was keep all of my clips organized by date in a folder, I’d be done right now, but I want to burn DVDs of this footage for easy physical portability, universal playability, and as a last-ditch backup solution.1
A cron job runs every night, checking if I have accumulated 2 hours of named footage in the folder on the Drobo that has not yet been burned to DVD. If I have two hours, but it’s not all named, I get an e-mail reminding me to name it. If I have two hours and it is all named, it will automatically create an iMovie project, populating it with the clips (using ffmpeg to convert all the non-.MOV videos into a format that iMovie won’t choke on). It also auto-creates chapters in the project by stripping the timestamps from the clips and adding a new chapter for each new event. For example, this set of clips:
- 2001-01-01 00 00 00 – New Years Day.MOV
- 2001-01-01 00 04 03 – New Years Day – 2.MOV
- 2001-01-01 00 17 12 – New Years Day – 3.MOV
- 2001-01-21 17 00 00 – Christina’s Birthday.MOV
- 2001-01-24 14 00 00 – Hiking.MOV
- 2001-01-24 14 11 00 – Hiking – 2.MOV
- 2001-01-29 15 00 00 – Snowboarding.MOV
would result in this chapter list:
- New Years Day
- Christina’s Birthday
- Hiking
- Snowboarding
Once the project has been generated, the rest of the process is manual, since I want to hand-check the chapters and make sure that a multi-day event (like a family vacation) hasn’t been split across DVDs. I also like to customize the DVD with relevant background images and an appropriate theme. (I’ve even been known to pick out some complementary background music for the menus if I’m feeling particularly creative.)
Once the DVD project is ready, I export it as a disc image to a different folder on the Drobo and eventually burn it to a physical disc.
It’s a solid setup for my next project, which is to find a way to easily watch all of these videos on our Tivo, our iPhones, or at least via a Mac Mini connected to our TV, because having all of this video of family memories doesn’t matter if it’s too much of a hassle to actually watch it.
How does this compare to your own process for organizing and managing your personal media collection? Any tips or tricks that you’d like to share?
1. It’s not my only backup: in addition to the inherent drive failure protection provided by the Drobo, I have everything backed up to two external drives as well, one of which is kept at home, the other off-site. Every six months, I update the contents of the on-site drive, swap it with the off-site drive, and then update the contents of that one too.
Love your thought process – where would you recommend I start reading about cron jobs? Do you think i can get apple automator to do the same thing?
This is a nice overview of how cron works: http://blog.niklasottosson.com/?p=507
I haven’t used Automator much, since I had to write custom scripts to do my video processing, so Automator’s built-in actions weren’t of much use to me.
impressive. At what point of the process do you go in to edit?
Can you recommend any simple tools for windows users to mimic some of your processes?
I don’t do much editing. I remove unwanted clips when they’re still on our phones/cameras, and if there’s an event that I want a special edited version of, I do that at the end, after the iMovie project has been created.
I can’t speak to Windows; I left it behind five years ago and have never been happier. :-)
for family videos, PLEX . nough said
now hoe to manage the photos and “short” video recordings thats still a mystery to me
how do you access them again ? any app that have some nice interface like plex ?
i want them served from my server, not uploaded anywhere
Thank for this blog – has given me the inspiration to ‘talk less and do more’
You mentioned “Using exiftool, all of these new files are automatically renamed using the format “2012-01-02 03 04 05.MOV”, where the filename is the date and time the video was taken”
this is what I have typically been doing manually but want to automate .
however, my video files do not have an accurate creation day/time
where do you get the date/time the video was taken info in the meta data of the video file?
I’m not sure what to do in that situation, but if you run
exiftool -a [filename], it should tell you what data it does have, and that might give you a lead.This is a response to Jason’s question. From 2004 – 2009 I used a Sony Digital8 video camera. To extract the metadata from the Digital8 tapes, I used software called DVDate. Basically, after you’ve transferred the .avi file from the tape to your computer (I used MS MovieMaker), load the .avi file into DVDate and it will create a binary scene file which contains scenes with date stamps. From that scene file DVDate will then automatically generate spreadsheets pre-populated with scenes and their date and times. I found it to be very useful as I did not always “announce” the date or time when filming. There is no GPS info, of course, as the cameras didn’t have that, but I found that once I knew when the date was it was easy to jog my memory as to where the filming took place.