FCP X: Divide Large Libraries [u]

Posted on by Larry

[Updated Aug. 12, 2014, with additional tips on improving performance.]

For smaller projects it is perfectly fine to store all media and projects in the same library. But, for larger projects, this can cause problems. While the specific number of open files varies based upon RAM and project complexity, a rough number that I use to determine when a Library is getting full is around 3,000 clips.

Creating multiple libraries for large projects can simplify media management and improve the performance of Final Cut Pro X.

However, before we start moving media around, we need to define two key terms:

Managed media is media which is stored inside the library bundle on our hard disk. External media is media which is stored in a folder outside the library bundle, but linked to the folder using “symlinks.” Symlinks are a high-power version of an alias.

In general, there is no performance difference between Managed and External media; assuming the hard disks they are stored on are roughly the same speed. However, if you plan to use the same media in more than one Library, external media will decrease the amount of storage space you use. This is because when FCP uses media from one library in a project stored in a different library, it will copy the media file if the media is stored inside a library (Managed) but only copy the link to that file if the media is stored outside the library (External).

So, for very large projects, storing media outside the library will provide greater flexibility and reduced file storage requirements. Storing all media inside the Library on your hard disk is easier and simpler to manage and backup.

Final Cut has several media management functions built-into it:

Consolidate a library is the best option when media is stored in multiple locations and you want to store everything in one spot.

Copying media is the best option when you need to have multiple copies, say to share with another editor.

Moving media is the best option when you want to break up a large library into multiple pieces.

NOTE: The techniques described here require Final Cut Pro X 10.1.2 or later.

WHY THIS MATTERS

First, there is a limit to how many clips FCP X can have open at one time. While this numbers in the thousands, I’ve had several editors contact me who’s clips numbered in the tens of thousands! Finding simple ways to manage media means that we need to limit the number of clips that are open at one time.

Final Cut allows us to have multiple libraries open at once and to edit media from any open library into any project. However, when a clip is edited from, say, Library 1 into a project which is stored in Library 2, FCP makes a copy of the Library 1 media into Library 2. This makes sense because if you deleted Library 1 and all its media because you are done with it, you don’t necessarily want to delete the media which is referenced by projects stored in Library 2.

But, for large projects, copying media between libraries negates the advantage of storing media in separate libraries in the first place AND it takes up more disk space.

So, Final Cut provides a second option: When you edit media which is linked to Library 1 (External media) into a project which is stored in Library 2, only the link to the media is copied, not the media itself.

This means that when we use External media, we can use thousands of clips stored in dozens of libraries and, provided that all media is stored external to any library, performance won’t suffer and our hard disks won’t fill with duplicate media.

NOTE: Again, to obtain the best performance, only open the libraries that you need, and keep the total number of open clips and libraries as small as possible.

UPDATE: THOUGHTS ON PERFORMANCE

Apple made major improvements to system performance in the 10.1.2 update. So, as a start, if you are experiencing slow-downs, be sure to upgrade. After that, while there are many elements that affect the performance of Final Cut, the three big ones are:

Final Cut uses as much RAM as you have installed, so adding more RAM – assuming your system supports it – is always a good idea. As well, other applications or utilities can divert system resources, which also slows FCP down.

However, this article looks at using multiple libraries as a way to improve performance.

NOTE: Another factor affecting performance is network speed, if you are editing clips or accessing libraries store on a server, rather than editing clips or libraries stored locally.

THE IDEAL APPROACH

Ideally, we should think out media management before we start importing our first clip. This is very easy to say and almost impossible to do. Still, if you are someone who can actually plan that far ahead, creating multiple libraries and storing media externally will vastly simplify media management for large projects.

After you create a new library, and BEFORE you import any clips, select the Library and choose File > Library Properties (type Control + Cmd + J). This opens the Library Properties section of the Inspector

Click the Modify Settings button and click the Media menu.

Select Choose from the menu and navigate to the folder where you want all media for that library stored. In this example, I’ve created a folder on my 2nd Drive RAID called “External Media Folder,” and selected that.

NOTE: You can name this external folder anything you want. Pick a folder name that helps you remember what’s inside. Final Cut Pro will work with any folder name that the Finder supports.

Now, when you import clips into that Library, Final Cut will store the media in this external folder and create links to the media in the Library itself.

Cool. But who can plan that far ahead?

CREATE NEW LIBRARIES

If you’re like most editors, you won’t realize you need multiple libraries until you are drowning in files and wondering why FCP X is so slow. The answer is your libraries are too big and need to be divided up.

NOTE: Libraries and the clips they contain require RAM to open. When you have thousands of clips in a single library, you can exceed the RAM that FCP has available. Divide your media into separate libraries, and only open the libraries you need for the work you are doing right then. Get in the habit of closing libraries you don’t need to reduce the amount of RAM FCP uses to track all your clips.

Remember a project can contain media from any library, whether that library is open or not.

We want to create external media, so that we can edit any clip from any library into any project without duplicating media files in the library. This is actually quite easy – but not intuitive. The secret is to create the new libraries first, then move the media into it.

Here’s how.

Let’s say that we have one big library: “Main Library” and we want to divide this into three new Libraries: “Library 1,” “Library 2,” and “Library 3.”

NOTE: Remember, you can use any name for a library. I’m just trying to be really obvious here.

First, create a folder on your external hard drive to store the media for each library you want to create. Here, for example, I’ve created three folders on my 2nd Drive:

Next, create three new libraries in Final Cut Pro X using File > New > Library. Store the Library files wherever you want – again, I recommend never using your internal drive – and name them whatever you want. These Library files will not contain media, just links to the media.

For this example, I created Library 1, Library 2 and Library 3.

Select each of the three new libraries – one at a time – go to Library Properties (Control +  Cmd + J) and change their Storage Locations so that Library 1 Media is stored in Library 1 External.

Set Library 2 to store its media to Library 2 External.

Set Library 3 to store its media to Library 3 External.

UPDATE: IMPROVING PERFORMANCE

Something I just learned is that Final Cut can point multiple libraries to the same media folder. In fact, Apple recommends that when you need to create multiple libraries to minimize the number of clips open at the same time for a specific project, have all the libraries point to the same external media folder.

When importing media, FCP X assigns unique identifiers to distinguish clips from one another.  This means that clips with the same name shot on different days or using different cameras won’t overwrite each other. This is why it’s important to use the media management commands — like Consolidate — within FCP X because the app can properly track all links to media files wherever they are located.

Storing all media files from multiple libraries into one folder is the same as outlined above, except that when you set the Media location using Library Preferences, point each Library to the same external media folder. While storing media in multiple folders works perfectly well, creating a single external folder makes it easy to locate media and assures that there’s no confusion about where new media files should be imported.

While I think the idea of storing all media for a specific project in a single external folder makes a great deal of sense, I will create separate media folders to store media for projects that are not related to each other.

MOVE MEDIA

While you can drag media from one event in one library to another event in a different library, since we want to take a big project and break it into smaller groups, it is MUCH easier to do this by moving media contained in entire events.

Here, in the Main Library, I’ve created three events – one for all the media that I want to move to each of the three libraries.

NOTE: You can create multiple events for each library, because each event name will be retained when you move it to a new library. Also, when you move media files all keywords, ratings and metadata move with them.

Now, organize your media by dragging it into the appropriate event. To keep things simple for this article, I’m dragging three clips from an event in the Main Library, into the event for Library 1.

Let’s pretend that all our media is now organized and ready to move. The process is the same for each event and library.

Select the Event containing the media you want to move to a new Library. Choose File > Move Event to > [Name of Library]. In this example, I’m moving the Event “Media for Library 1” into “Library 1.”

Final Cut gives us the option to move camera original, optimized or proxy files. in general, because you don’t want to keep recreating optimized or proxy media, you should move everything.

NOTE: If the media is inside the library, it will be moved. If the media in the Main Library is linked, only a link will be created.

Open the Background Tasks window to see the status of your media. (This is optional, but fun to watch.)

Because we set Library Preferences to store all media external to the Library, Final Cut moves the media to the external folder – “Library 1 External” – while creating symlinks in Library 1 that point to the media location.

Here we see that the clips were moved from the Main Library to Library 1.

While the media was moved OUT of the Main Library into Library 1 External. We have successfully created external media from an existing library!

SUMMARY

If media is stored externally to the original Main Library, Final Cut will only move symlinks to the new libraries. No media will be moved from its external location. This is still a good idea, because we need to reduce the size of the open Library.

If media is stored inside the original Main Library, Final Cut will move the media to the media location specified in Library Preferences, which, in our case, is an external folder and create symlinks in the new library.

Either way, you end up with as many libraries as you need, the ability to open and close libraries as needed and no duplication of files or hard disk storage space.

Very cool.


Bookmark the permalink.

29 Responses to FCP X: Divide Large Libraries [u]

Newer Comments →
  1. Bmann says:

    Hi Larry,

    You write, “Remember a project can contain media from any library, whether that library is open or not.” (just after you start the “Create New Libaries” heading.)

    So if I close a library that has media referenced in a project stored in a different library, are you saying that media won’t turn red, or offline/missing media? Can you clarify this statement, please?

    B.

    • LarryJ says:

      Bmann:

      Correct. Here’s an example. Media stored in Library 1 that is used in a project stored in Library 2 will copy references of that media from Library 1 to Library 2.

      So, if you close Library 1, the media used in your project will NOT go off-line because references to that media are also stored in the Library that also holds the project.

      Larry

  2. Richard says:

    I don’t know if you’ve pointed this out before or not, but what I have found to be VERY helpful when it comes to increasing performance, is to make sure that within Library Properties, Media and Cache files are stored on DIFFERENT drives. I think this makes a bigger difference than simply storing media outside the Library if both are still using the same drive.

    • LarryJ says:

      Richard:

      I’m not sure I would agree with your enthusiasm, however, storing them on different drives won’t hurt.

      The reason I’m hesitant is that the only cache file that plays in real-time are render files. So you don’t really see major performance benefits.

      However, this is not a bad idea, just not one that – from my perspective – will significantly boost performance.

      Larry

  3. CG says:

    Larry:
    Is it necessary to create new library folders on the media storage drive vs. just creating new libraries in FCPx? I’m using FCPx 10.1.1. Thank you.

  4. Reza says:

    Hi Larry,

    I have finished working on a project with a massive library on fcpx 10.1 and now have to handover the projetct file to my client.

    The issue is they only want to recieve the bits of the clips which was used in the storyline; not everything imported to the library.

    As I understand Consolidating copies all the clips from start to end regradless of how much of it was used into a new location.

    I remember FCP 7 had an option called “manage project” and I could only save the used pieces in the timeline.

    Is there anyway to do the same in fcpx?

    Thanks,

    Reza

    • LarryJ says:

      Reza:

      At this point, no, FCP X does not support exporting only the portions of a clip that is used in the Timeline.

      Larry

  5. Reza says:

    Thanks a lot Larry,

    I’ve just put a request of imporvements on FCP’s feedbaack page to add that option. Hopefully, they’ll add it in the fututre.

    Thanks again.

    Reza

  6. Sara says:

    I have tried these step by step instructions a good 5 times, with no luck. Please direct me in the correct path with what I might be doing wrong.

    • Larry says:

      Sara:

      Um, its a bit hard to figure out what’s going wrong unless you provide a few more details. For example, what makes you think its not working? Describe the problem in a bit more depth so we can figure out the problem.

      Larry

  7. Very very kind of you to respond.For true clarification please comment:

    I was new to FCP period but began in X, which is great.
    I immediately had to edit files and export. Now I am in the true work phase & want to make sure I store my media properly.

    I have 14 different Media Events all stored in the main library. Right now I only want to work on 1 project which I expect to have a lot of files.

    Do I understand correctly: I can create, for ex, a library for each Media Event? I can then move my Event- ie the imported data for that event- directly into the newly designated library? All of the media, all the data has now moved into that new library location? I can then leave my current main project in the original library, and continue to edit at will.

    Is this the correct understanding? Right now because I am still so new at FCPX- I don’t want symlinks. I don’t want to point the mdeia to a library or vice versa. I want all the media consolidated according to the event, with all the project files located under that umbrella.

    If I have misunderstood, how can I best achieve this?

    Again, MANY thanks for being such a generous guru!

    • Larry says:

      Michelle:

      A Library is the master collection of everything, Events are just folders which are stored in the Library.

      So, a better way to think of this is that you create a Library for each “project” – then, inside that Library you can add as many folders (Events) to store media, audio, project files, graphics… as you want.

      To create a new Library from an Event, use File > Move Event > New Library.

      Whether media is stored IN a library or pointed to a library is totally determined by how you import that media. Selecting “Copy to Library” means that media is stored inside the Library, selecting “Leave Files in Place” creates simlinks.

      Larry

  8. Many many thanks. I am going to try this now.

    I truly thank you for your generosity to everyone in the field. While this no doubt only helps your own success, and was probably helpful in the sale of your business, without a doubt the service you provide so kindly, is truly magnificent and magnanimous.

    Thank you.

  9. Hi Again,

    Under my Main Library with the 4 Stars in Purple Logo
    I have 15 Events, each with 1 Star in a Purple Logo.
    Each of these is housed under the Main Library right?

    However it would be helpful to speed if I created 14 other libraries for the other “projects” – ie, entirely different works I wish to create. Is that correct?

    If so, I tried 2 ways to create a new library. The first was to go to File – New – Library. I wanted to name the new library I wished to create. The fcpxbundle is in grey. I have this housed on my external drive. I only want all data to be on my primary external drive, and then my secondary, back up external drive. I appeared I could not highlight the fcpxbundle to direct the new library to be housed in there. Perhaps it goes directly there?

    I have a screen shot I can share for clarity.Cannot attach here.

    I highlighted an event in the main library. I then went to File- New-move- I got a new library in my external drive, shown as a new Logo with the 4 stars.
    That event moved out of my Main Library. However no files, no media opened in that new 2nd Library that I created. Fortunately- I don’t care about this event which is why I used it as a guinea pig.

    I am confused to say the least.

    Many thanks, and I apologize if I am being thick about this.

    It is critical to me that all the new work is done & housed in the right place.

    Many thanks,
    Michelle

    • Larry Jordan says:

      Michelle:

      You create a new Library – as you did – when you have not imported any media. Then, when you import media, IF you check “Copy to Library” in the Media Import window, all imported media will be copied into your Library. This is what you want.

      For existing events, be SURE that the event has media in it before moving it to a new Library. That library will only contact media if, when you imported the media, you selected “Copy to Library.” If you use “Leave in place,” all that copies are symlinks.

      So, that setting: “Copy to Library” is critically important when selecting where you want media to go.

      Also, within FCP X, be sure to drag all media that you want stored in an event before moving that event to a new library. You can always move it later, but it is easier and faster to move it before a library is created.

      Larry

  10. Michelle Jaffé says:

    Thanks again.

Newer Comments →

Leave a Reply to Larry Jordan Cancel reply

Your email address will not be published. Required fields are marked *

Larry Recommends:

FCPX Complete

NEW & Updated!

Edit smarter with Larry’s latest training, all available in our store.

Access over 1,900 on-demand video editing courses. Become a member of our Video Training Library today!

JOIN NOW

Subscribe to Larry's FREE weekly newsletter and save 10%
on your first purchase.