Industry of Change: Linux Storms Hollywood
Before the summer of 2001, Linux supporters often pointed to any of a number of single-company deployments as a measure of success for the fledgling operating system. There was Burlington Northern, which committed in February 1999 to deploy Linux in 250 US stores. That was followed by Japan's Lawson, which struck a deal with IBM to supply that convenience store retailer with 15,000 IBM Linux-based eServers running on Red Hat software. Ford announced a plan where they would deploy 33,000 Linux desktops. These were big wins for the open-source faithful. But they were corporate waves in a sea of change. What Linux needed was a tidal wave--an industry-wide migration--to signal that the penguin had come of age.
Enter the visual effects industry, the collection of studios that produce special effects, or VFX in industry parlance, for movies and animated tales like Toy Story and Shrek. This is an industry ripe for change, an industry struggling to shake the bondage of single-vendor solutions and high-priced specialized hardware. It's also an industry that tested the waters of Windows and found it flowing in the wrong direction.
This isn't a story about one or two studios adopting Linux as servers in their renderfarms, those back rooms full of servers used to produce the individual sets of frames used in a movie. We're talking about the entire industry--from Rhythm & Hues to Pixar, from Digital Domain to DreamWorks. DreamWorks-PDI had over 2,000 Linux-based CPUs on-line by the summer of 2001. Their summer blockbuster Shrek was rendered on 1,000+ mostly Linux machines (see GFX: "DreamWorks Feature Linux and Animation", August 2001 issue of LJ). Pixar has only deployed 15 stations in production and 25 in software development, but VP of Technology Darwin Peachey says the studio is on the verge of a major purchase and deployment of desktops to replace their current SGI desktops. Even Industrial Light & Magic is considering a major switch to the penguin OS.
And this isn't the infrastructure saying they will support Linux, like IBM or Compaq or HP announcing they will support the OS--it's the end users demanding it from suppliers of applications and hardware. Back in June 2001, Ray Feeney, technology committee chair of the Visual Effects Society said, "For the high-end part of movie making, 80-90% will be Linux-based inside of 18 months. Everything is going Linux." This sort of mass migration has never happened before in the Linux world. The tidal wave is here.
Understanding how this wave was formed requires some understanding of the industry itself. Effects studios talk about movie production as pipelines, the set of processes required to create effects and integrate them into a movie. A pipeline has two distinct sides to it: the graphic workstation and the renderfarm. The latter is like any other room full of servers, crunching away on any given problem. In this case, the problem is producing the 3-D imagery from models fed to the farm by the many artists working for the studio. The artists work on the other end of the pipeline, on the graphic workstations.
The first ripple in this tidal surge came with the use of Linux by Digital Domain to render frames for the movie Titanic. Involved in this film was well-known Linux graphics guru Daryll Strauss, who covered this story for Linux Journal back in February 1998. At the time, Daryll used a room full of Alpha-based Linux systems networked together to render some of the water scenes used in the movie. In this early stage, Linux still was used in its traditional role as a back-end server. The front-end graphics workstations were still primarily the domain of SGI IRIX systems.
In 1999, SideFX software ported their very popular (and very expensive) high-end 3-D modeling and animation package, Houdini, to Linux. Linux Journal again covered the story, this time in an interview I did with SideFX's Director of Research and Development, Paul Salvini. Houdini is an artist's tool used to create the models that renderfarms crunch on. At the time that Houdini was ported, Linux still had graphic-related limitations, such as a lack of support for hardware-accelerated OpenGL (a de facto industry standard for doing 3-D applications and games). This created a chicken-and-egg problem, according to Salvini. Doing a product like this for Linux required hardware acceleration to make it really viable, but hardware acceleration often requires applications in order to warrant drivers to be written." Drivers from video card makers weren't being written because there were no applications that needed them, and applications weren't being written because no drivers were available. SideFX sidestepped the issue by using software-accelerated OpenGL, a slower and problematic alternative that didn't require special video card drivers. Still, it was enough to entice the VFX industry toward Linux. It also provided motivation to graphics card vendors to provide both assistance to XFree86 and to begin work on their own proprietary drivers.
With the operating system proving viable on the renderfarm side of production and improvements being made on the graphics workstation side, the stage was set for the earthquake that would shape the VFX industry's tidal force: the VES 2000 Linux Summit. The Visual Effects Society (VES) is the professional member organization for the VFX industry. That summer, 45 representatives from 24 effects companies met in Santa Barbara, California to talk about the need to move away from SGI IRIX and into Linux. The event, organized by the VFX Society's Technology Chair Ray Feeney and Kate Swanborge of DreamWorks, began the process of thawing the usually frozen communications between studios. Stress was placed on the need for third-party applications to be ported. At the time, only five of the more than 20 standard graphics packages used by the industry had been ported to Linux. And worse, application vendors felt that Linux was not the operating system being used by the industry.
Most studios began porting their own in house software over to Linux on their own. Meanwhile, the VES invited software vendors into workshops to discuss the need for ports of their applications. By moving in unison, the VES membership felt they could apply enough pressure on the vendors to get applications ported quickly.
Some applications were already available. Dana Batali, Pixar's director of RenderMan product development, said that they actually had the PRMan renderer available on Linux for a couple of years. SideFX's Houdini port was the first publicly announced product. The pressure applied by the society worked. By the time SIGGRAPH 2001 rolled around in August 2001, ported applications included Alias|Wavefront's complete Maya modeling and rendering toolset (see "GFX: Alias|Wavefront Maya 4", October 2001 issue of LJ), Avid's Softimage XSI image compositor, Kaydara's Filmbox content authoring package, Silicon Grail's RAYZ compositor and Nothing Real's compositor, Shake.
The VFX industry's migration to Linux exposes some interesting interactions that open-source advocates might not have noticed previously. For example, the issue of cost isn't necessarily important when compared to Microsoft products, and it also isn't a factor when considered without the underlying commodity hardware.
Toronto-based Axyz Animation did much of the early adopter testing for SideFX's Houdini on Linux. John Coldrick, senior animator of Axyz, which has already replaced all of their workstations with Linux-based PCs, says the migration was a cost issue when coming from IRIX, but a technology issue when coming from NT:
[Linux] doesn't offer more than IRIX except it's substantially less expensive. But it is the scalability on Linux that is phenomenal. If you start with eight workstations with NT you're fine, but if you have to balloon up to 70 or 100 you run into some major problems. You do that with Linux with no problems.
Cost wasn't the most important issue for Pixar either. "Most people tend to focus on cost" when it comes to migration issues, says Pixar's Vice President of Technology, Darwin Peachey:
But cost isn't the most important thing at these price levels. The most important thing is to look at what is the best performing hardware. Right now that's Intel-based workstations, with NVIDIA or ATI equipped graphics solutions. These have now eclipsed traditional RISC workstations (such as SGIs) in terms of graphics and CPU performance. Quite apart from price/performance, if you look at absolute performance, it appears that this is the way the industry has to go.
DreamWorks Animation Head of Technology Ed Leonard says that, these days, computing costs have to be recouped with each film:
Historically we purchased a large amount of SGIs. Those were amortized over several films. You'd want to get five years out of some of that hardware due to the expense of that investment. With the Intel/Linux strategy today, we're moving toward what we refer to as disposable computing. Film productions are generally two years long, and during that time the technology takes several steps ahead. We normally anticipate to recoup a large portion of our hardware costs with every production. So with each new movie, we go out and purchase a new renderfarm.
But it was SideFX's Salvini who put it most bluntly: "It's one thing to have your product run on Linux, but so what if you save $200 on the OS and you need a $5,000 graphics card?"
The use of commodity components is allowing the industry to remove their dependency on SGI, a company with an expensive, specialized hardware and a questionable future. And, Peachey adds, there are only two ways to go with the Intel solution: Microsoft NT or Linux. Feeney says NT turned out to be both a technical problem as well as a political one.
"Once upon a time there was a great focus on the part of Microsoft that the VFX industry would be the next realm they would conquer", says Feeney, who is also the founder of VFX studio Silicon Grail. "They would fix big data transfer issues in their OS and so forth, taking Windows from a consumer tool to the enterprise." But that never happened, he says,
This industry is like a team sport--it's a collaborative effort. The ability to share data and other material, aside from office-style documents, is extremely complicated and made more so by the limitations of the Microsoft environment. So it was for technical reasons that the industry is looking back to the [Linux] market.
Axyz's John Coldrick agrees and adds that the idea of going NT sent shivers through him. "We were used to working in a UNIX environment where we had control and networking and stability. It's more like NT doesn't offer enough for us: the networking is awful, there are no links and the stability is not as good." Moving to Linux provided less technical problems from a porting standpoint. Because Linux is UNIX for all practical purposes, porting from IRIX was far easier than going to NT.
A recent Giga survey found that a large group of Microsoft customers not related to the VFX industry were not willing to upgrade because of the new licensing rules for XP, which locked them into two-year upgrade cycles. Instead, many are planning to migrate to other options, mostly because they plan to keep their PCs for longer periods. Surprisingly, the group that wants to upgrade fairly often--the VFX industry--didn't get much support from Microsoft. Feeney says,
They're off to work on issues that don't solve the set of issues relative to the high-end effects industry. They decided that they would be better off spending their time elsewhere, like on the Web with Hailstorm and .NET. So they never bit on the enterprise market.
But it isn't just that Linux is more like UNIX. Before the first Linux Summit, the studios were contemplating their upgrade policies. "This industry is a relatively new industry that grew up in the past five or seven years", said DreamWorks' Leonard, "And that is about the lifespan of the SGI hardware."
Industry insiders began wondering what to do about that--either buy new SGIs and deal with the issues of corporate stability that might imply, or go with commodity platforms. Leonard says that Intel's IA-64 is going to make a quantum leap for the VFX market. "It will force everyone to migrate to commodity hardware over the next 18 months", matching the IA-64 release cycle.
As part of their migration, the industry has had to consider upgrade policies in a new way. Studios find that upgrading commodity hardware doesn't make much sense. Desktops and renderfarms have a two- to three-year lifetime, matching the time frame for a given film production, after which faster hardware makes them replaceable. As new systems are brought in, new software installations are too, and that means verifying applications continue to work on both. In many cases, studios will continue to depend on a single-source vendor--though they now have a choice of vendors--to supply Linux and third-party applications certified for their hardware. From the start of the migration the vendor with the most interest in the industry movement has been Hewlett-Packard.
Much of the early work in migrating companies like Pixar, DreamWorks and Axyz Animation was helped in no small part by the energetic team at HP's Ft. Collins graphics group.
Both SideFX's Salvini and Axyz's Coldrick praised HP for their help in getting the initial port of Houdini working. Recognizing the lack of accelerated hardware support in XFree86, HP ported their own server from HP/UX over to Linux along with a supporting OpenGL environment. This solution, while not completely open-source oriented, moved the process along to the point prior to the first VES Linux Summit.
Nothing Real had ported their Shake compositing software to Linux by April 2000. The company would have had it sooner, except for the lack of hardware-accelerated drivers. They credit HP for helping them make the move to Linux, as well as video card maker NVIDIA, which they say moved more slowly but eventually came around to providing proprietary drivers of their own.
But while these early offerings from companies like HP and NVIDIA included some closed-source software components, such situations are considered interim solutions only. Ed Leonard says that the industry is willing to accept these short-term, closed-source solutions because of the early stage of the overall migration. But they still want to see long term strategies that include open source. "We've said to vendors like HP, 'In order for us to partner, we really want to see you embrace Linux and open source.' That gives the industry more flexibility in choosing hardware. The industry is driving open-source solutions from vendors." And HP, for their part, agrees. They have already noted their desire to exit the X server business, leaving that work to both the XFree86 group and video card makers.
Solution providers such as Hewlett-Packard and IBM offer VFX studios complete system solutions running various distributions of Linux, with the industry default standard being Red Hat Linux. HP's digital content creation (a.k.a. DCC) systems are the x2000 and dual processor x4000, and both are certified for use with Red Hat. IBM's Linux Digital Studio Solution is that company's DCC offering and includes their IntelliStation M Pro workstation, the eServer xSeries for rendering duties and various other hardware for storage options. Threshold Entertainment produced the Berkeley Breathed short Edward Fudwupper Fibbed Big, shown this year at SIGGRAPH 2001, entirely on IBM's offering.
Pixar's PRMan, Alias|Wavefront's Maya, Nothing Real's Shake, Silicon Grail's RAYZ and SideFX's Houdini are all certified to run on Red Hat. Softimage, which makes the XSI compositing package, plans to support multiple distributions. But there has to be a limit to the number of distributions they can support simply based on the requirements of having those distributions ready and available for testing. Each distribution has a good chance of working, but the management of all those distributions would be too much for any application vendor to handle.
For this reason studios will continue to rely on single source vendors. Leonard says that in DreamWork's case, partnering with HP for their initial work just made sense:
They owned the entire stack, just like SGI did. Not by directive or choice. It just happened that they manufactured the hardware, such as their FX line of graphics cards. They had their own drivers for their graphics cards. If we had a problem, they could own that problem through to resolution.
But eventually, when open-source drivers are readily available for the NVIDIA or ATI cards of choice, DreamWorks and other studios will expect to get the open-source solutions from any vendor. The studios can keep a single source to get the complete solution if they choose, they'll just have more than one vendor to choose from to get it.
The industry has learned that the secretive nature of their business is in its twilight, soon to be replaced by the more sharing nature of the open-source world. The cooperative spirit required to move the industry to Linux is allowing studios to solve common problems more quickly. Pixar's Peachey says his studio welcomes the change:
We're all competitors in one sense or another, but [migrating to Linux] has helped us share a little more than we historically have about our thoughts and plans. Not surprisingly, those problems are very common across the many studios. We're starting to see that if someone solves a problem that isn't central to the art we do [as individual studios], and there is a technological component to that art, but many of the things we have to solve are not related to that art, there is suddenly a feeling we can all benefit from it. That's encouraging. It's fun to see.
DreamWorks' Leonard says he'd still like to see the Open Source community look toward entertainment as a partner in innovation as well as recognition:
One of the hard parts of dealing with open source is that it's still viewed as a bit of a hacker's world, as long as you're willing to hack at the code you'll get what you want. A reality for the VFX industry is that as real businesses we need to find a way to channel the talent in the open-source world so that we can get value from it.
Leonard adds that the industry is probably easier to work with for open-source developers than other industries might be:
The VFX industry is willing to take risks. Our solutions don't have to be wrapped with a pretty bow--we're willing to work together to make these things work. I think the application for what we solve together with open source can be applied to a much larger community.
Ray Feeney agrees. "We're the canary in the mine shaft", he says of the willingness the industry has to push Linux in new ways.
Still, many in the industry have concerns about working with open-source developers. One early example of conflict came about while making the GIMP, the open-source world's answer to Photoshop, better suited to the needs of film industry. GIMP is highly thought of in the industry, but the 1.2 version lacks 16 bit color channels, something the industry needs to maintain the high quality color found in films. A separate Hollywood branch of GIMP was created to solve this problem, but the solutions it offered, which were made available back to the Open Source community, weren't accepted directly by The GIMP developers. Instead, they opted for a redesign that would be more practical over the long term, but also not available for a much longer period (they still aren't available more than a year later).
Unfortunately, says Pixar's Peachey, this left a bit of a bad taste in the industry's collective mouth:
Extensions were done by the VFX industry but weren't picked up by the project. This left some members of the industry feeling that because we're a market segment different than what open-source developers are interested in, the industry can't get any open-source developer seriously interested in what the industry needs.
"The industry", Peachey says, "is interested in doing open-source work, but if you do the work and no one accepts it, how do you get that work to survive?"
Feeney agrees. "The Linux community works well with the development savvy, but how do you get film makers and open-source developers communicating? For the industry, it's a complicated issue." For example, The GIMP always has lacked a professional color management facility. There are plenty of external proprietary systems that are available and could (with the right technical work) be added to the program. But there are logistical and political issues to resolve as well, such as how those proprietary systems can be made to work with GPL code like The GIMP.
Does the VFX industry know what the GPL is or what it means and how it relates to proprietary software? Feeney says probably not. "In our community, the open-source push is great when you need to reach consensus and standards. But what our customers do requires differentiation through their own specialized approach. There is a big difference between open source and shared source." Feeney thinks that the application level that the VFX world works in is more of a shared-source world.
Despite concerns on how to work with open-source developers, the VFX industry continues its wholesale conversion to Linux. The operating system gives them more control of their own futures, both individually as studios and collectively as an industry.
At this point most of the major tools used by the industry are available. The major conference and tradeshow for the industry, SIGGRAPH, was full of Linux offerings in August 2001. This includes modeling and rendering tools, tools for distributing rendering work and 2-D compositing tools such as Avid's Softimage XSI.
While VFX is the first major industry to adopt, others may follow very soon. If Ford does deploy 33,000 Linux desktops, industry insiders say that it will likely push GM, Daimler and others to make similar moves due to the cost competitive nature of the auto business. And that will make Hollywood's Visual Effects industry move look like a silent film.
Web Sites
News Stories
IBM's Linux Digital Studio Announcement
The Little Engine That Could, Millimeter Magazine
SIGGRAPH 2000: News and Views From the FloorPixar/PDI debate in 1999 which left the industry OS-agnostic turned to a push for Linux, thanks in part to the backing of Alias/Wavefront.
Michael J. Hammel is a noted author with books on The GIMP and GTK+ to his credit, a graphics artist wannabe and a software developer currently living in Houston with his wife, Brinda, and daughter Ryann. Most recently, Michael worked as a senior editor for Linux Weekly News (LWN.net).
email: mjhammel@graphics-muse.org
