Drive up and down the 101 Freeway in Silicon Valley, or cast your gaze north toward Seattle, and media companies, which expect to book over $20 billion in advertising in 2011, appear to be everywhere. But visit the biggest of these companies and ask them to define themselves, and you’ll be hard-pressed to get them to say they’re the new generation of media, attracting audiences for the purpose of selling advertising.

“We’re a core technology company,” says Nikesh Arora, Google’s chief business officer, whose team netted $10 billion in U.S. ad sales for the search giant last year.

“I don’t know [if we’re a media company], and to be honest, I’m not that interested in the answer,” says a slightly annoyed David Fischer, Facebook’s vice president of advertising, who sold $1.8 billion.

“Microsoft is first a technology company,” says Keith Lorizio, the head of sales for the software giant’s advertising unit.

The exceptions? Yahoo and, on the East Coast, AOL—both once seen as tech leaders, but now laggards, trying to remake themselves as media companies. “I embrace it,” Ross Levinsohn, who joined Yahoo last November to run its Americas operations, says about the media label. “The media business is the most exciting business in the world.” But you won’t find any tech platform worth a few billion or more that admires Yahoo, AOL, or, for that matter, anybody else in the media business.

Advertisers and other publishers have pointed out for years now that assembling audiences, and selling advertising against them, makes these companies, ipso facto, media businesses. And that to survive when the inevitable next big innovation makes mass selling even cheaper, they have to lure big brand dollars onto their platforms, something they’ve struggled to accomplish so far. To do this, they may finally be forced not only to embrace the media label, but actually act like media companies, building context as naturally as they build functionality. It’s a need the technology companies are just beginning to grasp.

Premium brands have always depended on the context that media provides to get their messages across. And they want partners who recognize this too. As Levinsohn notes, it’s why Thursday nights have historically been big programming nights on television: The networks know they can get their biggest premiums if they put their best programs—or at least the ones most popular with audiences who have disposable incomes—on that night because movie studios and the auto industry and every major retailer want to prime the pump of weekend shopping and entertainment. It’s why David Ogilvy ran his most memorable ads in the pages of The New Yorker in the ’50s and ’60s, when that magazine was the lifestyle bible of every sophisticated suburban housewife. It’s why corn lobbyists buy Sunday talk shows and not soap operas. Context is all.

And if the digital world is ever going to build a $200 billion digital advertising market (which Google’s Eric Schmidt has declared as the industry-wide target in the next 10 years), it needs to create the 21st-century version of that context—environments that can help build brand identities.

Underneath a veneer of success, the digital media industry as a whole has masked a growing existential crisis. CPMs on the Internet continue to drop like a stone, and most brand advertisers have resisted moving their dollars into digital, to the point that a now infamous $60 billion gap has opened up—the amount of additional money that, given the amount of time consumers spend with the medium, say industry analysts, advertisers should be spending on interactive media today.

The existence of the gap raises an uncomfortable question. How committed to the advertising business can the management at any of these companies be if they’re letting $60 billion in potential revenue fall through the cracks?

Advertisers are used to having a symbiotic relationship with traditional media partners, and each side understands the other’s roles. The media partners build content and charge premium prices to appear alongside it; the marketers get their desired audience(s)—consumers willing to pay for their brands. That symbiosis hasn’t developed yet in digital, however, and agency executives can sometimes sound like they’ve reached a breaking point in their frustration with Silicon Valley’s refusal to claim the media mantle and fully embrace advertising as their raison d’être. “[They’re] the new media owners, masquerading as technology companies,” WPP chairman Martin Sorrell recently told Adweek.

The reason they haven’t behaved like media companies, yet, has a lot to do with the culture of where they’re based. Indeed, the engineers and the engineering culture of the Valley have always had a certain distaste for advertising, and for advertising salesmen (the kind of people who usually end up running media companies)—too messy, too intrusive, and, worst of all, too dumb. As any Silicon Valley VC who first invested in one would tell you, the goal of a technology company is to develop an essential product that will gain a dominant market position—ideally, so essential that generating revenue off it is more about collecting rent than about selling. And it’s the tech rent collectors who draw the highest valuations on Wall Street.

Media companies, on the other hand, are different, and the VCs know it. Get typecast as a media company—a mere seller of advertising—and you get a media company valuation. Which makes it a great irony that advertising is the only way the Silicon Valley platform companies have ended up making their money.

Their discomfort with this may be why, since before the time Google built AdWords, these tech companies have often seemed more attracted to fixing the flaws they see in the advertising business model than to making it easier and more effective to advertise a brand online. (Steven Levy, in his book In the Plex, writes that “contempt for traditional advertising permeated Google from the top down.” As Eric Veach, the engineer behind AdWords, put it to Levy, “I hate ads.”) 

“They’re always looking for a black box solution to everything,” says one annoyed advertising buyer about the Silicon Valley companies. Veach admitted as much to Levy when he described Google’s formulas for getting your search ads deemed “relevant” as something “you can’t easily explain to advertisers.”

The question now: Can the dominant players in Silicon Valley finally do the kind of explaining, and the kind of selling, to Madison Avenue that advertisers want them to do, and hence attract big brands and big brand margins?

On the surface, digital publishers appeared to have had a banner year in 2010. Total ad sales were up more than 16 percent from $22.7 billion to $26 billion, according to the Interactive Advertising Bureau. Digital has now eclipsed print to command the second-biggest audience in time spent on media, surpassed only by TV. But to this day, most of that money has been dollars shifted from direct response. As the IAB’s CEO Randall Rothenberg says, the Web has “been replacing the U.S. Postal Service.”

The $60 billion that digital “should” be getting is almost entirely money from big brands. As the 2011 upfront season demonstrated, advertisers were perversely willing to pay higher CPMs for the reach they could get on TV—amounting to more than $18 billion in immediate commitments—even while the overall TV audience has declined.

Meanwhile, premium pricing for digital inventory is becoming more elusive by the month. CPMs continue to trend downward, according to the IAB, not only for the much-loathed but still widely deployed banner ad, but also for the newer display ad units that are supposed to be luring brands online. The 16 percent growth in 2010 came courtesy of what Booz & Company consultant Chris Vollmer labeled two years ago as the “low hanging fruit”—stolen much more from below-the-line forms of traditional direct marketing, junk mail, and the Yellow Pages, than from those lucrative brand budgets.

Those direct marketing offers have been placed against an infinite universe of inventory that continues to expand. Rothenberg, speaking at a recent event to IAB members—the people who sell digital advertising space, including the big platform players—said, “What we’ve spent a lot of time and a lot of our energy doing is pushing tonnage out into the marketplace.” And the tonnage is what’s driven down CPMs, and thus potential profits.

Big brands continue to spend relatively insignificant parts of their ad budgets on digital: In 2009, Procter & Gamble, for example, devoted only 7 percent of its ad budget there; at Unilever it was 3 percent, according to IAB figures. In P&G’s case, that percentage actually declined in 2010, by Adweek estimates, to 5 percent, even as the total dollars tripled to $169 million. Meanwhile, Unilever CMO Keith Weed claimed at Cannes last year that the company would merely double its digital budget in 2010.

One big reason for the relatively small digital ad spend is that brand managers don’t want their ads caught anywhere near much of the existing online inventory, derived as it is from the direct marketing world of junk mail (the ubiquitous LowerMyRates.com banners, for example). “We’ve got decades and decades of experience that says building my brand drives sales—and all that direct response and discounting does is drive my brand into the toilet,” says one former brand manager.

And yet it’s their exploitation of all that direct response inventory that has Google, Facebook, Yahoo, and Microsoft (which is a Silicon Valley company in temperament if not in physical locale), with their vast scale, pulling away from the rest of the competition online—together they’re expected to grab some 65 percent of the digital ad market in 2011, according to eMarketer, and as much as 70 percent in 2012. It’s the ad sales equivalent of rent-seeking, on inventory that is often the digital equivalent of a tenement slum (and, as the CPMs demonstrate, priced accordingly).

All that excess inventory keeps prices down at the expense of just about everybody producing content—but ultimately, at the tech companies’ own expense, as well. It’s what’s fueled the growth of content farm companies like About.com and Demand Media, for example, whose response to low CPMs has been to produce more low-quality, low-cost pages—pages that actually reduced the value of Google’s own search results.

A growing proportion of these tenement transactions are taking place via the dizzying array of automated services—exchanges, demand-side platforms, supply-side platforms, agency trading desks, real-time bidding—that have turned much of the “tonnage” referred to by Rothenberg into a low-margin commodity business. Digital advertising people these days often sound more like Wall Street quants than Mamet pitchmen.

The ad agency holding companies, used to getting their piece of media buying through the traditional firms that they long ago split off from their creative agencies, have invented their own quant systems to trade with the supply-side platforms that Google and the others have devised (and, perhaps, to hedge their bets about their own inability to deliver for their brand clients in digital). Sorrell’s WPP, for example, added the latest iteration to this in late June when it unveiled Xaxis, a subsidiary intended to buy digital audiences in bulk.

David Kenny, the since-departed founder of VivaKi, Publicis’ digital arm, which launched the first agency trading operation, acknowledges the need to create better context for brand advertisers, but is unapologetic about the commoditizing effects of the trading platforms decried by Rothenberg. “Content commoditized awhile ago,” he says. “I don’t think the [agency] platforms created a Demand Media. And if the media has become commoditized, all a trading platform does is expose it much more quickly.”

Arora, who manages revenue operations at Google—which took in more than half of the available online ad money last year through its mastery of these automated systems—turns almost gleeful when discussing the opportunities in disaggregating audiences from content and indeed context. “We’re selling network,” Arora says of the 2 million Web publishers Google reps via AdSense. “To me, that’s a longer sustainable business, because I can build a network based on how many publishers I can acquire [as Google clients].” If Google doesn’t have enough men 18 to 24, for example, it just expands the pool of sites that draw them.

Only a fool, Arora seems to suggest, would want to be caught in the business of trying to attract an audience, the way his competitor Yahoo does just down the road in Sunnyvale, Calif. Compared with Google’s network scheme, in which it simply adds website publishing partners whenever it needs more audience, “Yahoo can’t grow,” Arora explains, leaning forward on a couch in his Mountain View, Calif., office and bouncing a small toy ball. “Their media business is going to be dependent on the number of viewers of their content properties. You can tell me how they’re going to fare,” he says, dismissing the possibility of a positive answer with a laugh.

Of course, both Yahoo and Google have hedged their bets here: Yahoo is now trying to develop its own network of websites through an affiliate sales organization; it’s also gotten into the content-farm game with the purchase of Associated Content. And Google itself appears to recognize the need to create owned-and-operated premium spaces for advertisers; witness its recent additions of Hollywood content to YouTube and reported pursuit of Hulu.

Beyond Yahoo, others representing digital companies who accept the media label (particularly those grown out of traditional media companies) dismiss an overreliance on automated trading schemes to sell long-tail inventory as quickly as Arora dismisses Yahoo’s reliance on its own content. The skills required to make a market for stocks or bonds or pork bellies, they suggest, aren’t the same as those required to build brands. “It’s either about trading commodities, which means the people most adept at trading those commodities will win, or it’s about strategies and client relationships,” is how one digital media executive puts it.

Curt Hecht, Kenny’s former colleague who oversees the automated ad trading services of Publicis’ digital properties through its VivaKi Nerve Center, agrees that ad platforms won’t ever be enough for the ad world’s most important clients: “I think on the branded side of it, when it comes to building ideas and experiences, I don’t think that can ever be standardized. I think we do want a level of customization for it.”

Rothenberg is even more pointed. “If the economy were based solely on efficiency, we’d have to bleed all desire out of it,” he says. Responding to desire, Rothenberg suggests—and more importantly, prompting it—is what brand advertising has always been about, and the media business has always been about creating a place for that kind of advertising to work most effectively. It’s the difference between a low-margin business and a high-margin one.

To different degrees, Silicon Valley players—even those protesting otherwise—understand the need to create spaces apart from those managed by automated selling tools, so they can give brand advertisers a chance to stimulate desire. What remains in question is their ability to deliver on it.

Desire demands storytelling, which is why brand advertising has always been intimately involved in the creation of mood, emotion, character, look, and feel. The people who run digital companies for the most part don’t have those talents (except to the extent look and feel is required in a software interface), and aren’t interested in them—and only recently have they started to recognize the need to add employees who do.

“It’s really about you and I remembering that awesome Nike ad from three years ago,” says Neal Mohan, who oversees Google’s display business, about the emotional impact a brand can have. But Mohan also notes that creating those sorts of experiences is still far too expensive in digital, with as much as 28 cents of every dollar spent online lost to the costs of actually producing ads, versus 2 cents in television, leaving much less of the relative budget for buying eyeballs.

No one talks more smartly (or more eagerly) about the need to build a better brand ad environment than the folks at Microsoft Advertising, but they’re doing so while trying to shed a reputation for ineptitude in the industry. Part of that reputation has come because the company’s byzantine structure for managing advertising sales has undergone multiple reshufflings in recent years; the latest resulted in longtime Microsoft supply chain manager Frank Holland installed as the head of the sales unit. (And yet even after the reorg, a chunk of staff responsible for many advertising services remains in a parallel organization, reporting to Online chief Qi Lu.)

Microsoft Advertising’s head of U.S. sales, Lorizio, who oversees a team of 180 selling everything from MSN to the Xbox, is candid about the difficulties. “We have not made it easy and scalable for buyers to be able to buy,” he says. Like its competitors, Microsoft attempts to make things easier for brand advertisers by absorbing much of the friction cost described by Google’s Mohan. Lorizio’s colleague Stephen Kim, for example, runs a creative team devoted exclusively to working with some of Microsoft’s largest advertisers, such as Lexus, on custom products.

Meanwhile, Facebook, the new kid on the block, is still trying to figure out exactly what it is, even as it floods the market with cheap display inventory and drives down CPMs for everyone. It has built its own custom creative unit, and Facebook’s David Fischer has a well-honed pitch on the value of social media for brand building.

But VivaKi’s Hecht suggests Facebook’s future path might lie in positioning itself more as a research platform than as an ad venue. “[They’re] definitely a consumer insight company,” he says. “They can survey at a census level if they’d like.” On the other hand, that risks the potential data overreach that angers users and invites government regulation.

Kenny, the VivaKi founder, is no longer in the advertising business, having become president of Akamai, the Internet traffic management company, in 2010. He’s also a recently installed Yahoo board member. He warns observers not to get too caught up mistaking a snapshot of what’s happening right now for the future, or even worrying about the $60 billion gap when TV itself is migrating toward the Net.

“Would there even be such a story 10 years from now about digital media, and television?” he responds, when asked to explain how the gap will be closed. “Because it will all be digital. And it will all have the capability of targeting, of segmentation, of better metrics, of new levels of engagement, of different kinds of interactivity.”

But the brands still have to follow, and Rothenberg isn’t as sure as Kenny that they will. “When all television is addressable,” Rothenberg says, “when magazines and newspapers are moving online, when that time online goes above 50 percent—if the brands don’t find what they want, then you reach a full-blown crisis.” If brands can’t find their customers—or worse yet, if the customers are spending too much of their time visiting places where, as Rothenberg says, “marketers are just simply uncomfortable following,” whole industries—but especially packaged goods—could be in trouble. Along with the media business itself.

Which brings us back to Yahoo, which built its business in the ’90s via ad banners, as a low-price CPM operation that reached mass eyeballs. Google trumped Yahoo by reaching even more eyeballs via search, then used text ads to drive banner pricing into the dumps. Now Google faces the same prospect for its burgeoning display ad business when it looks at Facebook. Yahoo is trying to go back to the hard work of turning itself into a more valuable and higher-margin media company—but it may be too late, a lesson the newer platform companies might want to note before the Next Big Thing comes along and makes it too late for them as well. On the other hand, if you’re Facebook and you have what seems like unlimited impressions growing faster than their cost is sinking, why would you? Especially with an IPO coming—the present values you too highly to worry about the future.