We live in an era of big data. Enabled by technology capable of tracking every consumer sale and even the whereabouts of consumers themselves—within two feet if you’re running certain apps—data capture drives growth across industries. It’s not a coincidence that your favorite streaming service is teasing you with that new series you Googled last night, or that the banner on The New York Times hit you with a ski pass sale shortly after your last day on the hill. Knowing who buys what, when, and where is a clear advantage—an advantage the outdoor industry just does not have. I’ve spent my career reporting on the outdoors. There’s something very fishy in our numbers.
Hell, we can’t even get the easy point-of-sale (POS) stuff right. At least, that’s true if you believe the folks behind Grassroots Outdoor Alliance, which acts as the unofficial voice of much of outdoor specialty retail.
This past January was a WTF moment for Grassroots. The dispute started with a column written by Dirk Sorenson, an analyst for the consumer research firm NPD Group, in the January 2021 edition of Outdoor Retailer magazine.
Most of the commentary was innocuous and obvious—consumers wanted to get outside during the pandemic. But in paragraph six, Sorenson slipped in an incendiary statistic. Outdoor specialty retailers, he wrote, were down a staggering 32 percent through October of 2020. He followed that up with a vague claim that “outdoor specialty retailers have faced challenges due to store closures.”
Grassroots—a collective of independent specialty retailers—was tracking different numbers. Over the same time frame, its research showed top-line sales for the 196 storefronts in the group down just 2.18 percent. This, during what The Washington Post called the worst economic downturn since World War II. As for permanent closures, Grassroots lost only one shop—due to retirement. Grassroots was done with the tired narrative about brick and mortar dying. Ditto with the running oversimplification that specialty brick and mortar and specialty e-commerce are disparate entities; 50 percent of Grassroots shops run e-commerce platforms. More than that, Grassroots disputed the notion that NPD’s analysis speaks for what most industry people think of as specialty retail. Of 73 stores Grassroots surveyed (the coalition has since grown to 96 members) only two reported to NPD.
In a letter to Outdoor Retailer and NPD, Grassroots demanded clarification and an apology. The magazine published their demands. Some back and forth between Grassroots and NPD followed, but Grassroots wasn’t satisfied with NPD’s counterargument. There’s too much at stake, said Rich Hill, Grassroots’s executive director. If a CEO on the vendor side believes that specialty is in trouble, Grassroots asked, what happens to the co-marketing dollars or the test products designed for specialty shops and their opinion leader clientele? When CEOs rely on incomplete or just plain wrong retail sales data, Hill said, those types of investments get cut. In effect, the prophetic narrative “specialty is in trouble” fulfills itself.
“NPD put out a misleading statement about the health and wellness of our industry,” Hill said. “That’s all that CEOs read. Somebody has to say it: nobody that we talk to in outdoor specialty retail trusts their data. The path to redemption starts with an apology.”
Whether derived from political pollsters or POS transactions, data are usually taken as fact—it’s human nature to assume there can be no nuance in projected numbers; no sampling errors; no muddy language in surveys. But it’s high time we embraced some skepticism. In politics, we now know those most likely to answer a call from an unknown number are older liberals. Even with robust margins of error, when a sample isn’t representative, neither are the projections. The more you extrapolate, the worse it gets. Something similar might be at play with the NPD and Grassroots misfire. Do the Grassroots and NPD definitions of specialty retail even line up? It depends on whom you talk to. NPD says it has a good handle on outdoor specialty. Grassroots says it’s not even close.
But the story of data in the outdoor industry is bigger than the recent spat. Tracking participation is even tougher than the POS stuff. Unless we’re selling tickets, booking campsites, or issuing licenses, what is there to count? And what about all the people taking advantage of free access to our public lands? All we can do is estimate the number of people running, hiking, backcountry skiing, climbing, paddleboarding, and biking. And frequency is even harder to track—after all, there’s no turnstile at the trailhead. For an industry that prides itself on its bona fides, we often don’t have a clue about what’s actually happening outdoors.
How Do You Count an Outdoorsperson?
My favorite participation stat comes from telemark skiing. Back when I was the editor of Skiing magazine in the mid-aughts, Outdoor Industry Association (OIA) estimated that in the U.S. there were 4.2 million telemark skiers. Sound accurate? There were only 7 million total skiers then. The slow hippies in their Peruvian hats would have outnumbered snowboarders two to one. Of course, this was total bunk. Even in tele hotbeds like Telluride, Colorado, alpine skiers dominated. I knew this because I looked around lift corrals.
My editors at Skiing tried to investigate the telemark glitch. But with OIA standing by its data (they have new researchers now, by the way), we were left to surmise the survey respondents confused telemark skiing with telemarketing, which had reached critical misery at the time. As in: “oh hell yeah I’ve been telemarketed. I can’t stand those people.” Just as peculiar, though, in 2007, OIA’s Outdoor Foundation (OF) counted 1.7 million telemark skiers, but by 2016 that number was back up to 2.8 million—this during a stretch when anyone in the ski business would attest that telemark skiing had fallen off a cliff.
This stuff matters to the outdoor industry. Do you really want to be producing telemark boots, or trail running shoes, or expedition backpacks with a cloudy estimate on participation numbers? It also matters for advocacy. “If you want new trails,” said the International Mountain Bicycling Association’s executive director Dave Wiens, “there are many boxes to check along the way. One of those is building community will. That’s true nationally and locally. Mountain biking is a hard sport to really calculate user days on. Right now frequency is going way up because of better bikes and trails, but the industry numbers don’t reflect that. Which means it can be hard to communicate to someone how important trails are to people. Better data would help.”
The trade groups and their research arms have done better lately, but I still see suspect numbers. Backcountry snowboarding is one example. Snowsports Industries America (SIA) counts 650,000 “snowboard tourers” in the U.S. For perspective, that’s only 50,000 fewer snowboard tourers than backcountry skiers. But even though splitboard sales have been strong for years, those sales don’t add up to 650,000 users. (And certainly not to the 1.5 million snowboard tourers that OF tabulates.)
Since 2016, SIA told me, the splitboard market has done roughly $17 million in retail sales. If each splitboard sells for $800, that’s 21,000 splitboards sold. Even if you doubled that number ($34 million top line in splitboard) by going back to 2010, that would mean that 42,000 people bought splitboards in the last decade or so. Let’s be generous and say that another 100,000 snowboarders who hike the backcountry in boots or snowshoes, or on approach skis (haven’t seen that in a while) identify as “snowboard tourers.” Rounding up, that gets us 150,000 snowboard tourers. Maybe. Brendan Madigan, owner of Alpenglow Sports in Tahoe City, California—the healthiest snowboarding market in the country—told me that he sells skis to snowboards at a ratio of 30 to one. As for participation, said Madigan, in Tahoe it’s more like seven to one. “There’s no way those participation numbers are accurate,” Madigan said. “Snowboard touring has grown, but the sales don’t come close to those estimates.”
It’s this type of calculus—mine, not SIA’s—that people like Adam Howard, the publisher of Backcountry magazine, do all the time to gauge market size. “Frankly, we’ve never trusted the numbers that OIA or SIA provide,” Howard said. “Our best numbers have always come from talking to friends in the industry who make and sell gear. If passionate backcountry skiers—the ones we focus on—burn through AT boots every three years, we can get a feel for how many of those skiers are out there.”
Trail running is another tricky one. With 11.8 million trail runners in 2020 according to OF, it would seem that running on dirt is leaving the stratosphere—doubling participation since 2012. OF researcher David Mudd told me they feel good about the data. After all, the survey goes out to 18,000 people.
But do the numbers bear out? I’d argue they don’t. And here’s why: simply owning a pair of trail running shoes might make a survey respondent identify as a trail runner. “There is no way that the trail running numbers are accurate,” said Wes Allen, co-owner of Cody, Wyoming’s specialty retail shop Sunlight Sports. “Not to denigrate trail running, because participation is certainly up. But most trail running shoes are worn in grocery stores. Bad analytics have warped the outdoor industry. And worse, it distracts the industry from what it should be doing. Instead of focusing on trail running, what if, as an industry, we were talking about trail access and diversity? We wouldn’t be so far behind the eight ball right now.”
Imbroglio Revisited
And then we have the Grassroots and NPD squabble. To an outsider, it might seem overblown, but to the Grassroots crowd it isn’t. Grassroots retailers are convinced there’s a fundamental disconnect between the shops NPD collects POS data from and the ones that Grassroots considers specialty retail. In fact, said Grassroots’s Hill, the two may barely overlap. “The way they describe specialty retail and the way we describe it are two different things,” Hill said.
NPD is a global corporation that runs market research on more than 20 industries—everything from toys to makeup. The group collects POS data from more than 600,000 retail locations and issues more than 12 million consumer surveys each year. Under the NPD umbrella you’ll find sports. And under sports you’ll eventually find specialty outdoor. (NPD claims 1,450 sports specialty doors, including specialty outdoor, cycling, snow, and run.) Contrariwise, Grassroots, as its name implies, is a bottom-up collective of independent specialty retailers. If, as Grassroots asserts, NPD is missing its type of store, that’s a problem.
It’s in proving or disproving that claim that things get nebulous. Because NPD won’t share its independent outdoor specialty retail list, all we really know is that when Grassroots interviewed 73 of its members a few months ago, only two were reporting POS data to NPD. While NPD asserts that its take on outdoor specialty retail only includes stores with five doors or less with a core focus on the goods shown at the Outdoor Retailer trade show, independent analysis done by people like Allen and Hill make it easy to question that claim.
Case in point: the products NPD says are top movers are sometimes duds within Grassroots. By way of example, Allen singled out a tent from a few years back. He wasn’t aware of much buy-in by outdoor specialty, but NPD data called it a top performer. Upon asking another retailer about it, he learned that a bunch had been dumped on closeout. He also learned that the tent maker hadn’t even produced as many tents as NPD predicted it would sell.
Now imagine you’re a tent maker. Should you build a tent to compete? If you’re a specialty retailer, should you buy such tents? On the flip side, Allen strongly suspects that major vendors (he doesn’t want to say which) have killed or defunded successful specialty product launches because of incomplete data like this. Part of this is specialty’s fault—they haven’t been as good as Amazon and Backcountry.com types at capturing sales live—but part of it, said Hill, is on NPD and how outdoor specialty retail is defined.
The next level of confusion, said Grassroots, arises when NPD produces outdoor specialty analysis based on such data. NPD’s analysis is often obfuscated with lines like this: “over the last year, the sports industry has exceled [sic] in using unrequited demand to drive consumer interest.” But NPD’s logic frequently doesn’t track, either. Information vacuums, like the confusion between brick and mortar and specialty e-commerce, are common. Because NPD often doesn’t do a good job of explaining its data and analysis, the fallout can be confusing. Would you know, for instance, that the category “accessories” includes backpacks, bags, and duffels? Or that NPD doesn’t “see” sell-in data, only sell-through?
Which gets us to yet another Grassroots beef: confirmation bias. Because NPD has been so bullish on e-commerce, are they too beholden to the trope that if e-commerce is strong then brick and mortar must therefore be weak?
Sorenson denied the allegation and described himself as optimistic on independent specialty retail. In fact, Sorenson disputed most of the allegations I presented. The discrepancies in product performance between Grassroots and NPD, he said, are due to the fact that NPD’s sampling is larger, more diverse, and includes retailers that aren’t part of Grassroots. Sorenson also discounted the weight of NPD analysis among outdoor industry CEOs. “Many of our retail and manufacturer partners dive far more deeply [into the data] than an article,” he said. “It’s far more robust. Those conversations aren’t based on a single article, but more on deep analysis.”
Nuance Beats “Data,” but Good Data Helps
Alpenglow’s Madigan recently told me that retail is sort of like legalized gambling. You study the market and your customers, and place your bet. The same is true with the industry at large. With so much uncertainty and so much at stake, it’s natural that we’re attracted to numbers. When we’re looking to defend a decision—to our bosses, investors, or families—numbers feel like our best armor.
But are they? A ski product manager who wanted to remain anonymous* told me that when the market research firm Leisure Trends was sold to NPD in 2013, the quality of some of the data he got via SIA immediately sank. (SIA partners with NPD for part of its research.) As with outdoor, he said, NPD’s winners and losers in the gear wars just weren’t matching up with what he was seeing. A bike industry source on the vendor side said much the same thing, adding that when he’s in the business of identifying trends, he isn’t turning to NPD—he’s working the phones. Yet another anonymous source, this one a market researcher, put it more bluntly: “NPD data have long had a blind spot in specialty. I think that’s agreed upon in the industry.”
NPD disputes that the quality of its specialty retail data declined when it acquired Leisure Trends. On an initial phone interview, NPD’s David Riley called this idea “sour grapes.” Sorenson was more careful. By increasing the scale of the sample and introducing more rigorous methodologies, he said, NPD’s specialty data grew stronger. But by expanding to that larger collection of shops, the numbers also changed. They would have to. “I could see that causing some disconnect,” Sorenson said. “The old [sample] might have been a really targeted subset. [But now] NPD allows retailers to compare [themselves] to the aggregate—the rest of the market, not just a small subset.”
That, of course, brings us back to the fundamental question: Is NPD talking to specialty retailers as we know them? Again, Grassroots says no way. And as a result, Grassroots is in the midst of building its own market research tool to better serve its member shops and vendors. The platform, called Switchback, is the brainchild of Hill and Greg Squires, the founder and CEO of Pivot Point Solutions. The seed of the idea, though, came from work that Squires did for a similar category—independent booksellers.
Like specialty outdoor, ski, and even bike, indie booksellers had long lived under the cloud of a narrative that spelled their doom. But the prognostications didn’t match reality. Between 2009 and 2018, new independent bookstores grew by 49 percent. This, while chain retailers lost storefronts and Amazon consolidated its power. Could it be simultaneously true that independent bookstores could thrive while the world’s largest bookseller did, too?
The answer is yes. Business is nuanced, and tired tropes are symptomatic of lazy thinking. A Harvard Business School researcher made a case study of bookstores in 2020. The takeaway? Indie bookstore success could be attributed to “community, curation, and convening.” Meaning they served specific customers, found products that worked for them, and opened up their stores as gathering places. To Grassroots, that sounds like modern specialty outdoor retail.
To those three Cs I’ll add a fourth: capture. To push back on doomsday narratives, one needs solid data. With Switchback, if a product is flying off shelves somewhere, the rest of the network will see the trend coming. If vendors want to know how a flagship jacket is performing with early adopters, they’ll know in days. There will be no projections made based on a sampling. Thus far, Hill and Squires have signed on 82 Grassroots storefronts with 65 more in the works and have plans to include a broader coalition in a project called Indie Outdoors. They’ve invested more than $1 million in Switchback. “Extrapolation is often misleading,” Squires said. “This platform does not attempt to extrapolate. The data are the data. And that’s meaningful to the market, to the brands, and to the retailers.”
Live the Lifestyle and Believe Your Eyes
Keeping with the theme, I’m not going to end this piece with projections and extrapolations. The outdoor industry’s data problem is just the reality we live with as retailers and manufacturers, and even guides, magazine editors, and trail advocates. It’s tough to know what’s actually happening in our world.
That telemark anecdote? It dates to a time before the current research team at Outdoor Foundation. OF researchers will continue to hone their craft by cleaning up survey language, excising joke respondents from lists, and diversifying sampling. They have to. OF researcher Mudd once saw stick-and-ball-sports category data that indicated that one in seven humans on earth play volleyball. “Data science is significantly better than it was a decade ago,” he said. “I feel confident in our current data. We’re not leading the witness. Or using confusing terminology. But with any research methodology, as much as you try to perfect it, there will be imperfections.”
That type of humility and the Switchback experiment offer hope. Some healthy data skepticism helps. From what I can tell, nobody is trying to get the numbers wrong. When in doubt, make some calls, ask questions, get outside, and believe your eyes.