"Pre-filters" vs. "Post-filters"
One of the themes that I'm developing in the book is the notion that "a Long Tail without good filters is just noise." But what are good filters?
To begin, I'm using the catch-all term "filters" (which I'm not crazy about; anyone got a better word?) to describe the tools that help you find what's right for you in the massive variety of the Long Tail. The examples I use most often are search and recommendations from either people (be they influential bloggers or just friends) or software, such as Amazon-style collaborative filtering ("people like you bought...").
There are, of course, many other kinds of filters. Rankings (by anything from sales to reviews) tap the wisdom of the crowd to identify quality or value. Everyone loves best-of lists, and playlist sharing is a fast-growing way to discover new music, whether through the good taste of other fans or the questionable taste of Beyoncé. And then there's the role of the critic, tastemaker or editor, for which there is now demand in even the narrowest niches.
But when you think about it, the world is already full of a different kind of filter. In the scarcity-driven markets of limited shelves, screens and channels that we've lived with for most of the past century, entire industries are created around finding and promoting the good stuff. This is what the A&R talent scouts at the record labels do, along with the Hollywood studio executives and store purchasing managers ("Buyers"). In boardrooms around the world, market research teams pour over data that predicts what's likely to sell and thus deserves to win a valuable spot on the shelf, screen or page...and what doesn't.
The key word in the preceding paragraph is "predict". What's different about those kinds of filters and the ones I've been focusing on is that they filter before things get to market. Indeed, their job is to decide what will make it to market and what won't. I call them "pre-filters".
By contrast, the recommendations and search technologies that I'm writing about are "post-filters". They find the best of what's already out there in their area of interest, elevating the good (relevant, interesting, original, etc.) and ignoring or downplaying the bad. When I talk about throwing everything out there and letting the marketplace sort it out, these post-filters are the voice of the marketplace. They channel consumer behavior and amplify it, rather than trying to predict it.
This is an important distinction. In the existing Short Tail markets, where distribution is expensive and shelf space is at a premium, the supply side of the market has to be exceedingly discriminating in what it lets through. These producers, retailers and marketers have made a science of trying to guess what people will want, to improve their odds of picking winners. They don't always guess right--there are surely as many things that deserved to make it market but were overlooked as there are things that made it to market and then flopped--but the survivors get a reputation for some sort of mystical insight into the consumer psyche.
But in Long Tail markets, where distribution is cheap and shelf space is plentiful, the safe bet is to assume that everything is eventually going to be available. The role of filter then shifts from gatekeeper to advisor. Rather than predicting taste, post-filters such as Google measure it. Rather than lumping consumer into pre-determined demographic and psychographic categories, post-filters such as Amazon's custom recommendations treat them like individuals who reveal their likes and dislikes through their behavior. Rather than keeping things off the market, post-filters such as MP3 blogs create a markets for things that are already available by stimulating demand for them.
Here, in chart form, are some examples:
Interestingly, when I consider my own role I find that I do both. As the editor of a magazine with a finite number of pages, I'm a classic pre-filter. I indulge in all sorts of brutal discrimination and guesswork to decide which articles to run. But Wired also does lots of product reviews, and in that respect, we're a post-filter. We look at the universe of what's already out there and bring the best stuff to our readers' attention.
As long as there's a market for a pre-filtered package in the deliciously finite medium of bound glossy paper, I suspect there will continue to be demand for my old-fashioned discriminatory side. But the day when people like me decide what makes it to market and what doesn't is fading. Soon everything will make it to market and the real opportunity will be in sorting it all out.
(Note: If the image at the top of this page were mine, and not just randomly stolen from some site, I'd title it The Pre-Filters. Guys with shades manhandling someone in a limo is pretty much my mental image of the music industry.)



It seems to me that the important differentiator is that that pre-filters are picked for you but that you chose which post filters to apply.
I think there's also a lot to discuss regarding the role of personal recommendation channels - and the people who provide them. Personally, I feel that I'm probably some way along some part of the long tail in terms of the music I listen to. However, I rarely have stuff recommended which I actually like - whilst finding that lots of my friends like music that I originally introduced them to. For me the long tail - and the various recommendation mechanisms like amazon/audioscrobbler/musicplasma are almost universally useless, because it will generally triangulate on something a lot more mainstream than the bulk of my listening - and therefore be almost definitely something I've heard of anyway.
Posted by: Chris Stiles | July 04, 2005 at 03:44 PM
Actually, from the point of view of the user, they are all pre-filters, they are just pre-filters of varying levels of authority.
The actual scarcity will always be the scarcity of time and attention on the part of the searcher. When I want to find out something, or buy something, I go to magazines, web sites, friends, search engines and books at the library. Either I find what I am looking for after a certain expenditure of time and effort, or I give up.
If you are talking about what producers create, then you have a true pre-filter, since a consumer cannot find what does not exist, and instead must find some way for the item to be created or substituted for.
Your list of pre-filters really just consists of high hit "authorities". That is, information sources that have developed a general level of respect in the community, sort of like Thompsons, Edmonds or Moodys. The post-filters are just lower hit information sources with a higher variance in reliability.
If you consider the spectrum of information sources, and start eliminating the high hit, lower variance sources, one generally finds a wider range of possibilities for your search. The high hit authorities can only cover so much ground and still provide good coverage. As they are removed and the range of items covered increases, the variance of the review quality goes up, and eventually you get to the spam sites that are basically scams and of no reliability or use whatever.
In other words, the pre-filter/post-filter is really about scale, focus and variance. It is impossible to be an authority on everything, just as it is impossible to search everywhere in a finite time. Eliminate the more reliable filters with low variance and the noise of variance increases.
Posted by: Kaleberg | July 04, 2005 at 07:23 PM
I agree, you need a better term for filter.
If, as you say, the role of filter is shifting from gatekeeper to advisor, then perhaps a synonym for advisor is what is needed, something like:
guide, counsellor, attendant, chaperon, escort, pilot or--my favorite--mediator.
The media future is a mediated reality.
Posted by: Jay Oatway | July 04, 2005 at 09:13 PM
Here's a related paper I wrote that was published five years ago (in an academic journal). I address some similar points, but put particular emphasis on the role of commercial interests in the filtering/gatekeeping process. It may also be of interest since I point to some academic literature on the topic of gatekeeping (going back all the way to the 50s). I also identify the original location of that famous Herbert Simon quote on attention scarcity. Then last year, this book chapter I wrote came out with some follow-up discussions.
Posted by: Eszter | July 04, 2005 at 11:17 PM
The term we used in the early days of the web was "tastemakers" these are online users be it bloggers or posters that have attain a certain legitimacy by their writings so that people will follow - the thing about tastemakers is that it is difficult for them to gain this tag and quite easy to lose it.
Posted by: smartone | July 05, 2005 at 08:26 AM
The term we used in the early days of the web was "tastemakers" these are online users be it bloggers or posters that have attain a certain legitimacy by their writings so that people will follow - the thing about tastemakers is that it is difficult for them to gain this tag and quite easy to lose it.
Posted by: smartone | July 05, 2005 at 08:27 AM
The term we used in the early days of the web was "tastemakers" these are online users be it bloggers or posters that have attain a certain legitimacy by their writings so that people will follow - the thing about tastemakers is that it is difficult for them to gain this tag and quite easy to lose it.
Posted by: smartone | July 05, 2005 at 08:36 AM
wow sorry typepad posting glitch
Posted by: smartone | July 05, 2005 at 08:38 AM
Your post is interesting but I agree that the term pre-filter and post-filter might not be best.
One is a "pre-production" filter and the other is a "post-production" filter.
So, you are saying that much more content will get produced and become available. That is true.
However, there will be two kinds of "post-filters"
One is the basic search engine (Google) or basic recommendation engine (Amazon).
However, those are all machine based. There is a new form of recommendation emerging wich is closer alligned to what people already do and that is word-of-mouth based (the single most valuable filter out there).
Blogging is part of this trend and so is social bookmarking. I have found great sites using www.blinklist.com because I can see the links my friends are saving and what people like me that are also interested in similar subjects are saving.
Posted by: social bookmarking filter | July 05, 2005 at 01:27 PM
Hmm, well, one word that industry already uses for these "post-filters" is "trendsetter". But then they definitely aren't trying to inhabit the long tail -- they're trying to have a sales chart like Seth Godin's "hero curve". Another name for this is "alpha teen", in the kid context, or just "influencer".
But all of these assume someone is looking to a single person, hierarchically, for their taste choices. A collective taste choice, such as what I get at Audioscrobbler, is another thing entirely. In one sense I'm my own primary influencer, because the process begins with my own profile of what music I like, based on what I actually play.
The term I like isn't so much "filter" as maybe something akin to "pointer", like in a hunting dog. Advisor has some of this quality, but also has connotations of a valet or salesman, both of whom seek to flatter the buyer in some fashion. "Miner" might have some utility, and this is a form of data mining, but also connotes selectivity which this isn't.
Posted by: Dan Hartung | July 05, 2005 at 01:41 PM
How about "prospector" instead of "filter"? It has more positive, and fun, connotations. One imagines grizzled old algorithms out in the wilds of the web periphery, panning for nuggets.
btw, hey Dan!
Posted by: David Pautler | July 05, 2005 at 02:57 PM
As I view it, the term "long tail" implies the existence of a distribution, inventory and retailing system that's so low in cost that it can afford to make items available for purchase regardless of demand. But, on the consuming side, the same term implies the availability of some mechanism that allows consumers to easily and efficiently find items to buy.
You suggest that advisors are one form of such a mechanism. However, singular advisors generally don't scale well. How will the advisor be able to justify the time to evaluate stuff that few will use, when there's stuff of more potential popularity that also demands his/her/its time? The advisor solution simply begs the question.
Voting mechanisms have a similar problem - if there is a really unpopular item that only one (let alone no) person votes on, it won't come up in any listing. Thus, voting really focuses on short-tail - in fact, it defines it. The same outcome is driven by anything based on recommendations based on others' experiences.
Now consider a collaborative filter that analyzes your behavior and finds people with similar behaviors and then provides a hitlist of what they selected, or a recommendation engine that analyzes your own personal choices, parses them into some kind of query and then develops a hitlist from that.
My initial thought is that this won't handle items really far out on the long tail of the inventory. For such items, it would probably not be possible to analyze a person's general behavior and identify a need or interest in such items.
A search engine has more potential, if there were a sufficiently granular indexing mechanism that could capture subtle nuances about the content items when they were added. But to be able to practically take advantage of those nuances via a search engine, the searcher have to be come exceptionally skilled in query statement construction. That's a unlikely outcome. It fails to meet the "easy and efficient" requirement.
The only real solution that comes to mind is, when items are introduced into the distribution/inventory/retailing system, to classify them in a fairly precise way.
Then, using a classification-aware search engine might be easy enough for ordinary consumers to use.
Or, even better, a taxonomy-tracking browse/search mechanism operating at the topic (rather than item) level.
Posted by: Terry Steichen | July 05, 2005 at 04:07 PM
Bo' Selecta!
For when filter is too generic
http://www.boavid.com/faq/faq-question3.php
Posted by: jimmy | July 05, 2005 at 06:19 PM
In the new, non-pre-filtered world, there's another filtering mechanism you haven't mentioned yet: time. Anyone has a finite amount of time to do their own searching and post-filtering activities, this is partly why they want to rely on pre-filtering, or on recommendation systems in the post-filtering phase.
I'm not sure how much of an affect it has - but it must have some.
Posted by: Tim | July 05, 2005 at 07:03 PM
terry said:
> Now consider a collaborative filter that
> analyzes your behavior and
> finds people with similar behaviors
> and then provides a hitlist of what they selected
indeed, that is exactly how a collaborative filtering system
is _supposed_ to work. amazon corrupted the concept.
***
chris said:
> But what are good filters?
...
> playlist sharing is a fast-growing way to discover new music,
> whether through the good taste of other fans
> or the questionable taste of Beyoncé.
...
> And then there's the role of the critic, tastemaker or editor,
> for which there is now demand in even the narrowest niches.
...
> entire industries are created around
> finding and promoting the good stuff.
> This is what the A&R talent scouts at
...
> They find the best of what's already
> out there in their area of interest,
> elevating the good (relevant, interesting, original, etc.)
> and ignoring or downplaying the bad.
...
chris, if i may be so bold as to tell you that you are _so_
close to being right on the money, but _not_quite_. :+)
...
if you shift your thinking ever-so-slightly, you might find
that it makes a huge difference in the conclusions you get.
...
instead of using the concept of "good/bad" -- which implies
something inherent in the content itself, and thus leads you
invariably to words like "filter" -- think instead in terms of
"like/dislike". this re-locates the base of differentiation in
_the_receiver_. that in turn leads to a pointer/guide system
that focuses on answering the more poignant and useful question:
"how do i find _people_ who _like_ the kind of things _i_ like?"
...
when you ponder how to architect a system that does _that_,
you'll start to move a lot faster...
...
not that you're doing that bad on your own, mind you. :+)
but i thought i'd give you a little boost.
...
-bowerbird
Posted by: bowerbird intelligentleman | July 06, 2005 at 10:51 AM
The Pre-filters you talk about are trying to find out a lot about the kinds of people they think will be interested in their product. They then work on advertising that will speak directly to them. In the context of a TV show, they want to know when they are watching, how much money they have and how they respond to marketing. In a model like that, the intended market determines a lot about the content being offered. How would advertising change if more products were released with less pre-meditation? How does the long tail affect advertising?
Is it possible that advertising and demographics may continue similarl