Why is eBook Metadata So Hard?

Today many continue to pour the physical book content into the digital container and believe that they have addressed the digital opportunity. However many have also adopted the same approach to contextural data, metadata, bibliographic, burbs etc. It’s as if we think that what works in the physical world is equally applicable in the digital one.

Is this blind faith approach down to a conscious decision to do as little as necessary and incur as little cost as possible, or is it a fundamental misunderstanding of the digital environment? Some would suggest that it’s like continuing to distribute AI sheets like confetti via a fax machine in the email and internet age.

Way back in the eBook dark ages (the late ninties) we were part of the highly acclaimed publishing research series edited by Mark Bide and Mike Shatzkin, ‘Publishing in the 21^st Century’ and one of the reports we produced was simply titled ‘From N to X’ and was about the growing importance of Context (metadata) over Content. What was clear then was that discoverability was as important, if not more important than the content and finding that digital needle in the Internet haystack was going to be a challenge to all.

Interestingly, the industry choose to retain their physical taxonomy of classification and apply restrictions often that only applied to the physical world. It as if all we could see was physical shelves and we had to place the titles thus. ONIX which did much to provide structure for the physical B2B supply chain started to become increasingly irrelevant in the B2C ebook supply chain. The major ebook retailers all adopted their own classification taxonomies some permitting many genre classifications others a few. Keywords became important but the response was varied. The important blurb was often just the physical one again merely poured into the digital description. Some indexed the first chapter, others the whole book, but searching index words often then became highly subjective and often irrelevant to the intended search. As seen with some generic search engines today even your individual profile may present different search results to others.

Some battled with the semantic tagging of the likes of illustrations. Some sectors recognised the importance of citations, references and continued to support those established to manage these in the physical world.

The point is that we have a very rich bank of potential information that can aid discoverability and qualification. The richest source of content metadata being in the book itself. Some 95% of all contextual data is available via the book with the remaining being often available via hyperlinks.

The other source of information comes from our own profiles on our tastes, preferences and habits. Some will say that one of the reasons why social book sites work well is that they aid discoverability, recommendations and thereby add context. Its little surprise therefore that Amazon bought Goodreads and have just launched their Twitter relationship. Book recommending social services have mushroomed but  have they actually increased sales to the same proportion or are they just an extension of today’s ‘me too’ social dating society?

We therefore need to rethink what we give away to promote, aid discoverability and add value to drive sales. Merely to continue to pour the same physical information into the digital boxes is not going to work. Equally waiting for the white knight to provide the solution may well now carry a health warning.

Rethinking Marketing and Discoverability for Today

‘Digital publishing is publishing’ and the news this week from Bowker that online sales have reached 44% of total sales should not come as a shock to anyone. The question is what are we doing to exploit this, the rise in digital titles and sales and other cultural changes? We hear lots of talk about who is the retailer, publishers going direct and of course digital, but how are we planning up to service this? Finding and valuing anything on an every increasing virtual shelf is becoming harder, when in fact it should be easier?

Search and discovery in a digital era is very different to that we experienced in the physical only world and yet many appear to be trying to serve up the same basic information and to be crossing their fingers that one shoe will fit all – it won’t. Look at music and how services such as Shazam, Spotify, YouTube, digital online radio and social networks are redefining how we find, sample and enjoy music. 

The Onix XML standard was a milestone for common description of metadata and its construct and promotion was long overdue, but although it was built on some great work done in the music industry by the likes of Muze, it was built primarily on the back of the EDI market and XML standards which were aimed at a physical product market and servicing established B2B supply chain channels. As with many industry standards some would suggest that it often played to the lowest common denominator and rather than expanding the vision and usage of contextual material it confined it to the established B2B service model. It continues to service the online retailers but does it do enough or just enough? Does it really work in a world of self-service where the consumer wants to define their own discovery approach? Does it work when the content is digital and itself can become the metadata to describe itself?

We would suggest three observations towards a different way to view Context and in doing so service Content better.

   

Book in hand

Years ago the major wholesalers created their ‘book in hand’ processes which captured and validated the information about the book as the first copies were handled in the warehouse. The key to this was the fact that the book itself was the best authoritative source of the information. They still needed basic information on which to pre order the title prior to publication, but as many know, the title, jacket and other detail was often subject to change.

We have seen various ‘search inside’, or online samples of the book which are in the main driven by the distributor/wholesalers and online retailers. With the exception of certain sectors it was driven at the discretion of the publisher by the retailer and digital distributors.

Search Inside could be the new book in hand environment, which now can not only service the intermediary, but can also now service the consumer. All that is required is a digital file – the digital rendition itself, the rest is down to technology. So we all can have a book in hand and this can change how we create, develop, control, market, promote and offers a new and different context opportunity.

Contextual rights may be a term we have just created and some would say they already exist today as rights, but it makes sense to recognise them. The could create a leveller playing field and rules in which both intermediaries and consumers are able to fully exploit the digital content for digital content purposes within a standard framework.

When the likes of Amazon started to scan jackets some would have suggested it wasn’t strictly allowed but years later who would suggest that today? Using the full content to provide authority and context is plain logical. A better framework could enhance the discoverability of works and help remove those embarrassing search inside and often meaningless ‘surprise me’ and blank pages.

Far too often we have retained the physical book process and merely tipped the end product into a digital container. This approach was sensible yesterday but could be viewed as very naïve tomorrow.

Word of Mouth

Goodreads and others have demonstrated that many wish to read what others recommend.

The word of mouth recommendation has prevailed throughout history. However with the emergence of social networking it has exploded and we now can see where our ‘friends’ are, what they are reading right now and their views on it and recommendations. It’s like being given the keys to not only their lifestyle, movement but also their library. The challenge isn’t the people we actually know but those who are merely ‘friends by some association’.

Many have long treated the ‘independent review’ with some scepticism. After all, when the reviewer is actually also offering the book for sale, it makes one question how independent they are. Newspapers today sell the books they review and the question must be as to which the driver is and whether the reviews are the reviews independent or mere adverts. We once did a business survey on Home Depot and loved those wiring boards that looked like they were created by the local electrician, but discovered that they were in fact giant POS boards produced centrally. We also loved those little hand written book recommendation cards in some chains, but the sceptic questioned whether these were paid for space, or genuine. We would have loved to see if the same books got the same reviews, or were reviewed in other stores of the same chain. We are also all aware of the numerous claims over false reviews made by interested parties on online service.

Reviews and recommendations are often only as good as the last one and you still need to validate that it is what you actually want.

Looking at the same house through different windows

Today it can be difficult to determine who needs what information and we still build system silos to service only one aspect of the business. Search and discovery has gone beyond the basic bibliographic and metadata information that serviced the B2B market. The market now needs more information in order to sell both physical and digital into the channels and to the consumer.

The virtual bookshelf has given us the challenge of finding needles in digital haystacks.

Digital content can be viewed many ways by many different people. The reader may wish to see the first chapter of a novel. The academic the index and table of contents of a monograph. The teacher or student a selection of pages. Even the bookstore, school or library may wish to see what they are buying. It is just marketing material that can be easily drawn from the same digital content. It’s like looking at a house through different windows and seeing different aspects of the same house.

This leads us onto the catalogue which is just a collection of works which have been compiled together for presentation or commercial offer. There is no difference to the concept of sampling and marketing merely an extension. The reward for thinking this way is that you can see what everyone looks at, or ignores, or even if it have been viewed at all! It can itself drive sales, adoptions and marketing and promotional programmes.

This starts to again beg the question as to whether the same material can serve both B2B and B2C?

It also starts to suggest that the content itself can become the richest source of context to help the marketing and promotion of works and the search and discovery of them not just post publication but across the total lifecycle.

Finding Digital Needles In Digital Haystacks

Remember those dark days before ONIX standards, search inside the jacket, or even picture of the jacket?

Contextual data , or as it in know, Metadata, is produced to help consumers find the book, validate it and also to promote it. When the physical bookshelf was the only option, then we relied heavily on the touchie, feely approach and the promotional sell was to the bookshop. After all,'if it isn’t on the shelf you can’t sell it' and bookstores often have finite shelf space at all levels. Then came the internet and it wasn’t so much about getting it on the shelf, but making it visible amongst the hundreds of thousand others. The onus shifted from supplying the retailer with basic information to supplying the consumer with rich information.  Now we are entering a new phase where the book starts to promote itself and it only the consumer that counts.

The new industry metadata standard (ONIX), has helped define the basic metadata and also its adoption across the supply chain. It served the intermediary world well , but is it enough in a consumer only world where titles are effectively on consignment and don’t have to be sold to the retailer? 

Many will argue that standards and agreed data structures are a must and without them books will just be lost in the new virtual space. Others will counter saying that they all too often act as a straight jacket and are still supply chain focused and not consumer orientated. Somewhere in between lies the reality, the challenge and the opportunity.

One of the greatest constraints the standards give us is on genre classification. We appear to have created the standard ‘tree, branch, twig’ hierarchical approach to genre classification. This was great when you had one book and only had one slot in a bookstore to display it, but is it really as relevant in a virtual bookstore with infinite shelf space where the book can sit in literally thousands of relevant slots? Is it relevant when digital content itself can define in which genres it belongs? Is it relevant to the consumer who is constantly finding and redefining genre and can’t wait for the standards bodies to endorse the name? After all, by the time new vocabulary is defined in the authoritative dictionaries, it already has been widely adopted and used.

So we now have a truly ‘mixed’ economy comprising, physical books sold through physical stores, physical books sold through digital stores and digital books sold through digital stores. Should the same construct serve all three channels, or should the physical model act as a loose base, but we adopt more appropriate and consumer facing methods for the digital world?

Amazon Kindle only allows two ‘dry’ classifications and seven keywords, Kobo is more tolerant and PubIt is only just about to finally go international.

Are genres truly hierarchical or now lateral? Are keywords now more important than the headline genre? Do we find ‘books like’ through sales or more sophisticated means? Do we enable keywords to be excluded as well as included? How do we express multiple demographic appeal? How do we rate relevance? How do we rate or harness genuine reviews? How do we use the content itself to define its appeal? Do new books always come before older ones?

We don’t have the answers, but we do realise that today’s cumbersome expressions are frustratingly restrictive and far from being engaging and in some cases relevant. They provide a base in the digital world, but sadly not much else.

Today, finding a new unknown title in these silos, is like finding a digital needle in a giant digital haystack.

eMartketing Thoughts: 5. Store Once, Render Many

Context is the metadata, bibliographic information, marketing materials that helps everyone find content and of course value it. A significant amount of the Context can be derived direct from the content itself and what we once thought of as content can now be used to promote, sample and validate before you buy.

Today publishers should be storing everything once and render many times. The rendering should be as much as possible dynamic and avoid holding multiple versions and formats in many digital folders and even databases. We should be creating context directly from content avoiding further duplication and effort. Publishers should ask themselves how many times they hold the same context information, which is the authoritative source and who and how it can be changed? We all saw the rise of the ‘book in hand systems’ in wholesalers who couldn’t trust the data feeds. We can now surely do it better.

Delivering emarketing marketing becomes so much simpler when the widget, the digital review copy, the digital catalogue, the online rendition are basically the same file or files seen through different viewers. Mass marketing and direct marketing materials are the same but again viewed through a different viewer by different people. Bibliographic feeds are no longer dry bits and bytes, but now are whatever is appropriate and in many cases can be cut from the content itself. Finally, we now see the emergence of the digital emarketing container which not only reshapes what we distribute but who distributes it and when.

eMarketing today is not just about creating many experiments, social sites, direct campaigns, viral marketing but all these and above all serving the existing channel and enriching their opportunity to sell more physical and of course digital works.

eMarketing Thoughts: 1. Show Me the Results

We aim to write four more articles this week to expand our views on eMarketing; the use of widgets, digital inspection copies, catalogues and the creation and management of context and content.

Marketing holds the biggest digital prize for the book industry in these changing times. Some have long argued that Context is more valuable than Content. In the late 90’s we were responsible for the highly acclaimed publishing research programme ‘Publishing in the 21st Century’ and some would suggest that our most significant report that we published was ‘N to X : From Content to Context’.

Today’s task is not finding digital content but finding good or the right digital content. Its about finding the digital needle in the digital haystack. We have many sources of information, many degrees of detail, many different ways to look for and value content and content itself is no longer a single format.

We are moving away from the advertising budget being spent on full page spreads in the trade press to viral and direct marketing, from scattering seeds in the wind to running campaigns where every hit, click and resolution can be tracked and the marketing department held accountable for spend! We know can find out not only what we know but potentially what we don’t know.

In yesterday’s world we produced glossy printed catalogues and posted them to anyone standing. Yet we had no idea who read what page or even opened them. Did they themselves provide all the information needed to make a decision or merely opened the door? We distributed advance information sheets which fell somewhere between a flysheet and a genuine information sheet and often ended up being filed with the waste paper. We spent money advertising to the channel to get the book on the shelf and had little left to promote the book to the public. The trade magazines were happy to take the money for the full page colour advert but did it deliver, give demonstrable results, or merely spend the money?

As we move it the digital eMarketing world we are not obviating traditional spend but supplementing it with more auditable and accountable campaigns that can even be integrated into the whole process and help join the dots from Author to Reader.

Our 2009 Predictions

2009 beckons and as we celebrate the end of what for many was truly an unpredictable year we look forward to what we may expect in the next 12 months. We will all be heavily influenced by the economy and its impact on consumer confidence and spending no more so than new initiatives within digital publishing. The unthinkable question is not what will happen, but who will survive 2009? The physical market is still the dominate one for all and is unlikely to be a smooth ride for anyone, with authors being placed on hold, agents having to battle to get to the front of what will be a big queue, publisher reining in the bets and looking for the safest route and retailers watching the credit and cash flow.

At times of austerity some will spend their way out, hoping that by being brave they will be rewarded, others will batten down the hatches and hibernate, waiting and hoping for Spring. What is obvious, to all but the die hard, is that there likely to be fewer ‘passengers’ at the Book Fairs this year and conferences will become a luxury?

We still find ourselves trying to create an ebook market in a vacuum and although the device is less of an issue, consumers want a ‘safe buy’ that has legs, or a price that makes the purchase a ‘no brainer’. Today the price is not right, the format not clear, the model is lost trying to live off the printed one and the content itself is just a carbon copy of the printed book and not special or a must buy.

So against this bleak winter what do we predict for 2009?


1. Content
The volume of digital titles will continue to grow but in tens of thousands not hundreds of thousands and the dominant format will remain PDF and Adobe eBook. The industry ePub format will grow in popularity for those who choose the detailed XML route but for many it will remain a cost too much.

2. Context
In a tight market, marketing is going to have to be on top form. That little extra effort could make the difference between an also ran and a hit. We must remember digital context sells both digital and more importantly physical books.
We envisage widgets will become smarter and offer a lot more that just the plain look inside the book. Digital renditions will be increasingly given away to promote physical sales. Authors should come to the front with podcasts, videos, blogs and web sites to promote them and their titles and the interesting question is whether they will do it themselves, be aided by their agents, leave it to the publisher, or look to others?

Finally the academic market will see the emergence of the digital inspection copy being used to sell adoptions and maybe create an academic social network environment. The trade on the other hand will miss the opportunity to digitise their review copies, or at best do it as an afterthought and believe that physical review copies are still a cheap run on option.

3. Rights
The introduction of the ‘Google’ Rights registry in the US is certainly going to focus the thinking. We believe that it will raise many of the challenges that have long been bubbling under the surface and many question are likely to be raised on the specification and scope of the work. Why the bibliographic agencies or the authors bodies stand up and do this is another question.
The whole spectrum of rights will continue to be questioned as POD is used to grab more orphans and retain rights in perpetuity. Permission rights will start to become more visible and an issue as ‘chunking’ will become more common together with the sale of digital fragments.


4. Digital Platforms
We stand by our vision of the mobile and the notebook still being the real platforms. There will be more ereader devices based on eink and expect a price battle but we still can’t see them becoming a must have device. We also see a shift towards online and rentals and away from download purchases. We believe that if Flash 10 makes it successfully put onto the mobile chip then mobiles devices will move forward and start to dominate the market.

5. Market Sectors
The educational, professional and academic sectors will continue to lead the way in digital both in publisher commitment and sales. One to watch in 2009 – religion. Trade will be more hype than substance but with a clear emphasis on digital marketing. The other sector we expect to raise its digital head is the public library market where the interesting question of its conflicting business model with trade (free to rent versus pay to buy) will remain unsolved.

6. Omnivores and New Entrants
The three main players will continue to steal the show, Amazon with its comprehensive covering of all bases, Google with the help it got buying the trade for ‘chump change’ and Adobe quietly plugging away on many fronts in the background.

7. Editorial / Production Content Development
We envisage that more and more publishers will review their antiquated development process and realise the opportunity benefits to be taken from digitising this. The key to success will be down to the implementation approach and managing the change programme. The old adage, ‘if a customer service clerk doesn’t like the system, you sack the clerk, but if and editor doesn’t like the system you sack the system,’ will still apply. Another important opportunity will be to pull the publishing business (content, context and rights) under one single digital process.

Review of our 2008 Predictions - 2. Context

At the end of 2007 we made 10 predictions for 2008. Many predictions made have now been conveniently forgotten but we stand by our thinking at the time and believe it appropriate we openly review these individually and see what we think happened and assess them appropriately.

2. Context – widgets , widgets everywhere. The effect will be similar to when book jackets were first developed in mid 90s and the impact will be as great. Rich metadata will be the order of the day. Bowker and Nielson will have to respond but may find the world has moved on again.

Well widgets did start to appear in the many forms. Some looked smart and clean others not so tidy. The major aggregators all declared and started deliver and of course Amazon and Google just moved their agendas forward with purpose. Did the Bibliographic agents respond – we think not?

We think the widget as we knew it as a cheap and cheerful ‘look inside the book’ has happened but has failed to fully live up to the marketing potential. Rich metadata however has exploded with major houses investing in many digital experiments to promote books. The experiments have often been so disjointed it leads us to question whether any dots have been joined?

The big question is how we now harness what is clearly an opportunity to sell not just digital but physical books and make book buying more fun.

We think a disappointing 5 out of 10, do you agree?

Who owns what?

The world of content would be lost without the world of context, which helps qualify it and enable the consumers to select it. In the early days the bibliographic agencies supplied lists of basic information which said little about the book, we then went through the jacket image production in the mid to late 90s and today we have podcasts, widgets, videos and digital marketing explosion.

So in a week when there has been much debate about Google’s sample pages in their book search programme, Amazon’s search inside offer, Thomas Nelson’s free book for a review and now Simon & Schuster setting up an in-house digital production studio. Where are we and what are are the issues we need to consider?

We must ensure that the current perceived demand does not create a two tiered environment with those who are big enough doing it themselves, their way and with others having to use 3rd party services and potentially give away their assets. It is important that the industry recognise that we need to provide consistency of information and presentation and yet not stifle innovation. The bibliographic agencies appear to have drawn a line in the sand on the richness of their records, the distributors and wholesalers continue to add extra value, some are making a land grab and some are merely experimenting. However, who is asking what does the consumer wants?

Next is the question of ownership. ‘He who creates owns’ is the rule with digital content. However, with context information, who owns what? What moral rights if any exist? Who decides the line between content and context and also free and for sale?
Finally, where do find what is available and what can or can’t be reused? Is a marketing promotion video free to copy and repurpose in a YouTube manner or is it like all content subject to permissions?

When we look at some of the scanned pages available, it’s often easy to see which were produced for free and which have been paid for, developed and are viewed as assets.

Flipping the Pages Onto a Screen

As digital content starts to evolve, we still have a couple of issues to address – price and value add. We have long argued that for consumers to switch onto ebooks there must be value, merely replicating the physical copy and presenting it digitally offers little. The experience may be different and offer some value, but the content is the same. This is a shame as the format offers so much extra that is currently largely being ignored.

We can add all sorts of marketing materials and extras, such as author interviews, original submissions, character biographies, historical notes etc. Penguin have stated to experiment with such in their renditions of the classics to digital. However, whereas the classics are relatively easy to collate public domain materials on, current works may prove a bit more taxing. First of all, who owns the intellectual property and is it tied to the specific rendition or the work? How do you clear author’s moral rights and if done does that gives them any rights of association going forward? Who gets paid what re royalties, or is the additional materials fee based? Can the additional materials be used in its own right and if associated with several works, can it be the leader and not the follower?
Some may say that we are talking about things that are already fully covered, but are they and do authors fully understand this aspect of the total work? Its interesting, following the highly published JK Rowling case, to perhaps stand back and think about this contextual work in a digital world.

The question of price will roll on forever, that's until, as in music, someone creates that price point and case that flips the market. Pricing ebooks equivalent to physical copies, is as ludicrous as applying VAT to digital copies of the same work. Its interesting that the reported consumer reaction to price in the press appears to indicate that the consumer is unaware the price is a VAT inclusive price.

Is anybody asking the consumer what they want? Do we have a collective view of the needs of the market, the potential barriers, or are we all assuming we know best.

Digital Publishing is Publishing

We hear of the latest salvo in the battle of the omnivores versus publishers in today’s Telegraph. The report states that publishers are converting thousands of titles in order that they mitigate the threat of the likes of Google stealing them or capturing the ebook market when it finally takes off.

If we step back and remove the emotion of digitisation, we can clearly see it's not about ebooks. Neither is it about which ebook technology will prevail. It is also not about the consolidation of digital content onto the mobile phone and MP3. It's about the wake up call to publishers.

Digital publishing is publishing – its not just delivering digital stuff but developing it, producing it, marketing it and selling it.

Digital Content is lost without digital context and digital rights – so it's about using digital content to market, sell and enrich marketing and publicity of both physical and digital product.

Digital distribution needs to live alongside physical distribution – the vast majority of books will be stay in the printed form and the existing channels will dominate for many years.

Publishers need to forget the hype, stop looking at the green eyed monster onmnivores and the large publishers who claim to be digital today. The driver is not fear, nor the elusive digital sales and consumer demand, but getting the product into digital form, the context digitally rich and being in a position, unlike the music industry, to react at take off.