We recently had an episode with google news that cost us a lot of good exposure and a fair amount of traffic. Apparently, the google news crawler is much more restrictive than the normal googlebot. Some of the people from our newsroom had noticed that we were no longer being featured on google news. As this was happening during the Société Générale scandal we took this issue very seriously. Using our google webmaster account to look at some diagnostics we discovered thousands of errors with “Article too long”. It took us a few hours to track down the source of our problems. We had, as a part of our article template, a comic-strip-like html construct that was serving as our “video box”. The box acts like a view port and lets the user click through the video offering for that section in a very easy way. As it turns out our list of videos was well over 100 for some sections and this seems to have upset the google news crawler. Google was interpreting this long list of video assets as a way of trying to cheat the system and subsequently blacklisted us from Google news.

Our response: pull down the videobox immediately across the whole site and reprogram the thing. Dedicated readers may have noticed that the box was missing for a few days but now it is back up with a list of 10 videos. Within hours our content reappeared in google news and everybody happy. There is an important lesson in this for all product development people out there and that is this: just because you can does not mean that you should. Filtered content and manageable lists are better than throwing everything at the user.

For those of you who access this site through google news please accept our apology.

It’s been a while since we’ve added anything to the Developer Blog. Holidays, vacations, and a mad rush to launch the new Business with Reuters section left little time to come up with cool new widgets for the site. But the new year is finally here, and as we slowly return to full staff the ideas will start to flow once again.

When we relaunched the new Business section Monday, we needed to accommodate some changes to the homepage. The bulk of what’s changed is in the C-column: 1) the addition of a Business with Reuters promotional box, 2) the relocated Reuters news feed box, and 3) the addition of a new Market Tools widget below Skybox 3 (designed and served by Wall Street on Demand). With the creation of this new Reuters “zone”, it was clear that something needed to be done about the “In Blogs” box.

Over the past few months we’ve run multiple CrazyEgg click tests and reports showed that the Blogs box was performing dismally. As a result, blog traffic suffered, but not for lack of visibility. We came to a conclusion which was similar to our thoughts on the homepage Reader Discussion box:

Visitors are not coming to traditional news sites to peruse a bucket of blogs, so context is more important than visibility.

To apply this to the rest of the site, “Blogs” shouldn’t be the product so much as each blog should be a feature of its respective parent section of the news. This should be reflected in how they are promoted at the article and section front level.

Referring once again to our click test results, we noticed that the news headlines at the bottom of the narrow column are some of the most clicked elements on the entire page, often surpassing some of the ranked stories. It became instantly clear that this “hot spot” on the page could be leveraged to generate blog traffic as well.

What at first glance seems like a gimmick to draw clicks actually serves a valid purpose to the visitor: it puts individual blog entries in the context of “The News”. Our blogs are written by some of the same journalists who write news stories, and many blog entries are about what’s in the news, so why differentiate the two products?

“Blogs” are just another method to package and publish content; the stigma of blogs having a less serious tone is in the past, at least in the corporate setting.

The way the new “In Blogs” section works is simple: every hour, a cron process queries the Wordpress database and finds the two which feature the most recent new entries. Notice, I didn’t say updated entries; that would have made it too easy to game the system, and a blogger-brawl would soon ensue over who out-updates who. The new setup is not without ethical implications, as I will soon explain, but the initial traffic reports show some very promising results.

In the first few days, all blogs have received a traffic increase ranging from 52% to an astounding 180% above their respective daily averages. Obviously this is something we will watch closely over time, but fresh new click tests demonstrate that visitors are indeed clicking the blog headlines, just as we predicted. And because people are going directly to an entry page, their next step is to start exploring what else the blog has to offer, resulting in even more page views.

The previous “In Blogs” box needed to be updated manually by a person, and only one blog would get to be “promoted” while the rest were simply listed. The new process eliminates the need for manual labor and also introduces a fairness factor, for better or for worse: active blogs are rewarded with top exposure and traffic to be sure, but less active blogs will still benefit when their entries are published.

Because the system is automatic, the door is open to “gaming”; a clever blogger could use a few strategies for gaining maximum exposure. For example, with “The Price is Right” method, blogger A posts an entry at 8:45, blogger B posts at 8:46, blogger C bumps blogger A off the list by posting at 8:47. There are several more underhanded strategies that I don’t need to list, but we’re hoping the goodness of mankind overcomes and everyone just gets along. The web, of course, is a 24-hour operation so there’s no need for every blog to post during the same hour.

That’s about it for now. Blog away.

A few weeks ago, we talked about how visitors are using discussion threads on IHT.com. For a while now, we’ve been thinking of more ways to capture, retain, and entertain this small but important subset of visitors.

One thing we’ve noticed is that once an article falls off the homepage, the number of comments drops drastically. This is unfortunate, as the only hope at that point is for the story to either a) make it to the Most E-Mailed list or b) get linked to from a major traffic-generating site like Digg or Drudge.

Often we have lively and intelligent threads featuring contributors from all over the world, so how can we keep the discussions going?

A few days ago we launched Reader Discussions, a single page where visitors can go to find out what people are talking about. It’s located in the lowest tier of the top navigation (or simply at talk.iht.com), and for the most part it functions like a lite version of an online bulletin bar or forum. Displaying all threads published in the past 60 days, it’s a surprisingly addictive page that you might want to keep open in its own tab and refresh regularly.

The default view, as you can see, sorts the discussion threads by when they were first created, followed by options to sort by Recent Activity and Most Comments.

A CrazyEgg click-tracking test over the weekend yielded a couple of interesting results.

First, “Blogs” was the most clicked-on element in the top navigation. This is contrary to every other click test we’ve ever done and it tells us that this page’s visitors are familiar with user-generated content and are seeking more. Perhaps we will consider adding a way to peruse all blog comments in the future.

Second, as indicated by the red dot in the screen below, the “Recent Activity” sort method was the most clicked element on the entire page, telling us visitors do in fact choose to use the page in the way they are probably most familiar with, which is like a common web forum where the most recently-update thread bubbles up to the top of the list. I’m not sure we’re ready to make this the default view, but it’s something we’ll keep an eye on going forward.

On the technology side, the implementation is as straightforward as can be. Since our comment engine is a tweaked version of WordPress (which is like Play-Doh for a decent MySQL/PHP developer), all we need to do is query the DB and spit out the results. Our developer Sebastian Tran did a fantastic job of whipping this all up very quickly.

This page also provides a benefit to those on our editorial team who create new discussion threads: since we can see at a glance which threads are getting the most activity, we can get a better feel for the type of stories our visitors want to talk about (turns out they don’t want to give away their secrets for getting airplane seats with more legroom).

Finally, the dev team can use the page’s back-end functionality to cull the data, perhaps even widgetizing the entire page for inclusion on article-level pages and elsewhere.

See, everybody wins! Now come share your thoughts on worldly matters.

As my colleague Michael pointed out in a previous post, we have found that most people enter IHT.com through an article page. A large percentage of this direct traffic is a result of people subscribing to RSS feeds. Our mantra here has been to RSS everything and expose those RSS links everywhere. For exampe, you will find on our homepage (if you still go there) a large offering of RSS links for most sections towards the bottom of the page. Our AP breaking news feeds also have dedicated RSS feeds, as well as our list of most emailed articles. A few months ago, we set up a system that creates an RSS feed for every active discussion that we have associated with an article. You can find a list of all of them here: iht.com/rss

In addition to publishing the files in RSS format we also publish the same files in Atom format. This is not the space to debate the merits of one format over the other. We simply provide both and let the audience decide. For the record, our traffic patterns indicate the RSS format is more popular.

The paradox here is that some of our most loyal readers, the ones who used to come to the homepage and click around, now have an easier way of reading IHT.com by using their preferred RSS reader. They browse us just like any other reader would. We value the traffic and also want to keep our readers happy but there is a difficult balance to strike. We have chosen to publish our RSS feeds without the full text and a link back to our site for the complete story in order to strike this balance. For now.

Remember the good old days when your home page was the place to go for your visitors to peruse your up-to-the-minute breaking news and special features? It was to be the portal page which was the realization of months of wireframing, mocking up, interdepartment arguments, and sanity-questioning.

Well it still is, sort of, in that it’s still the fancy display window to your department store. But what happens when customers find a way in through the back door, take only the item they want, and then leave without bothering to browse the rest of the store?

Enter search engines, the ultimate timesavers for people who want news about a specific subject and want it now. Day after day, your developers and copy editors work hard to improve SEO. The key to increasing page views and ad impressions is “all about SEO” these days, and that’s become the mantra, much to the chagrin of clever headline writers around the world. But what happens when you implement SEO techniques too well? Suddenly your site is turned inside out, and it’s all your fault. But it’s one of those good problems to have.

Today, every page is your home page.

Like most well-optimized large news sites, the number of visits to the home page pale in comparison to article pages. If you’re lucky enough to get on Google’s good side, it’s like being part of its entourage, with all the benefits that come along with it.

The problem is, search engine users are an impatient bunch, not looking for a committed relationship. Most don’t even care to know your name. They will find the article they are looking for (maybe about killer monkeys), read it, and only 1 in 5 will click something else on the page. If you’re not giving readers a peek at what else you have to offer, you risk losing them forever, or at least until the almighty hand of Google guides them your way once again.

We’re rolling out a bunch of new features in the coming months, all geared towards guiding users to the home page, article pages, or other features. Yesterday we added what we call the Home Page Snapshot to the bottom of every article page. It’s not an entirely new concept for news sites, and it’s been implemented many different ways, but we really wanted it to provide a window into what’s happening on “page one.”

The box, like most other bits of content we’ve been designing lately, is decidedly large. After all, it’s on the bottom of the page where you have all the real estate in the world. It gets inserted below the article body box using a standard include. To publish the box, one merely needs to republish the home page. The person producing the home page doesn’t need to worry about it. No one needs to cut yet another photo size because we use ImageMagick to do the work on the server side.

Note that the box is exactly the same on every article page. It is not contextual to the article’s parent section, and that’s by design. It’s a snapshot of the latest news featured “above the fold” on the home page…nothing more, nothing less.

We’ve done a few early heatmap tests and the results are very promising. We already knew that people do, in fact, scroll on the interwebs nowadays, so why not give them something to do after they’ve finished reading? Even if the box earns just a handful of clicks for each article, that adds up to thousands of extra page views directly, and likely many more if your visitor retains interest. All with a minimum amount of overhead. And who doesn’t love (almost) free things?

These days, as you know, it’s not enough to simply supply content to your visitors. We’re all working on ways to keep them — and their precious page impressions — returning regularly. Of course, one of the easiest ways to do this is by taking a page out of the Blogging 101 manual and implementing user comments, or as we call them, discussions. This entry isn’t about the merits and downsides to user-generated content, which has been discussed ad nauseum on any number of sites. It’s more about what we’ve learned about how visitors are (or aren’t using them).

We launched our Discussions feature last January and we found that there were three distinct aspects that need to be considered in order to create successful comment threads:

1. Location, Location, Location

When we were wireframing the new home page earlier this year we envisioned “Reader Discussions” becoming one of the destinations on the site, but it turns out that our visitors aren’t really looking for a “bucket of discussions” (although we are launching a Discussions landing page shortly). We’ve done some heat-mapping tests and we’ve learned that context plays a much larger role.

The click numbers on the discussions box on the lower-right are rather disappointing. Everything above, left, and below it performs well. However, when you add a “post a comment” link to a ranked headline/summary, suddenly the most clicked items of the entire page include some added-value. Perhaps the summary alone peaked their interest and now they are ready to join in. The results are a much greater click rate and comment count. But even a prime location and context won’t guarantee success.

2. Pick the right topic

This is the subject of much debate between cynical user-experience designers and those with earnest editorial visions. On one extreme there is the “lynch mob” method: catering to lowest common denominator audience; the ones who will gang up on the fallen celebrity or politician of the day. While this is sure to get people talking, it’s certainly not in line with the editorial vision of the IHT. A few hundred extra page views is not worth it when it comes with a tarnished image.

On the other extreme is the “intellerati” method: picking topics so specific and high-minded that there is only a minute number of visitors who would have anything to say on the subject. While this is undoubtedly the editorial high road, the end result is few to no comments and a lonely discussion thread. Further, if readers continually encounter empty threads, then they will think it’s a lame feature and not worth their time.

All hope is not lost because there is a sweet spot. The very first thing to look at is where the majority of your audience is coming from. If, for example, most of your visitors access the site from North America, then it makes sense to create discussions that are of interest to that geographic area. This is simply acknowledging your market. Stories about small, out-of-the-way countries are very difficult to obtain traction with.

Next, the topics should be somewhat divisive, where the seeds can be planted for a spirited debate. What we want to avoid is creating “me too” discussions. Beyond that you should decide between hard news and soft news. Hard news, of course, is more difficult, though war and religion are always popular topics. Soft news offers more possibilities, with stories about health, travel, and other “general interest” subjects.

3. Lead the discussion

Finally, when you do convince a visitor to check out a discussion thread, you need to give them something to do. This means presenting an engaging lead-in question. So first, set the stage with a one-sentence summation of the “divisive issue” contained in the story they just read. Then, open it up to discussion, keeping the readers focused on the topic at hand.

Admittedly, we occasionally open a discussion by saying simply something drab like “Share your thoughts on ethanol”. Well, ethanol is a pretty broad topic, and no one knows where to begin in this case. This is the wrong approach, and the result is predictable: the only sound you hear is crickets.

Conclusion

Creating successful discussion threads is not an exact science, but there are many strategies that are effective in improving the product and increasing reader interest. In a way, it isn’t much different from being a television network executive who needs to pick and choose which programs and time slots that will appeal to the largest audience and earn the highest ratings. In the end, it comes down to a combination of understanding your site metrics, experimentation with layout and topic choice, and instinct.

Blogs.iht.com has always been somewhat of an orphaned page for us. Our thinking has always been that the home page would serve as the main entry point to the individual blogs, but now that we’ve included a “BLOGS” item in the third tier of our navigation (as well as a link in the Blogs box and the global footer), we thought it was time to make it a bit more useful than simply listing the names and descriptions of all the offerings.

We’re not 100% there with the design yet, but we went ahead and launched a facelift yesterday:

Understanding that the user is more likely to click through when they can get a peek at what’s inside the blog, we implemented just that; using a bit of AJAX we’re now displaying the headlines of the three most recent for each blog.

There’s always the potential for a debate to arise over which blog gets “top billing”. Traditionally, this is an editorial decision of course, but we’re wondering if it would be more interesting for the visitor if the blog with the most recent entry bubbled up to the top of the list, followed by the second most recent, ad so on. Can a little bit of internal competition be such a bad thing?

Much can be debated about whether Twitter is a useful tool for communication, a simple way for friends to keep track of each other’s whereabouts, or just another way to kill time online. How and why you would use the service is up to you, but we took a look at its core functionality and decided that it can be used as a free mobile alert platform and RSS aggregator.

If you’re a Twitter user (signing up is fast and easy), you can “follow” the IHT. This means you can be notified about all our latest headlines in a few different ways:

• Visit IHT’s Twitter page to see a feed of our headlines. Make sure you decide to “follow” us.
• Subscribe to our Twitter RSS feed and follow us in your favorite reader
• Choose to have an SMS message sent to your mobile phone whenever we send a “tweet” (a new Twitter entry). Be warned, though, that we post every 15 minutes, so those SMS messages can add up, or just annoy you.

As an alternative to these, If you’re using a Mac, we recommend the very slick Twitterific program, by The Icon Factory. It’s a small application which runs on your desktop, displaying a constantly updating feed of all the Twitter friends who you are following.

On the technical side it was pretty trivial. We wrote a small PHP script that, every 15 minutes, grabs the latest headline from our main RSS. Next, since “tweets” are limited to just 140 characters, we obtain a small version of our article’s very long URL using TinyURL and append it onto the end of our message. Finally, using Twitter’s API we submit it via the script and we’re good to go.

We’ll be the first to admit that Twitter probably isn’t for everyone, and is definitely still a niche product in the corporate realm, but it’s a fun little service that’s worth checking out. We’re noticing more and more companies who are experimenting here, so you never know what you might find.

Growing up, I happened to be a major map nerd, pouring over the Rand McNally road atlas of North America, inventing games such as “see who can get from town A to town B using the fewest amount of roads”. Even today I sometimes find myself panning around Google Maps or whipping the Google Earth globe around when I should be doing other things.

The IHT mostly publishes international stories, and while most of us know where to at least find a country or its capital, there are often times when I find myself copying the name of an unfamiliar city from the dateline and pasting it into Google Maps, just to see where it is in the world. Wouldn’t it be cool if every article was able to display a map without the need for an actual person to spend time processing one?

We decided to harness the almighty power of Google and we came up with a simple, yet effective automated map overlay:

From the users’ perspective, using the map overlays couldn’t be any simpler. You click on the city or country name and a small Google Map appears in a layer above the article. If the city can’t be found by Google, then it looks for the country and zooms the map out an extra level to give you better context. To hide the map, you can click the city name again, or the close button, or even outside the map layer. We think it’s a fun — if not somewhat transparent — feature that gives the reader just a bit of geography knowledge with each click.

Admittedly, the feature isn’t perfect. Since it is completely automated, we’re only able to extract the city used in the dateline, which is often the location where the story is filed and not where the news is actually happening. Automating it to a point where it parses the article in its entirety would be very difficult for us and annoying for the reader, much like contextual ads which disguise themselves as hyperlinks. Perhaps in a future release we’ll allow producers or editors to “tag” select stories with specific cities.

On the technical side we’re using the Google Maps API for the meat of our mini-mashup, and to handle the UI we’ve decided to go with the jQuery JavaScript library. It’s the first time we’re using jQuery on a production-basis, and so far it scores high marks among our developers for providing easy DHTML/AJAX implementations.

The one thing you should know about the GMaps API is that you will need to register a key for each subdomain in your network. This means if you have dev.yoursite.com, staging.yoursite.com, and www.yoursite.com, you will need three keys and then attach the correct one to your story pages. It’s a little annoying but not a dealbreaker by any means. Further, Google has a (relatively high) set limit of GMap calls per day before it returns an error message (you can arrange a higher limit by contacting them). We would be quite surprised if 50,000 readers in a day found this feature and clicked it, so we’re not worried…yet.

One random thing we’ve discovered during the development of this feature is that Safari on the Mac seems to parse external .JS files differently from every other browser out there; evidently it checks the code syntax of all functions, even the ones which aren’t called at the beginning. It was a nasty little bug that took us some time to pinpoint, since everything seemed fine and dandy in Firefox and IE.

If you’ve discovered and tried this feature, feel free to tell us what you think or suggest a feature enhancement in the comments. There’s more to come in a few days.

Related: Our big brother NYT has been playing with GMaps too.

Welcome to the ever-so-creatively named IHT.com Developer Blog. If you somehow found this blog, then congratulations, because we haven’t linked to it from anywhere on IHT.com or the rest of the web we’ve tucked it discreetly in the global footer.

Our site’s had a storied history, known as a bit of a rogue in the world of online news. Most famously, we pioneered the three-column article page layout a few years ago (a feature we’ve relegated to the article tools box for many reasons…I’ll explain in a future entry), as well as the “Clippings” feature (which has evolved into the more useful “Share Article” tool). Now, as always, our readers visit daily for a cleanly-presented, international focus on the day’s news. Maintaining the design integrity, tending to the many editorial demands, and introducing innovative new features is a pretty big task for a relatively small development team.

Despite all that, we do actually have a lot of fun. Being small and efficient allows us to free ourselves from a lot of the “process” and practice more “guerrilla development”. We’re cracking open APIs, JS frameworks, and enjoying the freedom of a LAMP environment. So we’ve decided to open up our bag of tricks to the development community, or anyone else who is interested. We’re the open-source, free-love types and we’re open to comments, suggestions, and feature requests.

So now we’re joining the ranks of the the increasing number of developer blogs out there. In a very non-establishment move we’ve decided to give away the entries, in their entirety, in the RSS feeds, so subscribe away. Also, let’s be friends: feel free to add us to your blogroll or request to be added to ours.

In our first real entry we’ll talk about a couple of fun new mini-features.