I’ve been remiss in not celebrating an important Stack Overflow milestone. Sometime on Wednesday, February 25th, a user asked..

Our 100,000th Question!

I didn’t say it was a good question, mind you. But it was the hundred thousandth question posted to Stack Overflow.

So, for reference — and remember our private beta started on 7/31/2008, and the public beta on 9/15/2008 — here’s how many questions and answers have been posted to date, by month:

We are pretty regularly doing 500k pageviews per day now. Geoff has also been tracking our bandwidth usage through the most excellent Cacti tool. Here’s what the last 24 hour period looks like:

So far:

• peak of 6 Mbit/sec or 750 KB/sec
• average of 4 Mbit/sec or 512 KB/sec

That’s actually a lot for a site like ours which is nearly 100% text with precious few images.

And compressed text at that! We’ve had the GZIP religion for a long time, but last night we went through and double checked everything to be sure. I did some port 80 sniffing and found a few cases where content wasn’t served up compressed, even when the client asked for it. Looking for commonalities and doing research, I discovered that HTTP 1.0 traffic and proxy traffic wasn’t being compressed. Turns out those are both off by default in IIS 7. I turned them on like so:

%windir%\system32\inetsrv\appcmd set config
    /section:httpCompression /noCompressionForHttp10:false
%windir%\system32\inetsrv\appcmd set config
    /section:httpCompression /noCompressionForProxies:false

I was surprised how many proxies actually identify themselves as HTTP 1.0, given how ancient HTTP 1.1 is. It’s especially ironic considering many of these very same proxies asked for gzip content — I’m not sure GZIP compression is even valid under HTTP 1.0.

But I digress. Congratulations to everyone who has helped make Stack Overflow a success by asking and answering questions. Here’s to many more years of learning together!

* partial data up to Feb 24th

This is the 43rd episode of the StackOverflow podcast, where Joel and
Jeff discuss dealing with incompetent programmers, whether salaries should be public, dealing with technical debt, and programming for small businesses.

• Joel is away this week in Florida at the Future of Web Applications conference, where he was a speaker. He mentioned that the new Atlas web GUI builder was particularly impressive.
• We will try to be more careful about our use of Begs The Question.
• Joel asks about the rationale behind requiring 50 reputation to leave a comment, but allowing a brand new user to post a question or leave an answer. The reason is mostly because we have no mechanism for voting or marking offensive on comments, because they’re ultra-lightweight.
• One way to avoid the dilemma of dealing with bad programmers is to be selective who you work for — only choose employment at companies where they Hit the High Notes. It’s even in the Joel Test itself: do new hires have to write code?
• I’m not a fan of puzzle-based interview processes. I met with a Stack Overflow user this week, Chris Jester-Young, and he revealed a clever and potentially more useful strategy: give interviewees a C program full of bugs, and have them try to debug them! Of course, Chris is a big Code Golf enthusiast, so of course he nailed that one.
• Sometimes you have to try to change your organization to fix the root problems, otherwise you’re just fighting the symptoms and not the disease itself. This can be quite a challenge when you have no real authority. Joel offers some advice in Getting Things Done When You’re Only a Grunt.
• Are there happy incompetents? I argue that there are; Joel argues that there aren’t. Among the Inept, Ignorance is Bliss. Perhaps the better question to ask is, how can you help this marginal programmer find a career they’ll enjoy? Many so-so programmers can make outstanding testers, for example.
• We wonder: what would it be like to work at a commercial for-profit company where everyone’s salaries were public knowledge? I imagined it as something like The Lord of the Flies. Just make sure you aren’t Piggy!
• At Fog Creek, salaries aren’t technically public, but they have a formula through which everyone’s pay can be derived. This is the Fog Creek Professional Ladder. Fog Creek also does profit sharing, so when the company does well, everyone does well.
• Is there a two-class society at Microsoft, between Testers and Developers? Joel wonders why they need different titles. I had always heard Microsoft did a great job of giving test engineers a viable parallel career track.
• Joel believes manipulating compensation to motivate people does not work, almost by definition. Top developers will do a good job no matter what compensation system you put in front of them. It’s a “blunt instrument” that can cause as much harm as good. Joel is an advocate of “taking salary discussion off the table, paying people fairly and justly and well,  so they can stop worrying about it so much.”
• We have a lot of anti-bot code on Stack Overflow. What we didn’t think about was human-entered spam! Now we do — yet another example of the incredible power of rate limiting techniques.
• On matters of customer service, we do endeavor to not make the situation worse. Start practicing saying “I’m sorry, it’s our fault.”
• Updates may slow down on Stack Overflow for a little while. We have built up a lot of technical debt around our database, and we have to hunker down and refactor a few core database tables that affect 80% of our code. If you don’t pay down your techical debt every so often, you could end up like Twitter — a reliability laughingstock. But, somehow, still successful.
• There’s something liberating and energizing about going in and tearing down huge parts of your application to rebuild it and make it better. Unlike our previous discussion on learning languages, I’m more of an advocate of the big bang model here, whereas Joel proposes something more incremental.
• If you work in programming or IT for a small business, can you give an elevator pitch explaining the value you bring to the business to the powers that be? Are you scaling your solutions to the size of your business, and not over-engineering it? What can you outsource, versus what is strategic?

We answered the following listener questions on this podcast:

1. Matt Rogish: “How does one ethically get rid of of incompetent programmers?”

Our favorite Stack Overflow qustions this week are:

• Choosing a technology stack for a small ‘budget conscious’ business — if the company is unwilling to spend any money at all, that means they place no value on what you’re doing. Fix that first, if you can.

If you’d like to submit a question to be answered in our next episode, record an audio file (90 seconds or less) and mail it to podcast@stackoverflow.com. You can record a question using nothing but a telephone and a web browser. We also have adedicated phone number you can call to leave audio questions at 646-826-3879.

The transcript wiki for this episode is available for public editing.

Podcast 43 [ 1:10:10 ] Play Now | Play in Popup | Download

Stack Overflow is somewhat unique in that we encourage participation of essentially anonymous, random programmers. Our idea is to radically reduce the bar for participation, and take one giant leap of faith:

Trusting our users.

Well, most of them, anyway. We have the typical hurdles in place to prevent bots from gaming our system: JavaScript and “too fast to be human” triggered CAPTCHA. This works in conjunction with community use of the offensive flag, which auto-deletes any post after 5 users tag it “offensive”, and traditional downvoting. This system has been fairly effective to date. We’ve only had a handful of spam and trolling, and it’s all been handled with a minimum of fuss or impact to the typical Stack Overflow user.

That is, until tonight, when we were hit by a malicious user of a type we haven’t seen before:

In a way, I suppose I should thank this user for doing this on a Friday night when traffic levels are pretty low. Here’s my official response:

How does it feel when we vandalize you, Mr. Tupac Shakur? Eh? Not so good, I bet!

But in all seriousness, the surprising thing here is that this user was not a bot. Our anti-bot stuff would be challenging to get around. It was an actual human being, entering the CAPTCHAs, cutting and pasting text into every post. We verified this by looking at the logs, and the timestamps on the entries. The times are slow and variable, not at all what you’d expect to see from a bot.

Wow. How bored is this guy? (And yeah, it’s always a guy, who are we kidding.) I’m not going to name any names, here, but we tracked all the IPs that this activity came from and they were all geographically similar.

As if I needed another reason to hate Kangaroos and Koalas.

I’ve been thinking for a while that we should have more stringent throttles on new users, rate limits for asking and answering questions. This human spam storm was my excuse to implement them. So, effective immediately…

If you’re a new user, with reputation below 100:

You may only post 1 question every 20 minutes

You may only post 1 answer every 3 minutes

This is tracked at the IP address level, so multiple posts from the same IP, even as different “users”, will still be blocked.

So take that, Australian wanna-be Tupac Shakur!

As part of our new backup strategy, I decided to invest in an inexpensive Network Attached Storage (NAS) unit — the QNAP TS-409U.

The TS-409U has the following specs:

• 1u rack mountable
• 4 SATA drive bays
• 500 MHz embedded CPU, 8 MB flash RAM, 512 MB RAM, Linux based
• Supports RAID 0, 1, 5, 6 with hot-plugging drives
• Gigabit ethernet network interface w/ web UI
• Mac, Windows, Linux filesystem support

Being Linux, you can also do lots of other cool stuff on it, like run Apache+PHP, MySQL server, FTP server, BitTorrent server, Multimedia Server (whatever the heck Twonky is), and iTunes compatible media sharing. But for our purposes, it’s an inexpensive RAID-based filesystem. It’s a minimalistic set of hardware, as you can see in this Techware Labs review:

I populated our TS-409U with four of the best price/capacity hard drives I could source. As of today, at least, the sweet spot is 640 GB drives. Total price was $799 for the device, and $60 each for the drives, so around $1,200 total with tax and shipping.

Our Database Ninja^tm, Brent Ozar, gave me a little flak for investing in such a low-end NAS device. It’s true, for around $1,400 I could have built yet another quad-core, 2 drive mirror Lenovo RS110 server. However, I felt that having an embedded device dedicated to backups — and one with four drive bays that can do better than RAID 0 or 1 — was the more logical choice. But I agree, it would have been nice to have another full-blown server to pinch hit for scalability or anything else. It’s a grey area.

As for setting up the QNAP TS-409U, I was a little annoyed that you have to run a Windows or Mac executable for the initial setup, instead of just booting it and configuring everything through the comprehensive (and very nice) web UI. But it detected my drives and configured OK, at which point I could use the web UI for everything else:

Seventeen (!) hours later, the array was fully synced and ready for benchmarking. RAID 6 is just like RAID 5, but with one more parity block, so it can survive two drive failures instead of just one.

The downside of picking a low end NAS is speed. The relatively anemic 500 MHz embedded CPU, combined with software-only RAID and the inherent write penalty of RAID 6′s dual parity calculation, means … well, this is not what I would call a “fast device”. Here are some quick benchmarks I ran:

For our intended use, storing a bunch of large single file daily database backups, blazing speed isn’t particularly important. Now that we have a 1+ TB backup device on the local network, we can store a huge history of daily backups. These check in at about 5GB per day currently, with a growth rate of 50 – 100 MB per day. This also takes some write pressure off the database server, since whatever backups we take will be spooled to the NAS over ethernet.

We plan to complement our onsite daily backups with informal offsite “sneakernet” backups. As I’ve mentioned before, we are fortunate to have Stack Overflow team member Geoff Dalgas live about a mile from the data center. We bought him a 500 GB 2.5″ USB drive, so he can head over the data center every 2 months or so and copy some backups off to keep in the (hopefully) unlikely event something terrible happens to our data center.

This is the 42nd episode of the StackOverflow podcast, where Joel and Jeff discuss ethical email, backup strategies, how to learn new programming languages, and dealing with underperforming developers.

• The Conversations Network, a non-profit organization that graciously underwrites the bandwidth costs of this and many other great podcasts, is looking for a sponsor. Email us at podcast@stackoverflow.com if you know of any
• We finally rolled out email support at Stack Overflow. If you haven’t been to the site in 7 days, and have provided a valid email address, we include all the responses to your questions and answers (if any) in that period. And of course there is a true one-click unsubscribe. We’re still tweaking the parameters of how it works — what is the optimal email relationship between a user and a website?
• Sending email these days is a bit of a minefield. How do you avoid instantly going into people’s spam folder? One key piece is having a Reverse PTR record, which is set up at the ISP level. There’s a whole “Deliverability” industry around sending email to people.
• We’re encouraged by the emerging standard of entering your OpenID provider’s address as your OpenID login. For example, “yahoo.com” works for Yahoo OpenIDs, and eventually “gmail.com” will work for Google. (Today you must use “google.com/accounts/o8/id” for Google, which is not optimal for hopefully.. obvious.. reasons.) Microsoft is also coming on board, though their OpenID support is in private beta.
• We also implemented gold and silver tag-based badges, based on upvotes within a tag. It’s a way of rewarding people who participate heavily in certain topic areas. We did have to rule out discussion based questions for this algorithm to work. We are also considering a tag leaderboard, as suggested by Greg Hewgill.
• Our backup strategy has been half-hearted so far. To improve this, we invested in an inexpensive embedded Linux based 1u Network Attached Storage device, the QNAP 409u. This will become our dedicated backup device. It has four drive bays and supports RAID 6 (dual parity). Kind of a neat little device; there’s a whole subculture of inexpensive NAS devices I hadn’t explored until now. Drobo, for example.
• As it turns out, the cost of bandwidth ends up being the gating factor for us when dealing with our daily multiple – gigabyte database backups. Jim Gray had an eye opening piece on the economics of bandwidth and the surprising effectiveness of “sneakernets”, even today.
• How likely is it that your datacenter is going to explode? Unless you have a fancy multiple datacenter setup for redundancy, it might be more effective to do some trickle uploads to services like Amazon S3, or even some monthly datacenter driving runs to copy data off using a cheap USB 2.5″ hard drive. Luckily, one of our team members lives a mile from the data center, so that’s the approach we’ll be using.
• We had some semi-serious issues with our IBM ServeRAID 8k controller, having to do with write-through versus write-back caching. Write-through blocks on actual disk writes, whereas write-back writes to a fast RAM buffer, returns very rapidly, and spools the writes over time (e.g. “lazy writes”). The performance of write-back is dramatically better, but we were seeing some eventual system-wide I/O blocking under heavy write load with write-back caching on. Supposedly this is normal for some RAID controllers, but we opted to downgrade to write-through because the nightly backups would always trigger this behavior for us.
• Speaking of blocking: it’s funny how many of the techniques discussed on the High Scalability blog boil down to hashtables in memory. Memory is one of the fastest things you have in a computer, and it almost never blocks for any significant amount of time. Unlike, say.. hard disks or network.
• The act of trying to learn a new language will make you a better developer. Where do you go if you only know PHP? I think you should go to Java or C# to build up a bit more structure, then to Python or Ruby to tear down that structure, as a sort of natural ebb and flow progression journey that I’ve seen a lot of developers make over time. Joel disagrees, and thinks thinks you should go for something radical right out of the gate, like Haskell. “At some point it has to blow your mind, or you’re not learning.”
• Joel recommends the classic Abelson and Sussman lectures on Structure and Interpretation of Computer Programs. This of course complements the freely available on-line version of the SICP textbook. Go blow your mind! “Once you learn those concepts, you’ll write better code in any language.”
• Is it an legitimate argument to say that, if you haven’t tried something, you’re not entitled to have an opinion about it? How much does it add to your opinion to have experience in a subject? Joel references Paul Graham’s How to Disagree.
• Shouldn’t every software project you work on be a learning experience, regardless of whether the software ever sees the light of day? Why would you work on a project that isn’t teaching you anything at all?
• Is it possible to help programmers who can’t help themselves? We’re not sure throwing a copy of Code Complete at a sub-par developer will necessarily make them any better. I believe pair programing and mentoring is the only way to get this to work, insofar that it ever does. And tread very lightly here, because it’s quite possible to make things worse by being didactic or overbearing.
• How do you fix not caring? How do you fix lack of motivation? Perhaps the better way to look at this is to keep your project inherently interesting and relevant. Convince your teammates that you’re working on something that matters, at least in some small way. Are we happy with what we’re doing? And if not, how can we fix that?

We answered the following listener questions on this podcast:

1. Michał Tataranowicz: “I am a PHP web developer. I want to learn a new language to improve my skills. What language should I learn? It doesn’t have to be useful for web development, although it would probably help.”
2. Adam from PA: “I work for a R&D group at a large defense contractor, I’ve been told my project is ending. How do I keep my motivation going when I know this project is going to be put on a shelf and not used?”

Our favorite Stack Overflow qustions this week are:

• When to Drink When Listening to the Stack Overflow Podcast. Are you guys listening to the same podcast as us? We never do this stuff! I have a PMP certification, and the answer just happens to always be 6-8 weeks.
• Tips on how to inoffensively help an older co-worker to become a developer instead of a code-monkey? This is a tough one.

If you’d like to submit a question to be answered in our next episode, record an audio file (90 seconds or less) and mail it to podcast@stackoverflow.com. You can record a question using nothing but a telephone and a web browser. We also have a dedicated phone number you can call to leave audio questions at 646-826-3879.

The transcript wiki for this episode is available for public editing.

Podcast 42 [ 1:10:35 ] Play Now | Play in Popup | Download