Brian's Waste of Time
It would behoove us all to test what our sites look like in all popular browsers...
... with tabs enabled :-)
0 writebacks [/stuff] permanent link
Tue, 19 Jun 2007My name is Brian and I love lolcats.
It gets bonus points for making the jump into newspapers!
5 writebacks [/stuff] permanent link
Python and the Number of the Camel
The Number of the Camel shows its face again! Santiago pointed out that number 6 is weird in languages, and today Guido von Rossum points out that Python 2.6 will be concurrent with Python 3000!
0 writebacks [/src] permanent link
Scaling is the new theme of my programmering life it seems. The good part: it really means scaling, not performance. The bad part: things actually need to scale, performing really well doesn't let you squeak by with session affinity and half-assed replication. The biggest thing I have noticed, in myself, is a move from thinking about how to implement services to how to ensure that interactions between them don't become the bottleneck.
Sometimes this is a question of protocols in the IETF sense: HTTP, SMTP, DNS, XMPP, etc. Most of the IETF protocols are pretty damned good, as it turns out. Shocking huh? Many other times it is a matter of protocols in the traditional sense: what does the interaction need to look like. How many messages need to be sent, how quickly, how reliably? Who cares what is inside the box -- if between Martin, Diego, Tom, Tim, Lili, David and myself we cannot drag it to somewhere near enough an optimal solution, something is really wrong. What is inside a server is not the problem.
The problem is fun things like tolerating (or failing quickly in the face of) network partitions, managing work scheduling fairly across an arbitrary number of machines (or not, unfair scheduling frequently works better), avoiding cascading failures when one element goes down (and better yet, still working for the most part), knowing what is going on in the live system beyond a conceptual level, working out what is cacheable where, how, how long, how many, and so on. We've all dealt with these to some degree, but when they really become the key point it rocks!
The protocols between services are where the fun is, I had no idea! The turning point was probably the day "Enterprise" in a product description went from meaning "consultants required" to "probably cannot handle the load." Totally unfair to a lot of good products which have had Enterprise slapped on them by marketing, but those are sadly in the minority. When you say "we need to double the number of servers" then, minutes later, you have doubled the number of servers and things are copacetic, life is good.
0 writebacks [/ning] permanent link
When I started this blog four years ago I figured I would generally avoid talking about work. The most interesting stuff I was playing with was outside of work, and it seemed the safe and sane course.
Then I started at a place doing awesome stuff and suddenly the most interesting stuff I am doing is at work. D'oh! This has lead to me really wanting to talk about a lot of things and running into the "wait, this is work."
Net: I am going to start talking about work, it is the most interesting programmery stuff I am doing :-)
0 writebacks [/ning] permanent link
Sat, 16 Jun 2007
I was writing something to aggregate a bunch of JMX exports today and was really annoyed at adding environment variables to the test runner. Luckily, Martin pointed me to a fantastic howto on exporting an MBeanServer without supplying env variables to the jvm when it starts up. I specialized what he did a bit to orient it towards testing stuff and wound up with:
package org.skife.test;
import javax.management.MBeanServer;
import javax.management.remote.JMXConnectorServer;
import javax.management.remote.JMXConnectorServerFactory;
import javax.management.remote.JMXServiceURL;
import java.io.IOException;
import java.lang.management.ManagementFactory;
import java.net.InetSocketAddress;
import java.net.ServerSocket;
import java.rmi.registry.LocateRegistry;
import java.util.HashMap;
import java.util.Map;
import java.util.Random;
/**
* Handy-dandy tool to help with JMX Testing
*/
public class JMXHelper
{
private static int port = -1;
/**
* Ensure JMX is all set up, returns port for JMX RMI Registry
*/
public static int initialize()
{
if (port != -1) return port;
try {
final ServerSocket sock = new ServerSocket();
sock.bind(new InetSocketAddress(0));
port = sock.getLocalPort();
sock.close();
LocateRegistry.createRegistry(port);
MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
String s = String.format("service:jmx:rmi:///jndi/rmi://:%d/jmxrmi", port);
JMXServiceURL url = new JMXServiceURL(s);
JMXConnectorServer cs =
JMXConnectorServerFactory.newJMXConnectorServer(url, null , mbs);
cs.start();
}
catch (Exception e) {
throw new IllegalStateException("Unable to bind JMX", e);
}
return port;
}
}
Now you can connect to it like normal:
String url = String.format("service:jmx:rmi:///jndi/rmi://localhost:%s/jmxrmi", port);
JMXConnector c = JMXConnectorFactory.connect(new JMXServiceURL(url));
and there is a JMXConnector to your "test" MBeanServer. As with most of JMX, it is kinda werid looking and obscure, but it works well :-)
Update: Martin pointed out that I could use less code. Changed.
0 writebacks [/src/java] permanent link
Mon, 11 Jun 2007
David Pollak mentioned, in comments, about Scala's advantages over Erlang. I wanted to comment that, while the concurrency stuff in Erlang is all nice, and so on, it doesn't stop at the actor model. From my perspective, the support for distributed computing built into it so naturally is an equally killer feature. How distribution works falls naturally from how concurrency works, and global and rpc being built in to the standard library is non-trivial. More processors in a machine is nice and all, but it is still way cheaper, and I think will be for a while yet, to throw a couple thousand two core (soon eight or sixteen core) machines at a problem than one four thousand core machine.
I still believe that whatever the next dominant language is, it will look and smell a lot like Scala. Scala is darned tootin cool, even if the level of magic in it scares me (inside joke for David E., who still lacks a blog).
1 writebacks [/src] permanent link
Tue, 05 Jun 2007There is this strange meme that "$BigCo owns Apache!" where $BigCo changes periodically. For a long time $BigCo was Sun, but recently IBM seems to be the choice of the paranoids, and Google or Joost of those "in the loop."
To set the record straight, we own Apache. We Those are individuals, but in the interest of helping conspiracy theorists, most people are happy to list their employer or affiliation if you want to peruse :-)
0 writebacks [/apache] permanent link
Tue, 29 May 2007[tongue-in-cheek]As I don't use scheme heavily this is serious armchair controversy, but as R6RS stands, I think a language rename to scherl or peme might be appropriate to include.[/tongue-in-cheek]
2 writebacks [/src] permanent link
Wed, 09 May 2007
Long Tail Treasure Trove Slides
I just uploaded an initial export (Powerpoint -> Keynote -> PDF messed some things up) of the slides from Gianugo and my presentation at JavaOne last night, "Long Tail Treasure Trove." Thank you to all the attendees, you were awesome!
If I get a chance, later today, I'll clean up some of the import/export oddities and re-post them!
ps: Henri pointed out a typo in the sample ANTLR grammar, the POSITIONAAL_PARAM production. Sadly, that isn't a typo in the slide. The token name is spelled that way in jDBI... D'oh!
1 writebacks [/src/java] permanent link
Thu, 26 Apr 2007This needs to be highlighted. Thomas Mueller, the creator of H2, likes to point out the huge performance difference between H2 and other databases. It is extremely important to note that the main driver behind the performance difference is the fact that H2 explicitely forgoes durability. It does not sync data to disk on commit. Bears repeating, H2 does not sync data to disk on commit. You can force it to, in which case the performance hits the same bottleneck every other database hits, disk io:
-- 1024 bytes, 1000 times --
derby: 2192 millis
bdb: 1849 millis
h2: 2221 millis
h2b: 2351 millis
-- 2048 bytes, 1000 times --
derby: 2199 millis
bdb: 2129 millis
h2: 2578 millis
h2b: 2414 millis
The numbers here are running simple inserts and deletes by identity (so that I can include BDB JE). Shockingly, the numbers are all basically the same once you force H2 to sync to disk (executing a "CHECKPOINT SYNC" statement after every DML operation).
H2 is an awesome little database, but if you care about storing your data, please read the fine print where it notes that H2 is not ACID. It is just ACI.
10 writebacks [/src/java] permanent link
Wed, 25 Apr 2007Cynos says...
I believe that the Bush government is deliberately with-holding YARV, in order to increase profits for his oil drilling buddies.
In response to Toby's excellent post on the shortcomings of the Ruby runtime.
0 writebacks [/src/ruby] permanent link
Sun, 15 Apr 2007David says, "Rather than just sit around idle waiting for some vendor to fix your problems, you get the unique chance of being a steward of your own destiny," in reaction to Alex (of Twitter) saying Ruby is slow and Rails has no way to talk to several databases. A particular reaction to David's statement is where it went from yawn to what!?
Robert asserted that "when you are trying to build a business you don't have the luxury of digging through a ton of source code and then try to figure out the where the problems are and make changes." I disagree. You don't have the luxury of doing so, you have the requirement to do so. To put it simply, if you rely on a particular piece of software, and you need something changed, then you need it to be changed, and that is that. It is up to you.
Now, just because it is up to you doesn't mean you need to go implement the fix. Heck, you probably cannot even do it yourself unless the library is both open source and openly developed. If development is closed, your "option" is generally to throw money at the people who wrote the bug to make said bug go away.
If on the other hand you do have the option of fixing it yourself, you still rarely have to. An awful lot of the time merely pointing out the problem to folks already involved with the project will lead to it getting resolved. If just pointing it out won't get it fixed fast enough for you, it is downright shocking how many people will let you pay them to write software, so you can just pay someone to do it. If you don't have the funds or access to the information you need, you can play weird PR games to "force" them to do it (this actually works for security related stuff, frequently). If all else fails, you just fix the damned thing. That is about it: 1) ask nicely, 2) ask nicely with dollars on top, 3) threaten or bluster, 4) get sued by Nike, er, I mean, just do it.
The fact of the matter is that you need something changed in a library you use. You use this library, presumably, because it saves you a bunch of work. So you have to do a little work to still save you a bunch more work. (You might, alternately, decide that the amount of work you have to do to fix it is greater than the amount you saved, in which case you accept you made a bad call, and go do the right thing.)
Now, a key to understanding this is to realize that the people who wrote this library that is saving you a a bunch of work did it to save themselves some work. Frequently others chipped in as well. Shockingly, over time, you wind up with something like this, for example,
brianm@golem:~$ nc -v thebull.macsimumweb.com 80
DNS fwd/rev mismatch: thebull.macsimumweb.com != 209-59-185-60.webserversystems.com
thebull.macsimumweb.com [209.59.185.60] 80 (http) open
HEAD / HTTP/1.1
Host: thebull.macsimumweb.com
Connection: close
HTTP/1.1 200 OK
Date: Sun, 15 Apr 2007 05:56:30 GMT
Server: Apache/1.3.37 (Unix) mod_gzip/1.3.26.1a mod_fastcgi/2.4.2 \
mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 \
FrontPage/5.0.2.2635.SR1.2 mod_ssl/2.8.28 OpenSSL/0.9.7a PHP-CGI/0.1b
X-Powered-By: PHP/4.4.4
Connection: close
Content-Type: text/html
brianm@golem:~$
where a lot of folks contributed to apache, mod_gzip, mod_auth_passthrough, mod_log_bytes, mod_bwlimited, mod_ssl, OpenSSL, and PHP, so that the combined efforts have enabled a blog. Almost all of them said "oh shit, I need [foo], and [bar] doesn't do it" so went and did it. For the economics of it, go read Yochai Benkler. The gist is "you get out more than you put in."
But, you say, "Gee, I would love to write acts_as_infinitely_scalable but that pesky DHH guy won't let me change things in Rails, only the 'core team' get to do that." First off, that ain't true, they accept patches and have a good record of applying them. Second off, damned straight it is easier for them to do it -- they earned that trust. (Actually, acts_as_infinitely_scalable would probably be declined in favor of making it a plugin.)
They earned that trust for a number of reasons, but I will bet you pennies to bricks of gold that the majority got there because they relied on the project and needed more control over it. Maybe they rely on it because they are consultants who use it as advertising, possibly they need control over it in order to build up their personal ego, not infrequently they just needed increased responsibility in order to become better programmers, but most likely (I am not going to go find the citations, this is a blog not a journal) they were building products which relied on it. I doubt a single one thought about it in terms of "I need more control over [baz]," but in the end, that is what it was.
If you rely on [something], and you see it needs [something else], you do a quick calculation in your head to decide if the value of [something] is greater than the cost of [something else] and either do [something else] for [something] or ditch [something] in favor of [yet another thing] that already does [something else].
Figuring out the value and/or cost of [something], [something else], and [yet another thing] can be a pain sometimes, but that is why we get paid the big bucks :-)
2 writebacks [/src/ruby] permanent link
Wed, 11 Apr 2007
Guice with Spring Transactions
I futzed a bit with setting up Spring transaction handling in Guice. It was shockingly straightfirward :-)
So, there are several ways to do it, but I did it via having a Module export both a DBI and apply the transaction interceptor. Frankly, I probably wouldn't use this class in a real project because I would just wire it up more specific to how I needed it. However, for the common case of one data source, and annotated transactions. Bingo!
package org.skife.jdbi.v2.unstable.guice;
import com.google.inject.Module;
import com.google.inject.Binder;
import com.google.inject.matcher.Matchers;
import javax.sql.DataSource;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.interceptor.TransactionInterceptor;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.transaction.annotation.AnnotationTransactionAttributeSource;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.skife.jdbi.v2.spring.DBIFactoryBean;
import org.skife.jdbi.v2.IDBI;
public class SpringTransactionalGuiceModule implements Module
{
private final DataSource ds;
public SpringTransactionalGuiceModule(final DataSource ds)
{
this.ds = ds;
}
public void configure(Binder binder)
{
final PlatformTransactionManager ptm = new DataSourceTransactionManager(this.ds);
final DBIFactoryBean bean = new DBIFactoryBean();
bean.setDataSource(ds);
try {
final IDBI dbi = (IDBI) bean.getObject();
binder.bind(IDBI.class).toInstance(dbi);
}
catch (Exception e) {
binder.addError(e);
return;
}
binder.bindInterceptor(Matchers.any(), Matchers.annotatedWith(Transactional.class),
new TransactionInterceptor(ptm, new AnnotationTransactionAttributeSource()));
}
}
This module reuses the jDBI Spring transaction integration glue magic stuff, hence instantiating the DBI instance via the DBIFactoryBean. It then binds Spring's TransactionInterceptor, telling it to base its tx magic on annotations via the AnnotationTransactionAttributeSource and the PlatformTransactionManager.
This could bind the platform transaction manager as well, but for now I don't need it, so I didn't.
To exercise it, I need another module which has my actual things in it, and I need to do something, like:
public static class ThingModule implements Module
{
public void configure(Binder binder)
{
binder.bind(Thing.class);
}
}
public static class Thing
{
private final IDBI dbi;
@Inject
Thing(IDBI dbi)
{
this.dbi = dbi;
}
@Transactional
public void excute(final Callback cb)
{
final Handle handle = DBIUtil.getHandle(dbi);
cb.call(handle);
}
}
public interface Callback
{
public void call(Handle handle);
}
The only transactional element is the Thing#excute method, and it is using the Spring 2.0 @Transactional annotation.
I like to use callbacks for testing transactional stuff like this, reduces the clutter. The only really nasty bit to this, imho, is the Handle handle = DBIUtil.getHandle(dbi); which uses the Spring idiom of having a static helper to get transactionally bound resources. I dislike it, but it is Spring idiom, and I am reusing the jDBI Spring stuff, so... c'est la vie for now.
Setting it up and running, then is just:
public void setUp() throws Exception
{
super.setUp();
this.guice = Guice.createInjector(new ThingModule(),
new SpringTransactionalGuiceModule(Tools.getDataSource()));
}
public void testFoo() throws Exception
{
final Thing thing = guice.getInstance(Thing.class);
assertNotNull(thing);
try {
thing.excute(new Callback()
{
public void call(Handle handle)
{
assertTrue(handle.isInTransaction());
handle.insert("insert into something (id, name) values (?, ?)", 1, "Rob");
throw new IllegalStateException();
}
});
fail("Should have thrown an exception");
}
catch (IllegalStateException e) {
assertTrue(true);
}
final Handle h = openHandle();
assertFalse(h.isInTransaction());
final List<String> names = h.createQuery("select name from something")
.map(StringMapper.FIRST)
.list();
assertEquals(0, names.size());
}
And it all works! Woo hoo!
A drawback, and I haven't found a way around this, is that you cannot provide dependencies to interceptors, so the data source must be available outside the injector. Bob says it will be in the next release though, so woot!
4 writebacks [/src/java] permanent link
Sun, 08 Apr 2007Stumbled across Skim, an open source PDF reader for OS X. It crushes Preview. It has a good full screen mode, supports annotations/notes, and has a few other nifties. As the author puts it, "Skim is designed to help you read and annotate scientific papers in PDF. Skim includes many features and has a great user interface. Stop printing and start skimming."
0 writebacks [/osx] permanent link
Sat, 31 Mar 2007Yesterday, Toby suggested that YAWS might be a great option for Comet. Poking through the YAWS docs I think he is right.
0 writebacks [/src/erlang] permanent link
Sat, 24 Mar 2007I've been playing with Guice, and to repeat myself, Bob and Kevin know how to write a clean client API, that is for sure :-) In terms of IoC/DI/Flooflah I think Guice has finally knocked Pico out for elegance. Spring does DI, and is fantastic as a library, but I have never liked its dependency resolution options and how it functions container-wise.
Looking at Guice I did run into the thing most folks latched on to immediately, it has no lifecycle hooks! Egads, how shall we live! This actually matters as almost everything I do, lately, is multithreaded and embarrassingly concurrent, though. Because of the oddities of the Java Memory Model, simple lifecycle hooks are awfully useful -- just a start/stop is enough, really. You can work around it by having bootstrapping classes, but... yuck. Anyway, I digress.
Then the obviousness of it smacks me upside my head -- lifecycle crap is orthogonal to dependency resolution, we have just gotten used to seeing them bundled together. So I started futzing with non-sucky ways to do lifecycles outside the container.
Lifecycle events are just events, and despite the trend away from it, I think first-class events trump implicit events. This means you pass the event to an event listener rather than have a no-arg method invoked. Heresy, probably, but oh well. So what is an event? Well, crap, really there is no reason to restrict it, so an event is any Java object.
Java's lack of closure support makes for craploads of one-method interfaces as event listeners. This is idiommatic, and frequently idiommatic is good as it reduces the number of concepts you have to understand to learn something new, but I always find it annoying. Spring and other containers have been pushing folks towards having the classes which would register lifecycle listeners just be the listener (via the no-arg method), so that heresy has already gained some traction. Bah, go with it. We can dispatch events based on annotating a method as an event listener and use the argument type to match which events it cares about. So, stab #1:
public class EventBus
{
private final List<Object> listeners = new CopyOnWriteArrayList<Object>();
public void register(Object listener)
{
listeners.add(listener);
}
public void fire(Object event) throws IllegalAccessException,
InvocationTargetException
{
for (final Object listener : listeners) {
for (final Method m : listener.getClass().getMethods()) {
if (m.getAnnotation(EventListener.class) != null) {
final Class[] params = m.getParameterTypes();
if (params.length == 1 && params[0].isAssignableFrom(event.getClass())) {
m.invoke(listener, event);
}
}
}
}
}
}
This is extremely inefficient, but it is a couple minutes of hacking. It is also easy to optimize by analyzing listeners when they are registered instead of when the event is fired. That also allows for detecting illegally shaped listener methods at registration time, instead of just ignoring them, blah blah blah. Regardless, it works for now. Our thing which wants to receive events looks like:
public class RockingChair
{
private final List<String> events = new ArrayList<String>();
@EventListener
public void startRocking(Start event)
{
events.add("start");
}
@EventListener
public void fallOver(Stop event)
{
events.add("stop");
}
public List<String> getEvents() {
return Collections.unmodifiableList(events);
}
}
So here we have event methods on a listener class, and support arbitrary events. Driving it just looks something like this.
public void testBasics() throws Exception
{
EventBus bus = new EventBus();
RockingChair chair = new RockingChair();
bus.register(chair);
Start event = new Start();
bus.fire(event);
assertTrue(chair.getEvents().contains("start"));
}
public void testSubclass() throws Exception
{
EventBus bus = new EventBus();
RockingChair chair = new RockingChair();
bus.register(chair);
Start event = new Start() {};
bus.fire(event);
assertTrue(chair.getEvents().contains("start"));
}
Nothing fancy here, and it works fairly well. You can add lots of shiny knobs if you want to, but how do we go about hooking it into Guice? The obvious way is just to add the EventBus as a component and make things which depend on lifecycle events dependent upon it:
public class Glider
{
private final List<String> events = new CopyOnWriteArrayList<String>();
@Inject
public Glider(EventBus bus)
{
bus.register(this);
}
@EventListener
public void rock(Start event)
{
events.add("start");
}
@EventListener
public void fallOver(Stop event)
{
events.add("stop");
}
public List<String> getEvents()
{
return Collections.unmodifiableList(events);
}
}
This bothers me though. I do like passing an event listener to the bus other than the Glider itself, it seems to be polluting the heck out of the Glider's API to have those event listener methods we refactor it just a touch:
public class Glider
{
private final List<String> events = new CopyOnWriteArrayList<String>();
@Inject
public Glider(EventBus bus)
{
bus.register(new Object()
{
@EventListener
public void rock(Start event)
{
events.add("start");
}
@EventListener
public void fallOver(Stop event)
{
events.add("stop");
}
});
}
public List<String> getEvents()
{
return Collections.unmodifiableList(events);
}
}
By flagging the method as an EventListener and dispatching based on parameter type we can now have pretty painless multiple-event listener instances. If you could have an anonymous inner class implement more than one interface you could accomplish the same with interface based listeners. Hmm, you could also hide that behind a library where you do something like:
bus.registerInterfaceBasedListener(new Object() {
public void onFoo() {}
publc void onBar() {}
}, BarListener.class, FooListener.class);
But you are then dispatching on instanceof which isn't any prettier and is not really more optimizable. Need to think on it a bit more :-)
Regardless, I rather like argument-type based dispatch of events from an event bus which accepts arbitrary instances as events, and it does play pretty nicely with Guice, or anything else. Much more mucking to consider, I think :-)
2 writebacks [/src/java] permanent link
Tue, 06 Mar 2007With Joe Armstrong's new book, Programming Erlang, out in beta form I think I need to visit Erlang again. My first (and last) endeavor was a short and sweet reverser server:
-module(reverser).
-export([start/1]).
handle_connection(ClientSock) ->
case gen_tcp:recv(ClientSock, 0) of
{ok, Stuff} ->
gen_tcp:send(ClientSock, lists:reverse(Stuff)),
handle_connection(ClientSock);
{error, Reason} ->
exit(Reason)
end.
server_accept(ServerSock) ->
{ok, ClientSock} = gen_tcp:accept(ServerSock),
Pid = spawn(fun() -> handle_connection(ClientSock) end),
gen_tcp:controlling_process(ClientSock, Pid),
server_accept(ServerSock).
start(Port) ->
{ok, ServerSock} = gen_tcp:listen(Port, [list, {active, false}]),
spawn(fun() -> server_accept(ServerSock) end).
At the time I was very impressed with Erlang's available documentation -- I was able to make that from not knowing anything in just a couple hours, and it worked! O frabjous day!
I'm only about 50 or 60 pages into Joe's book, but those pages, at least, are extremely well written and make clear not only how to do things but what Erlang is actually doing under the covers (to some degree) which makes a world of difference.
0 writebacks [/src/erlang] permanent link
Fri, 02 Mar 2007Enough folks have emailed me asking for the details on the totally-unscientific and Silly Micro-Benchmarks I did that I decided to paste the last response:
> Could you share your microbenchmark test suite?
ab from httpd-2.2.4
> How did you test?
Spent about 10 minutes per server configuring it to the best of my knowledge (which is actually reasonably good for most of them) and wrote the most efficient hello world application I could for each.
> Which hardware,
An Apple PowerMac 2x dual core 2.66ghz xeon thingie with 6 gigs of ram for the server, an powerbook 2x2ghz CoreDuo with 2 gigs of ram for the client.
> networking stuff
An airport 8 port 10/100 switch
> , and so on. How your test result can be reproducible?
Pretty much cannot be reproduced in a meaningful way, which is the point :-)
-Brian
Anyway, that is that. I have been having fun with commons-math tonight while debugging some stuff, so might have to do some more, but with stats to make them look more valid!
0 writebacks [/src/ruby] permanent link
Thu, 01 Mar 2007Martin wrote a great visualizer for our live traffic data. It basically just listens in on ActiveMQ and plots what people are on what apps:
Each bubble is an app on Ning, each dot a user :-) More info and a full size version are available.