Twitter deals in data, and, at this 2011 MySQL conference, Jeremy Cole presents the specifics about how the data is stored, organized, and utilized. He gives this information in the context of several different subjects, such as employment, open source, software, and the social impact of Twitter itself.
Data redistribution, according to Jeremy Cole, is a major facet of upkeep in Twitter. Snowflake, which generates unique Twitter IDs, is one tool that Twitter uses to manage data and the massive number of members. This is also an example of open source data used by Twitter, as is the Gizzard framework, and Cole notes that Twitter generally shares the software they use by making it open source.
The Twitter site also uses other systems internally, including Cassandra, which is cheap, simple, schema-less, and has high velocity writes with low velocity reads. It also uses Vertica, which is 50-100 times faster than MySQL and good for structured data.
While the majority of Jeremy Cole's presentation is about the systems used to structure, store, and use the data in Twitter, he also pays close attention to the part that open source plays in the site: not only does it allow for easy innovation, portability, and storage, but people can continuously be a part of Twitter projects and infrastructure.
Jeremy Cole is a long time MySQL Geek and pioneer of MySQL scalability. Jeremy started his MySQL adventure ten years ago as the fourteenth employee of a startup: MySQL AB. His initial work was primarily on documentation and core development, soon transitioning into support, and finally running the training and consulting organization during his four and a half years there. After MySQL, Jeremy coined the “MySQL Geek” title at Yahoo!, supporting hundreds of internal users of MySQL and many thousands of MySQL servers within Yahoo!’s infrastructure, as well as building a large team of MySQL experts around the world. Jeremy went on to found one of the premier MySQL consulting companies, Proven Scaling, which during its more than three years of operation provided detailed advice, optimization, new tools, and hands-on DBA and emergency support to more than 100 influential companies.
This free podcast is from our MySQL Conference series.
Photo: JCole Weblog