timj is currently certified at Master level.

Name: Tim Janik
Member since: N/A
Last Login: 2011-04-03 01:59:01

FOAF RDF Share This

Homepage: http://timj.testbit.eu/


I'm a long standing GTK+ maintainer and also regularly or randomly hacking various other free software components. Of course I also have a couple of private projects that can be found on my homepage. Allthough i never user Advogato for blogging, you can find my personal ramblings on my homepage.


Recent blog entries by timj

Syndication: RSS 2.0

Tobin – Statistics from Webserver Logs

Reblogged from the Lanedo GmbH blog: During recent weeks, I’ve started to create a new tool “Tobin” to generate website statistics for a number of sites I’m administrating or helping with. I’ve used programs like Webalizer, Visitors, Google Analytics and others for a long time, but there’re some correlations and relationships hidden in web server [...]

Syndicated 2013-11-13 00:42:57 from Testbit

Open Source In Business at Campus Party Europe

Next Friday I’ll be giving a talk on Open Source In Business at the Campus Party Europe conference in the O2 arena, London. The talk is part of the Free Software Track at 14:00 on the Archimedes stage. I’m there the entire week and will be happy to meet up, so feel free to drop me a line in [...]

Syndicated 2013-09-02 00:24:56 from Tim Janik

Should We Include Documentation Builds In Tarballs?

Reblogged from the Lanedo GmbH blog: Would you want to invest hours or days into automake logic without a use case? For two of the last software releases I did, I was facing this question. Let me give a bit of background. Recently the documentation generation for Beast and Rapicorn fully switched over to Doxygen. [...]

Syndicated 2013-08-06 22:24:16 from Tim Janik

Tor exit node for less than a week

During a conference some while ago, Jacob Appelbaum gave a talk on the usefulness of the Tor project, allowing you to browse anonymously, liberating speech online, enabling web access in censored countries, etc. Jacob described how the anonymizing Tor network consists of many machines world wide that use encryption and run the Tor software, which are [...]

Syndicated 2013-07-25 09:44:33 from Tim Janik

5 Insights from the Black Swan

In the last few days I finished reading the “Black Swan” by Nassim Nicholas Taleb. Around last January I saw Günther Palfinger mentioning it in my G+ stream looked it up and bought it.

At first, the book seemed to present some interesting ideas on error statistics and the first 20 or 30 pages are giving good examples for conscious knowledge we posses but don’t apply in every day actions. Not having a trading history like the author, I found reading further until around page 100 to be a bit of a drag. Luckily I kept on, because after that Taleb started to finally get interesting for me.

Once upon a time…
One of the lectures I attended at university touched on black box analysis (in the context of modelling and implementation for computer programs). At first of course the usual and expected or known input/output behavior is noted, e.g. calculus it may perform or pattern recognition or any other domain specific function. But in order to find out hints about how it’s implemented, short of inspecting the guts which a black box won’t allow for, one needs to look at error behavior. I.e. examine the outputs in response to invalid/undefined/distorted/erroneous/unusual inputs and assorted response times. For a simple example, read a sheet of text and start rotating it while you continue reading. For untrained people, reading speed slows down as the rotation angle increases, indicating that the brain engages in counter rotation transformations which are linear in complexity with increasing angles.

At that point I started to develop an interest in error analysis and research around that field, e.g. leading to discoveries like the research around “error-friendliness” in technological or biological systems or discoveries of studies on human behavior which implies corollaries like:

  • To enable speedy and efficient decision making, humans generally rely on heuristics.
  • Displaying heuristic behavior, people must make errors by design. So trying to eliminate or punish all human error is futile, aiming for robustness and learning from errors instead is much better.
  • Perfectionism is anti-evolutionary, it is a dead end not worth striving for. For something “perfect” lacks flexibility, creativity, robustness and cannot be improved upon.

A Black Swan?
Now “Black Swan” defines the notion of a high-impact, low-probability event, e.g. occurring in financial trading, people’s wealth or popularity – events from an extreme realm. That’s in contrast to normally distributed encounters like outcomes of a dice game, people’s body size or the number of someone’s relatives – encounters from a mediocre realm.

From Mediocre…
Here’s a short explanation for the mediocre realms. Rolling a regular dice will never give a number higher than 6 no matter how often it’s thrown. In fact, the more it’s thrown, the more even it’s numbers are distributed and the clearer its average emerges. Measuring people’s weight or number of relatives shows a similar pattern to throwing a dice, the more measurements are encountered the more certain the average becomes. Any new encounter is going to have lesser and lesser impact on the average of the total as the number of measurements increases.

To Extreme…
On the other hand there are the extreme realms. In trading or wealth or popularity, a single encounter can outweigh the rest of the distribution by several orders of magnitude. Most people have an annual income of less than $100k, but the tiny fraction of society that earns more in annual income possesses more than 50% of the entire distribution of wealth. A similar pattern exists with popularity, only very few people are so popular that they’re known by hundreds of thousands or maybe millions of people. But only very very few people are super popular so they’re known by billions. Averaging over a given set only works for so long, until a high-impact “outlier” is encountered that dominates the entire distribution. Averaging the popularity of hundreds of thousands of farmers, industrial workers or local mayors cannot account for the impact on the total popularity distribution by the encounter of a single Mahatma Gandhi.

On Errors
Taleb is spending a lot of time in the book on condemning the application of the Gauss distribution in fields that are prone to extreme encounters especially economics. Rightfully so, but I would have enjoyed learning more about examples of fields that are from the extreme realms and not widely recognized as such. The crux of the inapplicability of the Gauss distribution in the extreme realms lies in two things:

  1. Small probabilities are not accurately computable from sample data, at least not accurately enough to allow for precise decision making. The reason is simple, since the probabilities of rare events are very small, there simply cannot be enough data present to match any distribution model with high confidence.
  2. Rare events that have huge impact, enough impact to outweigh the cumulative effect of all other distribution data, are fundamentally non-Gaussian. Fractal distributions may be useful to retrofit a model to such data, but don’t allow for accurate predictability. We simply need to integrate the randomness and uncertainty of these events into our decision making process.

Aggravation in the Modern Age
Now Taleb very forcefully articulates what he thinks about economists applying mathematical tools from the mediocre realms (Gauss distribution, averaging, disguising uncertain forecasts as “risk measurements”, etc) to extreme realm encounters like trade results and if you look for that, you’ll find plenty of well pointed criticism in that book. But what struck me as very interesting and a new excavation in an analytical sense is that our trends towards globalisation and high interconnectedness which yield ever growing and increasingly bigger entities (bigger corporations, bigger banks, quicker and greater popularity, etc) are building up the potential for rare events to have higher and higher impacts. E.g. an eccentric pop song can make you much more popular these days on the Internet than TV could do for you 20 years ago. A small number of highly interconnected banks these days have become so big that they “cannot be allowed to fail”.

We are all Human
Considering how humans are essentially functioning as heuristic and not precise systems (and for good reasons), every human inevitably will commit mistakes and errors at some point and to some lesser or larger degree. Now admitting we all error once in a while, exercising a small miscalculation during grocery shopping, buying a family house, budgeting a 100 people company, leading a multi-million people country or operating a multi-trillion currency reserve bank has of course vastly different consequences.

What really got me
So the increasing centralisation and increasing growth of giant entities ensures that todays and future miscalculations are disproportionally exponentiated. In addition, use of the wrong mathematical tools ensures miscalculations won’t be small, won’t be rare, their frequency is likely to increase.

Notably, global connectedness alerts the conditions for Black Swan creation, both in increasing frequency and increasing impact whether positive or negative. That’s like our modern society is trying to balance a growing upside down pyramid of large, ever increasing items on top of its head. At some point it must collapse and that’s going to hurt, a lot!

Take Away
The third edition of the book closes with essays and commentary that Taleb wrote after the the first edition and in response to critics and curios questions. I’m always looking for relating things to practical applications, so I’m glad I got the third edition and can provide my personal highlights to take away from Taleb’s insights:

  1. Avoid predicting rare events
    The frequency of rare events cannot be estimated from empirical observation because of their very rareness (i.e. calculation error margin becomes too big). Thus the probability of high impact rare events cannot be computed with certainty, but because of the high impact it’s not affordable to ignore them.
  2. Limit Gauss distribution modeling
    Application of the Gauss distribution needs to be limited to modelling mediocre realms (where significant events have a high enough frequency and rare events have insignificant impact); it’s unfortunately too broadly abused, especially in economics.
  3. Focus on impact but not probability
    It’s not useful to focus on the probability of rare events since that’s uncertain. It’s useful to focus on the potential impact instead. That can mean to identify hidden risks or to invest small efforts to enable potentially big gains. I.e. always consider the return-on-investment ratio of activities.
  4. Rare events are not alike (atypical)
    Since probability and accurate impact of remote events are not computable, reliance on rare impacts of specific size or around specific times is doomed to fail you. Consequently, beware of others making related predictions and/or others relying them.
  5. Strive for variety in your endeavors
    Avoiding overspecialization, learning to love redundancy as well as broadening one’s stakes reduces the effect any single “bad” Black Swan event can have (increases robustness) and variety might enable some positive Black Swan events as well.

What’s next?
The Black Swan idea sets the stage for further investigations, especially investigation of new fields for applicability of the idea. Fortunately, Nassim Taleb continues his research work and has meanwhile published a new book “Antifragile – Things that Gain from Disorder”. It’s already lying next to me while I’m typing and I’m happily looking forward to reading it. ;-)

The notion of incomputable rare but consequential events or “errors” is so ubiquitous that many other fields should benefit from applying “Black Swan”- or Antifragile-classifications and corresponding insights. Nassim’s idea to increase decentralization on the state level to combat escalation of error potentials at centralized institutions has very concrete applications at the software project management level as well. In fact the Open Source Software community has long benefited from decentralized development models and through natural organization avoided giant pitfall creation that occur with top-down waterfall development processes.

Algorithms may be another field where the classifications could be very useful. Most computer algorithm implementations are fragile due to high optimization for efficiency. Identifying these can help in making implementations more robust, e.g. by adding checks for inputs and defining sensible fallback behavior in error scenarios. Identifying and developing new algorithms with antifragility in mind should be most interesting however, good examples are all sorts of caches (they adapt according to request rates and serve cached bits faster), or training of pattern recognition components where the usefulness rises and falls with the variety and size of the input data sets.

The book “Black Swan” is definitely a highly recommended read. However make sure you get the third edition that has lots of very valuable treatment added on at the end, and don’t hesitate to skip a chapter or two if you find the text too involved or side tracking every once in a while. Taleb himself gives advice in several places in the third edition about sections readers might want to skip over.

Have you read the “Black Swan” also or heard of it? I’d love to hear if you’ve learned from this or think it’s all nonsense. And make sure to let me know if you’ve encountered Black Swans in contexts that Nassim Taleb has not covered!

Syndicated 2013-07-16 17:02:27 from Tim Janik

22 older entries...


timj certified others as follows:

  • timj certified raph as Master
  • timj certified federico as Master
  • timj certified miguel as Master
  • timj certified macricht as Journeyer
  • timj certified stric as Journeyer
  • timj certified Adrian as Journeyer
  • timj certified shawn as Journeyer
  • timj certified tigert as Journeyer
  • timj certified lewing as Journeyer
  • timj certified hp as Master
  • timj certified andersca as Master
  • timj certified jacob as Journeyer
  • timj certified nether as Journeyer
  • timj certified vicious as Journeyer
  • timj certified jrb as Journeyer
  • timj certified clahey as Journeyer
  • timj certified LotR as Journeyer
  • timj certified yosh as Master
  • timj certified flaggz as Journeyer
  • timj certified kenelson as Journeyer
  • timj certified bit as Journeyer
  • timj certified jmacd as Master
  • timj certified xach as Journeyer
  • timj certified jlbec as Journeyer
  • timj certified sjburges as Journeyer
  • timj certified alan as Master
  • timj certified Guillaume as Journeyer
  • timj certified Slow as Journeyer
  • timj certified cameron as Apprentice
  • timj certified pavlov as Journeyer
  • timj certified terop as Journeyer
  • timj certified neo as Master
  • timj certified shaver as Master
  • timj certified notzed as Journeyer
  • timj certified feldspar as Apprentice
  • timj certified johnsonm as Master
  • timj certified tml as Journeyer
  • timj certified mjs as Journeyer
  • timj certified pat as Journeyer
  • timj certified riel as Journeyer
  • timj certified martin as Journeyer
  • timj certified blizzard as Journeyer
  • timj certified jbuck as Apprentice
  • timj certified campd as Apprentice
  • timj certified Jimbob as Journeyer
  • timj certified chrisd as Apprentice
  • timj certified bertrand as Apprentice
  • timj certified jsh as Master
  • timj certified jamesh as Master
  • timj certified terral as Apprentice
  • timj certified kelly as Apprentice
  • timj certified justin as Apprentice
  • timj certified Ricdude as Apprentice
  • timj certified lupus as Apprentice
  • timj certified eskil as Apprentice
  • timj certified Raphael as Journeyer
  • timj certified DV as Journeyer
  • timj certified happybob as Apprentice
  • timj certified jonas as Journeyer
  • timj certified mathieu as Apprentice
  • timj certified Telsa as Journeyer
  • timj certified dcm as Master
  • timj certified rms as Master
  • timj certified munizao as Apprentice
  • timj certified mitch as Journeyer
  • timj certified aersoy as Apprentice

Others have certified timj as follows:

  • raph certified timj as Journeyer
  • hp certified timj as Master
  • lewing certified timj as Master
  • stric certified timj as Journeyer
  • clahey certified timj as Master
  • vicious certified timj as Master
  • flaggz certified timj as Master
  • yosh certified timj as Master
  • jacob certified timj as Master
  • Centove certified timj as Master
  • sjburges certified timj as Master
  • Slow certified timj as Master
  • shawn certified timj as Master
  • bit certified timj as Master
  • andrei certified timj as Master
  • ole certified timj as Master
  • cameron certified timj as Master
  • neo certified timj as Master
  • Acapnotic certified timj as Master
  • feldspar certified timj as Master
  • mjs certified timj as Master
  • harold certified timj as Master
  • bombadil certified timj as Master
  • dcm certified timj as Master
  • jsh certified timj as Master
  • Raphael certified timj as Master
  • listen certified timj as Master
  • mitch certified timj as Master
  • mathieu certified timj as Master
  • aaronl certified timj as Master
  • gstein certified timj as Master
  • duncan certified timj as Master
  • lupus certified timj as Journeyer
  • jimmac certified timj as Master
  • odaf certified timj as Master
  • asmodai certified timj as Master
  • kelly certified timj as Master
  • Darin certified timj as Master
  • Adrian certified timj as Master
  • eskil certified timj as Master
  • dsueiro certified timj as Master
  • Guillaume certified timj as Master
  • nils certified timj as Master
  • harinath certified timj as Master
  • jonas certified timj as Journeyer
  • nelsonrn certified timj as Master
  • lauris certified timj as Master
  • nomis certified timj as Master
  • rodrigo certified timj as Master
  • jae certified timj as Master
  • jsheets certified timj as Master
  • dbartold certified timj as Master
  • timg certified timj as Master
  • jules certified timj as Master
  • jonkare certified timj as Master
  • inri certified timj as Master
  • bratsche certified timj as Master
  • timur certified timj as Master
  • motty certified timj as Master
  • jLoki certified timj as Master
  • jfleck certified timj as Master
  • jamesh certified timj as Master
  • adulau certified timj as Master
  • rw certified timj as Master
  • andersca certified timj as Master
  • ricardo certified timj as Master
  • murrayc certified timj as Master
  • gka certified timj as Journeyer
  • carol certified timj as Master
  • mathrick certified timj as Master
  • dbrock certified timj as Master
  • lucasr certified timj as Master
  • cinamod certified timj as Master
  • kfoltman certified timj as Master
  • henrique certified timj as Master
  • nedko certified timj as Master

[ Certification disabled because you're not logged in. ]

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

Share this page