Critical thinking

(writing started 16th Sep 2019)

I want to start a series about what has been going in lately in French research circles about sustainability and resilience. Because a lot of intelligent debate is happening there without the rest of the world participating, even though it concerns everyone. I will attempt to present all that to you in English here.

However this is a subject that introduces shocking knowledge that is delicate to present in a manner that won’t have the reader brush it off too quickly.

For the sake of credibility, let us start with the base: the method.

Epistemology

I want to introduce you to my personal preferred youtube channel: Hygiène Mentale. I suppose you don’t need the translation (just flip adjective with nouns and you get English).

It presents elements of critical thinking. Notably the video maker is a member of the “observatory of zététique”. Get the definition here Zététique (wiki).

And is more or less a crash course to spot fake news. Let’s list the point covered in the channel.

Principle of non prejudice

When encountering new elements of knowledge, for example a conference or an article, you must turn off all a-priori beliefs about the content. And especially all prejudice about the speaker. Dismissing elements because of the speaker, is a classic ad-hominem fallacy. Unfortunate equivalent judgment biases will happen if any prejudice isn’t silenced during the time of the study. You can reactivate them to update your judgment, at the end of the presentation.

For example, in the Episode 1, he joins a conference of mediums; the a-priori would be “it will be a pile of manipulative crap served to poor desperate idiots”, but you shut down that prejudice, and listen with honest innocent ears, and at the end, confront the content to the evidence. One of the talks was about the wakening of human superpowers in recent years, thanks to babies being born with 3 DNA strands. The source seemed to be a British article. After a simple search, the baby was handicapped, and it was nothing related to psychic powers. This is just an example of cherry picking and sensationalizing. The woman was selling consultations to discover children with psychic abilities.

Fallacies

Talking about fallacies, some are treated in Episode 12. Most common are the “old pot” ancient cultures are necessarily wiser – fallacy. “ad populum” – everybody knows / calling to agreement from crowds – fallacy. “call to authority” it’s not because the dude has a white blouse that the speech is automagically is true. “call to nature” with the example of pseudo-medicinal plants.
He gives an example in marketing, at the back of a bottle of lemonade, the text is a pile of fallacies. “ancient recipe!” “loved by all the connoisseurs!”

publicitefausse

A good list can be found on this website that I recommend: https://yourlogicalfallacyis.com/.

Knowledge versus belief

In episode 19 he speaks about logic applied to debate.

set-of-knowledgeBlack cloud: theory / models / explanations
Dots: observations
Orange cloud: possibly know-able field
Grey zone: never-knowable. e.g. outside of observable universe / god …

[My personal 2 cents after losing too much time on philosophy.stackexchange: Logical propositions can only be stated, while the discussion is inside the domain of knowledge. Otherwise said: logic doesn’t apply to beliefs about unicorns or flying teapots.]

Next he presents a plane, with the axis of knowledge (proofs) vertically (Y)
and the axis of belief horizontally (X).
He notes that the word “belief” has many senses, like “I believe I forgot my keys”, or “I believe in you”. But here it means: reconciled mental model / things accepted / “I think that“…
One can be rational only when one respects a rigorous intellectual position of situating oneself on the central diagonal of: «I believe because I know» – «I have no belief because I don’t know» – «I believe that not, because I know that not».

position_rationelle
X: belief, Y: knowledge. On the left: ‘denial’, on the right ‘leap of faith’; and in green: the rational stances.

Later, he uses that 2D plane, cut in 9 quadrants to place beliefs in theism, and makes sure to precisely define theism, atheism, deism. And corrects a common abuse of the word “agnostic” which doesn’t refer to the axis of belief like usually misused, but to the axis of knowledge.

axedecroyance

In the above figure, belief is horizontal, knowledge is vertical, and the position of rationality is X=Y (the center diagonal). The fillings in the graph are an example of application of this graph to religion.
Note that «I don’t believe» (have no belief), is very different to «I believe that not». “not believing” means the absence of belief. Which is the rational position when there is no proof. He states, multiple times «no proof = no reason to believe». He says this is a zetetician’s proverb.

Humans can live with false beliefs, because what matters for survival is the cohesion of the group. This is explained in the article {this article won’t change your mind} by Julie Beck from theatlantic. This is a very long article so just 2 quotes:
_«people wear information like team jerseys. Especially because a lot of false political beliefs have to do with issues that don’t really affect people’s day-to-day lives.»
_«Having social support, from an evolutionary standpoint, is far more important than knowing the truth.»
(Farhad Manjoo is quoted a lot)

Proof-claim weight equivalence

“An extraordinary claim requires more than ordinary evidence” also called the Sagan standard (but really is from Pierre-Simon Laplace). And leads to “the burden of proof“, notably as explained by Betrand Russel. And finally two principles: The Occam’s razor, and the Hanon’s razor. I’ll let you google all that if you’re interested; and hint one thing: the Invisible Pink Unicorn (IPU) is the same thing as the Russel’s teapot.

The youtube channel “la statistique expliquée à mon chat” (channel which received the Wernaers first price 2017 for scientific vulgarization; and are sometimes doing cross references with hygiene mentale) gives the following example:

if my friend claims to own a frog at home, I believe him, because the claim is not extraordinary.
But if my friend claims to possess a frog-who-speaks, I have a serious doubt.

One of the interesting graphs Christophe presents:
niveauxdepreuvesThis is titled “the levels of proof“, from bottom to top (reverse order):
– lowest is “popular wisdom, rumors”
– next to low “reported testimony” (I know someone who..)
– low “personal anecdote” (it works for me)
————- scientific threshold ————–
– one scientific study (randomized, blinded experiment)
– replications (scientific consensus)

Media chain deformation

In episode 7 he speaks about «why is there so many falsehoods on internet?» and «why is it shared so much without verification?».
When people are comforted in what they already believe they become much less critical.
He takes the example of a news titled “serious revelations: 600 UK soldiers are coaching ISIS soliders”. And makes it a demonstration for his method of verification.

The first thing to do: go back to the source.

In the article we read “the daily star said…”, and link to the source at the bottom of the site, but the link is never directly the daily star. Following the trail, finally the original seems to be the blog of a political activist. Finding the real daily star article required to look for it independently. The title was click bait, but the body of the article is nuanced. It said «”We know there are people with English accents and a military background training members of IS”» The original phrase with the figure 600 is a catastrophic example of bad writing. It makes so little sense I won’t even reproduce it here. Anyway, here is Christophe’s source analysis:

sourcebacktracking
Imagine tracing backward from right to left. Your first encounter is a retweet.

He calls that the effect skin of sorrow. Where substance disintegrate through your fingers.

Don’t hesitate to take a pen and reconstruct the graph of information flow on paper when doing that. Learn how to use google reverse image search, or the date limiter in the query. Also use google scholar, to limit results to scientific research when looking for linked studies.

Disinformation

In that same episode 7 he presents echo chambers, and how above-mentioned sites are all copying each other without checking the origin of the information. It can give a false illusion of consensus to people drowned in the meanders of such networks.
reinformationsites

Example of an illustration to serve nationalist propaganda:
polefakenewsWith google chrome you can right click on the image and do “search web for this image“. Or manually copy link to image and go to reverse image search.
googlereverseYou’ll immediately find that this image was used by an insurance company in Australia for an advert campaign.
He suggests to make your own black list, where any site that didn’t bother verifying an article that is provably fakenews, like this pole story, should go straight to your bin.

Later he peeks into a documentary where the makers managed to score an interview with the originator of one of those fakenews. And the man casually is presented in front of 2 factual manipulations of his doings, a picture of a building with Algerian flags at the windows, claiming it’s in Paris, but the picture was taken in Alger. Or same tactic with a crowded bus. He defends himself saying it’s “for illustration”. Another classic is a spread sheet with (phony and mistaken) calculations for earnings of a working couple, versus an unemployed couple earning full welfare, and ending up with comparable income. Immediately shared in the tens of thousands by angry nationalists waiting for any crumb that would go their way. Here: how those lazy ass migrants profit the system.

A lie can travel halfway around the world while the truth is putting on its shoes

In modern times we prefer to invoke what has become the “law of Alberto Brandolini” or the “bullshit asymmetry principle” which states:

The amount of energy needed to refute bullshit is an order of magnitude bigger than to produce it

Think fast and slow

In episode 20 he introduces the two speeds of thought. Referring to the book “think fast and slow” by Nobel price Daniel Kahneman. He describes a model with two systems of thought. The fast: innate, low energy. And the slow: methodological, scientific, and tiring!
Christophe’s introduction aims to introduce that duality, to solve a paradox that his viewers were surprised with: why should we grant more importance to an expert than an anonymous, while we were told that this the fallacy of argumentum at verecundiam (argument from authority) ? It seems contradictory, but only because this precept hasn’t been placed within the two speeds of thought system.
fast&slowKahneman explains that as any evolved machine we have energy savings systems. And the fast thinking track is one of them. When we’re not awake or critical enough that’s the active system. The second system, the slow track, is the costly one. Taking significant amount of time to research the subject, inspect the source of data, compare hypothesis; read scientific publications…
When you don’t have time to switch to system 2, the energy saver system 1, will have to do, and it is in this condition that dismissing anon’s theory, and giving credit to the expert, is rational.

To decide when to use the costly system, Christophe says he constituted for himself, a little track switcher at the entrance: the bullshit-o-meter.

If he sees the usual signs in the form of the message that corresponds to forms taken by fakenews in the past, his warning lights will flash. Like mentions of conspiracies, or dramatic music, or scary messages. When a warning light flashes, it should be the time to switch to the slow system. What he calls, the “analytic” system.

In the analytic system, indeed, the messenger doesn’t matter, but only the content of the message. Because you can take the time to analyze the claim. Recoup it with other publications, read the history of bibliography etc.

He mentions cases where the fast system can make mistakes. For example «there is aluminum in this vaccine, aluminum is toxic, therefore this vaccine is toxic». The reasoning sounds appealing, but there should be a warning light: “isn’t it an essentialist reasoning ? it’s the posology that we should be looking at. The balance risk/benefit”. And this takes time.

Perhaps the most important plug we should wire the bullshit-o-meter to, is our own “confirmation bias” trigger. Indeed we are but too quick to validate and share, when the information goes our way. Christophe says, in fast thinking, it is natural to confront the information to our knowledge a-priori, it’s bayesian. But there is an asymetry of tolerance for bullshit when the belief update goes OUR way. That’s why we must be doubly as critical as usual when an information is too beautiful for us, before rushing on the share button. Say you have a liberal capitalist core in you, and some data about the failure of a parecon experiment seems like godsend news to comfort you and your group, in your position. That’s when you should take a step back and switch to slow system, verify the source and the quality of the data. Likewise, if you have an anarchist core in you, and you come across a report of violent crowd control police beating up demonstrators, it’s the perfect opportunity to prove how you were right all along. No: it’s the perfect opportunity to verify if it’s not fakenews first.
Inversely, picture a news that hurts your conviction. Say you’re an environmentalist, and there is a news like “NASA reveals that there never has been more green”. The switch to analytic thinking system will work very well. But inversely a title like “municipal water contaminated with tritium for 6 million people !”, you’ll be tempted to share that, very swiftly. To show your friends you’re right to think what you think. That’s exactly when the risk of sharing bullshit is the highest.

Christophe goes on to say that the innate system (fast), is otherwise pretty good. It serves as a Judge Dredd against bullshit. An expeditious judge for truth and fake. The fast system is very deeply rooted in us. Result of millions of years of evolution, and that made us a social specie.
However conversely, the slow system (analytic), needs to be learned. It is not intuitive, requires to learn logic axioms, statistics, blind experiment method, scientific language, epistemology, suspension of judgment (principle of non prejudice).

Bayesian thinking

In episode 26 Christophe mentions how he stopped bickering with friends and families as he came to realize that each person’s knowledge base, is an iterative evolution from proof to proof, that leads to a certain position of beliefs. And of course when confronting them, if you disagree, even if you present evidence, your “opponent” doesn’t change his/her mind right away. This is frustrating but completely obvious is the wake of Bayesian thinking.

history

He introduces by mentioning that over a century, scientists from 2 clans, the clan of frequentists, and the clan of bayesians, are clashing about the scientific method.
Frequentists who say: «probability of events according to a certain theory»
Bayesians who say: «plausibility of theories according to certain events»

He presents the history of the original formula, Thomas Bayes writes it in the margin of personal notes under the form of an equality of set intersections: P(A).P(B|A)=P(B).P(A|B). Laplace finds it posthumously,  and divides by P(A) to express it under its know form today:
bayes-laplaceAnd publishes it in a memoir called «probability of the causes by the events».

Platon’s cave & science

Christophe constructs a thought experiment using “which dice” as a placeholder for “which hypothesis”. Constructing a situation with a game master picking one of the dice we don’t know about, and spells a result. What dice was used ?
bayesian-dice1The frequentist has to answer that all possible dice is an equally probable source→33%. Christophe asserts that this a case where the frequentist is wrong. Let’s demonstrate.

The analogy by construction here, is one dye one explanation, or one model, as such:
bayesian-methaphoreHe says it is directly linked to critical thinking, because it determines the “foundations of decision taking”, that we need to decide if we believe in that or this hypothesis.
Example: There is a noise coming from the ceiling. “What is that noise?”
bayesian-what-is-that-noise– It’s a ghost. I believe at 2%.
– It’s the cold that contracts woods and it cracks. I believe at 32%
– It’s my cat that walks in the attic. I believe at 66%
Here we reason in terms of plausibility of hypothesis, we evaluate and compare them.
The frequentist, refuses to reason this way; he takes one hypothesis, and if the result of the experiment are too implausible, we can reject the hypothesis. This method is the norm in scientific research. He explains that it was suggested by Ronald Fisher in the thirties. And that Fisher found the bayesian practice too complex and too subjective. Christophe goes on saying he agrees; but argue that “complex” is not an argument anymore, it’s 3 multiplications 1 division and 1 addition. And “subjective”, indeed ! Because of the presence of P(B), the probability a-priori of the hypothesis tested now. But that’s not a sad thing at all, since we already have plenty of subjectivity everywhere if you take a global view of science within society:
bayes-in-society
I would translate it, but it’d be letter-to-letter redundant. You can perfectly read this. Just remove some silent endings –e | change –f with –ve | and –ie with –y | and -ique by -ic.

What we retain is that science is already used in a subjective way. What Christophe argues, is that he would rather encompass this whole system INTO science itself, were we to be bayasians !

there is a lot to win to incorporate this subjectivity in the heart of the method, in a transparent way; instead of trying to hide it under the carpet.

Back with the dice experiment, the real answer was:
dice-bayes-first-iteration
On the left you’ll see the frequentist probabilities, and it will be our bayesien a-priori. A-priori all dice are equal, 20% chance then (5 dice→ 1/5). a 7 is drawn. What’s the update to our beliefs ? Well we can infer that D4 and D6 are impossible. We can say that for D20, a 7 is just one of twenty → 5% etc. In bayes formula the numerator is the surface created by the rectangles. And the denominator is the sum of these surfaces. Effectively acting as a normalizer. The result is on the right, the answer to the original problem was not 33% but something like 19%, 32% and 48%. D8 was more plausible. If you had to bet, there was a clear probable winner to bet on: D8. Whereas the frequentist would not have been able to pick a clear dye.
And this is iteratable, imagine all draws from now on, return a number never over 8, your confidence in D8 will grow and grow and reach 99% after a few dozens iterations.

Replace D8 by Einstein’s gravity, or second principle of thermodynamics. And you see that Bayes is there indeed, it’s just under the carpet, not quantified.

You’ll note that, say, after 230 draws, a 10 comes out. 99% falls to 0% instantly. That’s exactly what science is. #allmodelsarewrongcertainsareuseful

The rational open mind

This series effect, applies for our evolution of understanding about the world, we have “cursors of belief” that we tune, as we are presented with evidence along the course of our lives.
bayesian-chainThe part of subjectivity fades out as we update our beliefs cursor

Indeed in episode 28 Christophe presents a scale of «belief cursors».
levels-of-beliefsWhich represents the logarithm of the P(B) value in the Bayes formula. He says:

The pragmatic bayesian, manipulates orders of magnitude rather than percentages

For the math persons, the level of conviction is the logarithm of the ratio of conviction percentages, as such:convictionlevelThis way you can add the levels of proof to your current cursor position instead of multiplying the percentages. (log is a morphism)
And same thing for “levels of proof” = the log of the ratio of plausibility of the tested hypothesis.
As such, if you are now “level -3” on, say, cold healing by basil tea, and I present you a study which shows with certainty +4 (10.000 times more likely than not) that it works, you should now believe, very little (level +1), in cold healing by basil tea.

lessons

People stuck in their convictions, that would be level ±infinite, are breaking the bayesian machine. This position is what we call a dogma. This is not very sane. P(B)=0 or P(B)=1 should be banned. But on the same time, it is not mandatory to grant at least 1% or at max 99% to a position. e.g. 0.00001% is perfectly reasonable, that’s the position -5.

On the other hand, most rational persons are not going to be convinced the same way as you in the light of the same evidence. They can be perfectly open minded and rational, and yet still appear obtuse to your demonstration. Because the update you caused to their cursor, is just not enough to reach your current cursor, given their original position. Everyone starts with different a-priori.

One point Monsieur Phi makes (Mr Phi, yet another of those French youtubers) in his coverage of bayesianism; is that Bayes, is the fix for the hardest, wrongest, dirtiest fallacy of all; my all time first, the undethroned: “base rate fallacy. In modern times, where articles, studies, figures and occurrences are thrown to you from all directions. Indeed the base rate fallacy is what causes people the most to be irrational. Bayes is there to the rescue, lest you forget to think about him, if you wire his voice in the back of your head you’ll ask yourself: «what is the probability a-priori?».
Example: a test for disease trollititis is reliable at 99%. You pass this test and it returns positive. What are your chances to have contracted trollititis ? You’d think “well… 99% ? but I know there is a trap right ?” you’d be right. What is the probability a priori ? trollititis has an occurrence rate of 1 person out of 200 in the population, therefore in 200 persons it will fail for 199*1%=2 people, and most certainly be right for the 1 really sick. Your positive result is one of this bag of 3. Therefore, your probability of being sick is one of 3: 33%. Not 99% at all.

Conclusion of this paragraph, which is the longest in this article, is that bayesian thinking is (probably) the most important chapter in critical thinking. It shows that everything has subjectivity in it, it re-demonstrates mathematically that extraordinary claim requires extraordinary evidence; it is a great fallacy repellent, and a medium of peace, since it explains why people have different opinions. It is even a constant reminder of humility, how many of you answered false at the trollititis ? As history painfully showed with Marylin vos Savant in her demonstration, that in a tv show with “2 goats and 1 car behind 3 curtains”, you should switch your initial choice if the tv host opens an unrelated curtain to reveal a goat (which is usually to rise suspense). 10.000 angry readers, 1000 of them with ph.d, wrote that she was “dead wrong” in aggressive tone letters. Turns out, 1000 ph.d were dead stupid (looking at you Scott Smith, Charles Reid, E. Ray Bobo, Paul Erdős…) and beleived their a-priori intuition in a frequentist method, which doesn’t work. wiki link of the story. And if you find it confusing like me, this way better article by Zachary Crockett is fascinating.

Christophe’s sources are from the book “la formule du savoir” by Ngyen Hoang. Who also happens to make the youtube channel science4all. He also has a website and presents very interesting series about democracy (and Condorcet voting system) or data science.
And from his peer youtuber Julia Galef. From whom he got the inspiration for his interactive diagrams about bayesian probabilities (visual guide to bayesian thinking).

The prosecutor bias

In episode 28 Christophe gives an example of statistics misuse, by a zealous prosecutor in the Great Britain’s Sally Clarke case (1998).
It goes like this: “according to this study, SIDS (sudden infant death syndrome) occurs once in 8500; the chances that both her kids died from SIDS are of one to 73 million. If you are rational, you must be convinced of her guilt”.
And convinced was, the Jury, the judge, the media and the public. She was jailed for the murder of her two kids. Then released in 2003 and later died in 2007 from over drinking.
The reasoning was a sophism. It was false. 1/73m is the likelihood of two subsequent SIDS. This is the “probability of the 2 deaths knowing her innocence”.proba-morts-sachant-innocenceDidn’t anybody forget to evaluate the likelihood of her killing them ? If you think in a bayasian way, you will reverse the formula, and try to compute the “probability of her innocence knowing the 2 deaths”.
proba-innocence-sachant-mortsFor that, you have to know one likelihood a priori: that’s P(innocence). Which will be 1-P(guilt). The probability of guilt for a double infanticide is 1/500 millions.
proba-culpabilite-sachant-mortsNow what we must do is compare both hypothesis (guilt vs innocence), and the astronomical gap between the original “frequentist” thinking, versus the bayasian thinking is mind boggling. The actual probability of guilt was really 1/500/(1/500+1/73)=12%.
Because of this abyssal mistake, and it took 4 years to notice, then UK banned the use of statistics in courts.

more on wikipedia: Prosecutor’s fallacy ; Sally Clark

Which leads me to more statistical analysis with…

Overfitting

For this paragraph, I will refer to the channel science4all by Lê Nguyên Hoang (a mathematician with a phd in mecanism design and a published book about bayesianism).
Particularly of interest here, his series about artificial intelligence. He has 54 episodes on it, and episode 11 —in collaboration with Christophe—, talks about overfitting.

To simplify: machine learning is the field of studying databases of samples, and try to fit models that would explain them. If you want to be confident at 90% that your explanation is universal, you can’t use it before having observed 100 (unbiased) samples that confirms this explanation. The result 100, comes from the «fundamental theorem of statistical learning» which gives a sufficient condition:
theoremefondamentalvc-dim-epsi01
One of the rare English spoken works by these French youtubers is actually available around the explanation of that theorem. https://youtu.be/RkWuLtFPBKU

Here a graph directly taken from episode 13 of the AI series:
overfitting
The dashed bar is the region where just enough “explanations” are used to explain a dataset of a certain size. Less “explanations” mean under-fitting, more “explanations” means over-fitting.

More details are introduced in subsequent videos, notably the bias-variance tradeoff. That is, stick too much to the data, or stick too much to the model. Which is the subject of a whole 300 pages book by statistician and philosopher Nassim Taleb, in the black swan; with the data represented by fat Tony, and the model represented by doctor Jones.

Because there is a philosophical aspect to this, it is more than just a mathematics and statistics matter. Lê Nguyên Hoang argues that this applies to how we interpret news. Because of our cognitive biases, we are naturally enclined to perform over-fitting.

I’ll drop-in a possible illustration of my personal experience. In the newspaper japantoday during the course of the last year, they repeatedly made articles about elderly drivers causing traffic accidents. Of course after two or three articles, the internet commenters were all «what is the government doing! we need to forbid these geezers to drive!». If they had critical thinking, they’d be looking for an academical source, in sociology, to get objective statistics on elderly drivers, and they would find this graph:
elderlyfatalitiesNo there is no boom of accidents caused by elderly. They are in fact diminishing regularly. Yes it is possible to interpret these 2 counts (orange&yellow) as “the percentage of elderly incriminated in accidents rising” though. But the boom was perceived not as relative, but as absolute simply of the occurrences in the news. Occurrence in the news doesn’t demonstrate of reflect anything about the frequency of a phenomenon.

In the comments of the video, a remark I found interesting emerged. Is it not fundamentally overfitting to do a literal analysis of a text in French class? (replace French by your native language) That is, trying to find all possible meanings and hidden meanings to an essay, or a painting, even an advert ? This is posed as a question but a teacher replying, seemed interested by the possibility.

Another natural deviance attributable to overfitting is generalization. E.g. “all arabs are terrorists”. Just because your experience is from fox news doesn’t make your generalization a good model that fits the data well. Nas Daily video about that sadness.

I will conclude on overfitting by mentionning the novels Earth’s Children by Jean M Auel. In many occurence she finds ways for the heroes to find some spiritual explanation for each and every event that occurs in their lives. This is of course superstition. But superstition itself sounds a great deal just like overfitting!

For the lulz, I’ll encumber your browser’s tabs yet again, with this funny example of overfitting by xkcd: https://xkcd.com/1122/
xkcd-overfit(the idea came from this article: elitedatascience.com/overfitting-in-machine-learning)

Simpson

Edward Simpson, a statistician from Cambridge postulated the Simpson’s paradox. It is covered in the channel Science étonnante, in this video.
You have a tumor that needs treatment, there is chemotherapy or surgery. Which one do you want ? A-priori you don’t know, so asking more data, the doctor gives you a study with these numbers:
chemovssurgeryThe choice seems all made, just pick the chemotherapy ?
After you divulge your choice to your physician, he seems skeptic and presents you a different view on data because he remembers for small tumors, surgery was better:
bigsmalltumorpartitionAccording to this view, the healing chances are separated between big tumors and small tumors. So surgery is better for small tumors, but… surprise it’s better also for big tumors. What ? Surgery is always better ! That seems to contradict the original study. But looking closer both results come from the same dataset. So what’s happening ?
simpsonparadox-totalIndeed when you make the weighted total you find the same figures.
The explanation of this paradox comes from the presence of “confusion factors“. Or in Anglo-saxon litterature: a confounding factor. If you look at surgery, big tumors are numerous but they are also more difficult to treat, which lowers it’s total success percentage. That’s because «which one do you chose?» is not only dependent on people choices (which depends on the statistics), but it’s dependent on the fact that we give difficult cases to surgery more often. A confounding factor is a factor that acts on both the output of the statistics but the input too. Second example: «students who have repeated  a school year perform less than average at the baccalaureate (HS diploma). Therefore repeating school years doesn’t work». You spotted the confounding factor ? It’s because they performed bad from before, that they repeated. So the factor is performance. Third example (invented): People seem to have more libido after drinking beers. Hard to spot ? Because confounding factors can be very subtle and hard to spot indeed. Here it’s sex. Separate the data between male and female and no correlation can be observed anymore.

Motivated Numeracy

This paragraph is for completeness, and does not originates from a French source. «Motivated reasoning» is a thing, and in particular I found the following study very interesting for critical thinking: motivated numeracy and enlightened self government
This rejoins the article by Julie Beck from theatlantic that I mentioned above on certain points. Let’s show the shocking curve:
motivatednumeracyThe question asked in the lower half panel was “what happens after passing gun control laws in a city ?”. A question meant to trigger partisanship. The upper half was about skin cream treatment. In all 4 cases they present controlled data and they ask your judgment.

To me this is scary. It means that even if you are proficient with mathematics, you’re as likely as anybody else to misinterpret data to fit your views in presence of strong motivation to do so.

I hope there would be a “rank 10” in this figure, where people read this article !

Conspiracies

Finally we’ll make a footer note about conspiracies as a conclusion. Christophe says conspiracy theories, are pit of despair for the critical mind, they put thinking in a place on the “Bayesian scale” which doesn’t allow for updates. He calls that position being dogmatic. When your thinking is not longer update-able by rational mind openness, it’s because you have “divided by zero” in a way; you’ve put yourself in a 10- position.

79% of French citizens believe in at least one conspiracy theory. According to a poll on 1700 people in January 2019 by Ifop (historical polling organism). I translated you one page of the report:
conspiracies-beliefs.png
The education about critical thinking is severely lacking in the common national courses program of the education ministry.

Conclusion

If there is one meta course that should be taught before anything else, it’s critical thinking.

So pass this along, share it!

Comment it on tweeter (@zabigfrench1)

Advertisements

Principle of Maximum Opportunity

22 March 2019 – By V.Oddou

Today it’s going to be a technical, computer science engineering article about C++ practices:

The Principle of Maximum Opportunity (in C++)
let’s call it MOP !

Preface

MOP is when you mind your programming style, so that your code becomes emergently interlockable.

When we code, we have a voice in the back of our head that drives us to make choices according to some gut-felt metrics. It is a permanently on-going process that sets in motion all our history of experience, digested in the form of acquired knowledge. Sometimes unconscious when it’s pure personal experience. Sometimes conscious, when we can even put a name on these metrics. Some of these can be:

  • cyclomatic complexity (too much branching)
    • by extension, the zen of python: Special cases aren’t special enough to break the rules.
  • readability
  • bad names
    • heavy cognitive payload (SubSystemKernelTimestampCanonicalizer)
    • too generic (data, manager…)
    • devoid of entropy (a, b…)
    • unexplained magic constants
  • lenght of a line
  • number of lines of code
  • depth of indents
  • number of responsibilities
  • whatnot…

A metric bubbles up at the conscious level when a “language” is defined to reason about it.
That happens when you learn about it from scholar sources for instance.
Nicolas Boileau said “ce qui se concoit bien s’ennonce clairement” (What you understand well, you enunciate clearly).
It’s not important to not have words to put on your practices when you work alone. But in teams as soon as code reviews come up, you need to communicate your intents and that only can find its way through the keyboard if you have letters to apply to them. So let’s all read more articles like this one !

That was for metrics, but there are also principles. There are a number of enumerated “good practices” principles already. e.g: SOLID, DRY, YAGNI, KISS, POLA, code cohesion, decoupling…
They are taken for granted throughout this article.

There are design patterns (Which are frankly less important because they are naturally re-invented as their need arise). Let’s cite some for illustration: Factory, Command, Policy/Strategy, Consumer/Producer, Decorator, Adapter, Iterator, MVC…

Finally there are idioms. These are language specific, by definition of the term. NVI, CRTP, RAII, SFINAE, ADL customization points, copy-and-swap, erase-remove, pimpl, safe-bool-idiom, rule of zero…

Let’s go Lego™

Now, on to the meat for today: practices of the Principle of Maximum Opportunity (MOP).

Let’s start by introducing this principle of classical mechanics I like: isostatism versus hyperstatism.
In the English-speaking world, most often known as statically determinate versus statically indeterminate. I’ll let you google it for explanation, or duckduckgo it for anonymous explanation. But in simple words: isostatism is when you won’t have an issue assembling parts because they have leeway for manufacture imprecision.
Example: an isostatic bookshelf has 2 screw holes to prepare on the wall, and wherever they are, it will just attach with no hurdle.
Hyperstatism is when you start to have a row of 3 or more screw holes to prepare. If you miss-align your screw hole by two millimeters, then your bookshelf just won’t hook.
In one case there is just enough constraints to get the job done, in the other case, there are too many constraints. The usual tradeoff is solidity, but it’s also possible to uselessly lose degrees of freedom. In computer science, that principle best matches DRY.

auto

For those familiar with C# and resharper, this one is a no brainer since you get constant rewrites as you type code into var instead whatever type you input. But for C++ people, decades of explicit typing practices are making it a hard-to-sell. Many coding rules advocates keeping explicit typing unless the type would otherwise appear two times on the same statement.

auto client = LookupClient("john");

In this case I would argue that nobody should care what type is the client variable, as long as it is readable that it is a client. It could be a Client type, but it could be a reference wrapper to a client object for all we know; and it should not matter. In the above case, the type appears nowhere so many C++ programmers are ill-at-ease with using auto​ since the type doesn’t appear. In my experience when I’m really curious I just let the intellisense’s tooltip tell me the type, but there is always someone to complain that not everybody’s editor is visual studio.

Client client = LookupClient("john");

In that case, older C++ geezers can rest at peace since the type is stated, but what entropy have you gained ? And what is the degree of statism in that expression ? It’s 2 degrees hyperstatic, because the word client appears 3 times. That is two times too many.

auto john = LookupClient("john");

This is now isostatic since client appears once. It means this expression is now free to evolve as maintenance goes on in the program’s life, if the type changes to a reference wrapper, or if the concept of client itself changes to customer ? The function can be renamed and this expression will still be OK because the variable doesn’t re-expresses the type in the name of the symbol.

To me, auto has a bit of an unfortunate name for hysterical raisins. But it should be as natural as var in C#. But also kind-introducing declarativevar and let in Nim, Swift and typescript, let and mut in Rust. Mathematics also uses this kind-introducing notation, and has inspired numerous functional languages for variable and function declaration. There is also a visible industry-wide convergence in languages towards the typescript syntax. The Curry-Howard correspondence hints in that direction too. I think it is a force to be reckoned with. auto is such a form, and it allows to reach isostatism. Which means →it’s MOP ! You will also have noticed how DRY it is as well.

Implied interface

That is a notion used in the stdlib through names like empty, begin, end, size… and in metaprogramming with the ::value and ::type names.
See a nice article about implied interface described as a pattern: http://www.shindich.com/sources/patterns/implied.html
extract:

«Establish an interface abstraction implicitly by using functional polymorphism in the cases where the use of explicit interface constructs is either unavailable or would cause excessive refactoring of object-oriented code.»

Another one I like, making the very-needed parallels with duck-typing: https://devopedia.org/duck-typing

I’ll criticize two points of interest, quoting:

«Duck Typing is a way of programming in which an object passed into a function or method supports all method signatures and attributes expected of that object at run time. The object’s type itself is not important. Rather, the object should support all methods/attributes called on it.»

Here: the “run time” part is not necessary, I think it’s only a historical consideration to circle the concept to where and how it really happened in practice. But I think it’s fine to enlarge the definition to build-time as well. That’s where the second quote comes in:

Python doesn’t have explicit interfaces: interfaces are implied. For Python, this is not different from duck typing. However, implied interfaces are supported by C++ (for example) in its STL library. Given that C++ does compile-time checks, we can’t say in general that implied interfaces is the same as duck typing. Note that when interfaces are implied, there’s no formal interface definition, and at best, it may a be shared documentation.

While the last bit is sort of true, since the advent of boost.traits combined with BOOST_STATIC_ASSERT the formality could be raised one notch. And in C++20, concepts are allowing full formalization of implied interfaces. If you take a look at the following doc page, you’ll see what I’m talking about:
https://en.cppreference.com/w/cpp/iterator#C.2B.2B20_iterator_concepts

extract: Readable:

specifies that a type is readable by applying operator *

The presence of operator * is the implied interface, but now it’s formalized. Thanks to that concept of implied interface, we start to see that it’s possible to decouple the requirements, from the type. Without that feature, we had to rely on Interface Segregation Principle. Why ? Because it’s an overcoupling problem to pass objects which have too much responsibilities/data when you only need a sub-set of them. If you free yourself from the type and rather specify the input constraints, in the form of a formal implied interface rather than a specific type, ISP is much less necessary, and you still get build time checks; best of all worlds. Let’s look at an example:

struct Client
{
    string address;
    string name;
    Date   birth;
};

bool CanViewPorn(Client const& c)
{
    return (now() - c.birth).asyears() >= 18;
}

There is an issue with that code. Its hyperstatic degree 1 because of the over-requirement on the Client type. What if at some other place of your code, you now have:

struct Person
{
    Stuff thing;
    string name;
    Date   birth;
    int friends;
};

You could have had the opportunity of gaining Person to be a type passable to CanViewPorn, had you not over-constrained with the Client type. In this situation java purists will shout «use ISP naW!» and you’ll find yourself in this situation:

struct IBirthed
{
    virtual Date GetBirth() const = 0;
};

struct Client : IBirthed
{
    string address;
    string name;
    Date   birth;
    Date GetBirth() const override;
};

struct Person : IBirthed
{
    Stuff  thing;
    string name;
    Date   birth;
    Date   GetBirth() const override;
    int    friends;
};

bool CanViewPorn(IBirthed const& b)
{
    return (now() - b.GetBirth()).asyears() >= 18;
}

Brrr, this is dreadful. ISP in action. What a pile of smelly java crap, yes ! C++ doesn’t need this. In C++ we can «want speed ? pass by value» all over. Having interfaces forces us into the references and pointers world. Not to mention this catastrophic verbosity. And obviously, the potential runtime cost of the virtual dispatch, here utterly useless. That’s how real men do shit:

struct Client
{
    string address;
    string name;
    Date   birth;
};

struct Person
{
    Stuff  thing;
    string name;
    Date   birth;
    int    friends;
};

template<typename Birthed>
bool CanViewPorn(Birthed b)
{
    return (now() - b.birth).asyears() >= 18;
}

There ! →implied interface in all its splendor. I’d argue that it’s even less coupled in that form than the explicit interface example above. You have opportunities to pass types with a «birth» field even if the original programmer never heard of the IBirth interface. At this point maybe you’re thinking of the python has_attr function. Because maybe you’re thinking that this template accepts any type, but it will fail to build were one to pass something that is not intended to be accepted. Then you’re thinking about using an existential.

template<typename Birthed>
requires( std::is_same_v<decltype(Birthed::birth),Date> )
bool CanViewPorn(Birthed b)
{...

That, is how real men do shit… next year. But well, joke aside, this is an existential.

typequantification
A requires clause will change the template Birthed from a universal type to an existential type.

Uniformity of naming

As previously mentioned, begin(..) and end(..) are an example of names that are uniformly observed, so that types «that can bind to these functions by ADL, or called through method call», will be drop-in compatible. This is a practice observed when working with custom containers, the usual API of the closest equivalent container in stdlib is usually adopted in order to be close to drop-in replaceable. This is all the more true in C++11 now that we have ranged based for, that require begin/end to be callable. This does not extend only to stdlib related works, but also in any place in your own code that is designed to rely on implied interfaces in generic code.

Arities

When working with overloaded functions, they can be called in generic contexts only as long as the call site has a uniform syntax. That means, the same number of arguments (arity). If some functions have arity 1, and some 2, it just is not usable in generic contexts because now you have to take branches depending on the case. Overload sets are useful in that they decouple function resolution from the algorithm. The responsibility is in the compiler, to do type matching. Like auto, that precept respects isostatism.
Example:

void Div(float a, float b, float& res);
void Div(double a, double b, double defaultforzero, double& res);

template< typename Arithmetic >
void GodKnows(Arithmetic a, Arithmetic b)
{
    // stuff stuff stuff ...
    Arithmetic c;
    Div(a, b, c);
}

Function overloads can act as nice generic-exit points. If you visualize the template path like such:

Brush

That’s code call flow, on the left it starts from concrete implementations, then calls from heterogeneous types can flow through a factorized template function with generic code, but the exit nodes must flow back to concrete implementations per type supported. If you have an overload set, the template call will dispatch to the best overload. But that can only happen if your arity is consistent.
In this case Div for double will cause a problem. We might need to resort to if constexpr(std::is_same_v) to call the function with arity 4.

Consider even higher MOP: your calling code may be working with a build-time higher order function object list. Instead of calling ​Div(a, b, res); It could be:

constexpr int index = mpl::find<indices_tl, ThisOpT>::value;
constexpr auto myfuncPtr = mpl::at< functor_typelist_t, index >::value;
myfuncPtr(a, b, res);

Because all arity 2 math operations can use this code path. Be it Add, Sub, Mul or Div. Therefore, the high need of uniformity in your personal convention about arities of overload sets AND other closely related families of functions. If you fail to your conventions, you’ll force your higher order client code to branch uselessly.

Duck-typing

This is the hottest concept of this whole article, maximum opportunity is the big brother of duck-typing, in the sense that it is mostly similarly defined.
All the “guidelines” referred to hereunder are flavors of duck-typing philosophically.
I find it unfortunate that some articles about duck-typing define it as a feature of dynamic languages; and static languages should instead speak of «structural typing».

In python culture, is considered pythonic, code that assumes as less as possible about the types.
From this guide: https://docs.python-guide.org/writing/style/
bad: if attr == True: Good: if attr:
What’s the difference ? One works with only booleans, the other is canonical. You can duck-type any type that has a defined bool conversion.
In C++ that’s a lot of things. Smart pointers respecting the safe bool idiom would have this expression work, but not the first one, they would need a special == operator.
Conclusion, the second version increases the usage opportunities, that’s MOP !

ADL

In some cases, you refer to a very unique symbol by name, you know you want this one and no other. Example ::GetWindowHandle().
But, sometimes, you refer to something more vague. like sort, or swap, or size, or begin, or end
For these cases, don’t use qualified names when you can open your chances to select a wider range of more efficient specializations in the whole scope of the program, via unqualified names and ADL.
This is a duck-typing-like principle but applied to overload resolution. It works only if you respect the implied interface principle. The committee is calling them, customization points.
example:

using std::swap; // activate the fallback
b = swap(a); // ADL lookup, now boost::swap will be considered
//or
using std::hash_value;
return hash_value(arg);  // mylib::hash_value has a better specialization ?

(Note that this does not apply to std::move, unless you are trying to emulate move in C++03)
This practice became a pattern around the time when boost realized it was possible to accelerate swap of containers, and provided specializations, in their namespace. Customization points are MOP !
For more about it, and an infinitely better explanation by master Niebler, I highly recommend: http://ericniebler.com/2014/10/21/customization-point-design-in-c11-and-beyond/
Along with why it’s non-ideal, and he also goes and proposes a fix (for the standard committee to consider), a great man.

Minimal type requirements

from the shindich article above, quote:

«libraries of generic algorithms impose minimal requirements on their subjects»

«type requirements» is a notion well known to STL designers (and hopefully clients).
It spoke of concepts, before the era of concepts. Requirements such as “must be default constructible” are listed for operations or containers which are template (and therefore take universal types). So until concepts, it was necessary to document these requirements on the side of the code alone.
It is naturally good practice to reduce requirements on types to the absolute minimum when you design generic code.
Reversely, type requirements are thus a notion of generic code; meaning templates come into play. But with C++20 templates are creeping innocently into all sorts of places thanks to the auto keyword. It has been yet again extended, do you remember the generic lambda of C++14 ? Now you’ll have generic functions too. So you start to see how easy templated code is going to popup all around, without the template keyword even to be seen. This is called abbreviated function templates. (However I’m not sure whether it will be accepted.)

Template (universal types)

A template brings maximum duck-typeability, because templates are completely type enabled (that’s the opposite of type erased). In type theory that means a template type T is a universal type. So let’s call these templates, universal templates. Or unbounded templates. This is extremely powerful because it’s barely syntax checked, this is called two phase lookup, and the first phase only looks-up non-dependent symbols.
Example:

template< typename T > void func()
{
   auto stuff = T::s_stuff; // valid because anything goes. just have to instantiate with T that has s_stuff
}

concepts (existential)

Concepts are the C++20 materialization of existential types. They are a balanced middle ground between java (type erasure) and unbounded templates.
This advent is an important turning point of this decade, major languages are showing a tendency to converge in feature set.
Nim has them, D has them, Go has them, C# has where clause, Haskell of course… Languages also converge in “functional”-ization.
With lambdas and algorithms, LINQ being a nice example. Pythonic constructs.
Which naturally brings us to… ranges 🙂

ranges (Eric Niebler)

link: http://ericniebler.com/2018/12/05/standard-ranges/

Ranges are a perfect example of the minimum type requirements principle. Let me do an introduction of ranges from my personal perspective:

#include <vector>
using namespace std;

int main()
{
    // say you have a concrete list of stuff
    int vals[] = {3, 80, 10, 5, 2};
    // now you want to refactor some action on it in a function
    sort(vals);  // c.f. under
}

// how is it going to look like ?
void sort(int vals[])
{
    // .. ah. how do I know the end ?
}

// change of plan !
void sort(int vals[], size_t len)
{
    // painful API...
}

// hmmm... ?
template< size_t N >
void sort(int (&vals)[N])
{
    // nice !
}
// all happy. but next day.. a new client appears

int main()
{
    .....
    std::vector<int> vals2 = {5,6,7};
    // ouch...
    sort(vals2); // wont do
}

// an overload for this sort of collection ?
void sort(std::vector<int>& vals)
{}

// back to the MOP principles I listed above:
// this is hyperstatic level 1:
// the type is not int[N] or vector.
// it is "collection of ints"

// this is why the stdlib does it this way:
void sort(iterator begin, iterator end)
{}

// unfortunately, the iterator type is not known upfront. so it's a template
template<typename iterator>
void sort(iterator begin, iterator end)
{}

// and there you have it. that's conceptually a range.
// this has been the solution for 22 years.
// but there are problems which led to the invention of real ranges.
// I'll let you explore Niebler's site or rangev3's github.
// I'll just say the new form is this:
template <std::InputRange R>
void sort(R&& r)
{}

A range can be viewed as the new form of 2 iterators. It solves everything and is magical and unicorns actually are painting rainbows over there. It brings the functional power of C#’s lazy LINQ expressions to C++. Ever worked with jinja2 filters (pipe operator) ? ranges has them. boost offered similar kewl stuff such as iterator adaptors back in the days, which is great when you have an algorithm that works on simple collections, but you have a map and you’d like to run that algorithm on the keys. If you don’t think about those kewl libraries, you have to incur the runtime costs of creating intermediate copies of your transformed collections.
Next time, think ranges, this is MOP !

universal reference

Or whatever new fad name came up for them, what was it again ? forwarding references ? Anyway, this stuff is maxi MOP, because it allows to feed-in anything from the client point of view, and you don’t have to duplicate code for const and non const stuff. For a better feel of how MOP this is, I’ll refer you to this standard proposal:  Range-Based For-Loops: The Next Generation, by Stefan Lavavej:
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n3853.htm
Quote:

This is a proposal to add the syntax “for (elem : range)”. In addition to
being easier for novices to learn and being clearer for everyone to read,
this will encourage correctness and efficiency.

Quote 2:

"for (auto elem : range)" is very tempting and very bad. It produces
"auto elem = *__begin;", which copies each element, which is bad because:
* It might not compile – for example, unique_ptr elements aren’t copyable.
This is problematic […] for users writing generic code that’ll happily
compile until someone instantiates it for movable-only elements.
* It might misbehave at runtime […]

for (auto& elem : range) is much better, although it’s not quite perfect.
It works with const/modifiable elements, and allows them to be observed/mutated
in-place. However, it won’t compile for proxy objects

[…] and manually specifying element types is equally bad or even worse:
[…] * "for (const Elem& elem : range)" is pernicious if the Elem type is incorrectly specified. If the range is map, then "for (const pair& elem : range)" will convert-copy each element (triggering the previously-described correctness/efficiency problems) because the map’s elements are actually pair

I’ll let you read the actual document for the redacted parts, I hope you are now fully convinced that auto is great, and that auto&& is maxi great. It deserves the ultimate MOP trophy. Because of course, while this paper is about loops (did you see what I did here?), it applies for all type deduction context.

const

I have a problem with this one. It feels to me that constness goes against DRY far more often than I can care. The more senior I become the more I tend to drop from the const fad train. It’s funny how const actually is anti-opportunity. When you think of duck-typing being an opportunity enabler, const is more of a disabler. It separates accessibility of symbols into two worlds, the world of consts and the world of mutables.  If you divide, you don’t unify →less MOP.
Maybe Rust handles that better with let and mut?

interface segregation

Use function parameter types that pin point the minimum needed. That maximizes the opportunities in the future for reuse. You know about that now from the Client/Person example from above. Let’s have a second one:

void func(iterator i) { code that assumes i is non end(), or don't care about next()/++i }
// better:
void func(container::value_type& i) { /* " */ }

It’s more related to dependency than segregation, but it’s by segregating that we can decouple. Here iterator and value_type have been segregated, therefore we can pass the abstraction that is a better fit for our requirements in the body of the function. This frees the dependency to a container. We can now use the function on free elements (not contained). Also we can’t pass an invalid iterator anymore, so the invariants are unbreakable.

Maximum genericity

canonicalizeallthings
Example: you make a compiler ? arrange your data model so that a class is a class, a struct is a class, a union is a class, an interface is a class, a namespace is a class…
If you can store everything in the same bag, already written algorithms downstream will emergently work, or only require minimum maintenance.

Though classic OOP requires interfaces to express genericity, it is in contradiction with “minimum type requirements”.
Therefore, I’m generally moving away from inheritance, and using techniques above instead. Not even mentioning the problem of risk of type slicing when working with value types. Which you should always prefer to avoid rampant fragmentation and cache misses. optional can help to avoid pointers too.

variant comes in handy for the considerations of the last paragraph. If you don’t have inheritance, that doesn’t absolve you from having to work with heterogeneous types sometimes. A trick that can be pulled out thanks to variant. You can now store adjacently in memory, heterogeneous objects, as long as they are not unreasonably disparate in sizes.
Activating genericity on variants will force you into the template game. Generic visit ? You’ll need a C++14 generic lambda, or make a visitor with templated operator().
And now, the implied interface principle comes back with a vengeance, because you see how important it’s going to be to have duck-typeable types in your variant.
Otherwise you’ll be doomed to implement an operator() per type, or switch on index (or holds_alternative), or at worst if constexpr (std::is_same<...) when it’s still possible to share some template code for some of the types. Or maybe go through an «extract method» refactoring, and create an overload set of functions that will canonicalize access ? example:

def type A with implied interface A
def type B with implied interface A
def type O as an outsider
using V = variant<A, B, O>;

// requires implied interface A
template< typename T >
void extracted(T t) { do stuff A style; }

void extracted(O o) { outsider things; }

V v = init;
std::visit( [](auto t) { ... preexec; extracted(t); postexec...; }, v);
// the extracted call will duck-type to the best overload at template instantiation. (the template is the lambda)

That’s the same broom graph from earlier (X with a – in the middle), on the left, the init line, is concrete. Then we converge to generic in the middle (the visit), then re-fan-out on the right to hit the concrete leaves (extracted). And again, another echo from before, this is the “respect your arity” principle. Now one generic call to extracted can fan-out to the proper overload.

Real life example: In the making of a compiler, I had to work on a function overriding semantic check. In the implementation of the validation system, I needed to verify that the declared function is hiding a name in the base. When bases can only be interfaces you know that symbols contained in a base are always functions.
So it’s tempting at first to write a query function named :
IsFunctionHiding(identifier currentFunc, identifier baseInterface). That’s a missed opportunity. Because the implementation of that query will have nothing particular to functions. So the signature should be changed to:
IsSymbolHiding(identifier overridingCandidate, identifier base). Now, for no cost, you have opened that query for reuse by, say, properties, when they get in the language. This is not YAGNI, it’s MOP !

Another example about variant use. In the codebase I’m working on we have a type using ConstNumericVal = variant; that allows to keep the information about the type while being able to store in a unified way (canonical) a parsed value of the activated type. Not having that facility would require boxing, and creating a polymorphic class hierarchy a la java. Which means pointers, and lots of boilerplate lines that never translate to any machine OPCode.

`Get` and `Has`

This pattern happens often. I noted that to avoid repeating yourself, Has ends up usually implemented as return Get() != end();
That’s a problem if you usually do in client side: if (Has()) auto elem = Get(); because now the query runs twice.
Today I prefer to get rid of Has entirely, and return an optional instead. This way you get the 2 functionalities in one function.
Let’s take an example in the stdlib: map‘s find() method. They applied the same principle, there is no equivalent to C# HasKey(). You find and check against end() yourself.
However that is imperfect, because of the now introduced dependency on the underlying collection. It was assumed that «since you could call find() on your collection, thus you have said collection». That becomes more nuanced when you expose iterators through your class API, but not the collection. There is no self awareness of invalidity in iterators, you can’t ask myiterator.isend() because an iterator can be a raw pointer; by principle of duck-typability, the C++ committee respected that an iterator implied interface is no more than the one of a raw pointer, and for good reason. So using optional here, decorates the iterator and gives it the ability to know its own validity, all problems nicely solved.

Anti canonical practices

#if platform
This is one of the worst offenders because of its pernicious presence in practice. This effectively multiplies your cyclomatic complexity with little mandate to do so. (meaning now you have 2*N tests instead of N tests to run)
It can be alleviated by leveraging libraries, rely on canonicalizing API like QT, boost.filesystem and whatnot, to avoid shouldering the cost of that cyclomatic complexity multiplication.
In general anything with a if is a smell to me. Especially introducing if checks for error cases. Error cases are not important enough to bloat your code. That’s by definition of the Zen of python. There are many discussions, still today, about Tony Hoare’s billion dollar mistake, about monadic error handling, about exceptions, or the std::expected proposal. All happenning because they recognize this same truth: that defense creeping in code is a bad-thing™.
Sometimes error cases are actual normal path of the business requirement; like in compilers, anything ending with a diagnostic, is mandated by specification; has a test, and deserves a if. But you see, there is a difference between a legitimate if that’s justifiable because it covers a mandated case in the original problem’s state machine; and an illegitimate if that’s just there because a particular implementation failed to canonicalize, or streamline errors.
Defensive programming is justifiable only when it is hidden at the level of expressiveness of interest. That means e.g, in C++ when you access an index through operator[], the defense is hidden in the implementation of that operator. Therefore you don’t have to cover your access with if (index = len) return ERROR; Because the operator will assert deterministically internally, so it’s guaranteed you’ll catch the problem at QA; all-the-while reaping the benefit of pure code at the call site level. This is true of null access too by the way. You can branch, only when accessing-out-of-bounds has a deserved fallback (e.g: is within the invariants of your program, or the function’s predicate in aspect oriented programming).

Uniform initialization

You know, the stuff that replaces constructor call with brackets. It is also part of MOP since it syntactically enables duck typing of construction calls for any kind of objects. It matters in templates.

JSON example

The wrong way to do it:
Nim:

import json
let jsonNode = parseJson("""{"key": 3.14}""")
doAssert jsonNode["key"].getFloat() == 3.14

Because now, what do you do when you have a slightly more complex access path, you want to factorize it, but potentially you have multiple types as clients.

float MyHelperGetKeyFloat(midpath)
{
jsonNode = ... obtain node
return jsonNode["stuff/nested/" + midpath + "/key"].getFloat();
}

double MyHelperGetKeyDouble(midpath)
{
jsonNode = ... obtain node
return jsonNode["stuff/nested/" + midpath + "/key"].getDouble();
}

And this non-genericity bubbles up to the highest client. In that situation, you’re now forced to return a json node to respect DRY. And now you leak an abstraction; and create un-necessary coupling. It now looks like jsonNode should use Interface Segregation Principle, and have a .getAny() which returns a variant type. But it’s a library, so no control over it.

The correct way:
https://github.com/nlohmann/json

json j;
// add a number that is stored as double
j["pi"] = 3.141;
// add a Boolean that is stored as bool
j["happy"] = true;

The channel to communicate type information has shifted from naming, to the type system. I don’t know about you, but to me, it sounds a bit better than a typing system should handle type stuff, and names should handle naming stuff. Thanks to that superior API, you are now enabled to work with factorized generic code.

This applies in the graphic world, for shader constant API too.
1:

SetFloat(f);
SetMatrix(m);

?
wrong.
2:

Set<Float>(f);
Set<Matrix>(m);

?
better, but over-constrained (hyperstatic). This can be useful when you actually desire this constraint, like a mechanical structure, it can be on purpose to increase solidity. But using the naming channel (example 1) for that purpose is a misguided, imposed, loss of opportunity for clients.
When useful (duck typing situations), find isostatism by leveraging type deduction:

Set(f);
Set(m);

Exactly as nlohmann’s Json is doing. The type information flows through the channel of the typing system, not the naming.

pushing it even further with yet inexistant language features

Uniform Function Call Syntax

With UFCS you can enhance this whole MOP rant to a new level. Nim has it and culturally the community makes extensive use of it. D has it. And Rust has a lesser version of it too.
This: "string".slice(2) is the same as slice("string", 2) and the very Nim-ic: 10.times (check it more in depth at https://hookrace.net/blog/what-is-special-about-nim/)
That syntax flexibility is paramount in a world with macros (lisp macros, not defines), or universal templates (like C++ has) to activate MOP.
Had we had UFCS, it might have been unnecessary to introduce a canonicalizer such as std::invoke.

Last word example

Let’s take a Yaml serializer, say you have:

struct S { string name; };
std::vector<S> m_esses;

and you want to serialize it to yaml. It’s natural to write a function:

string ToYaml(std::vector<S>& esses)
{
   return "{" + join(esses.begin(), esses.end(), [](auto&& elem) { elem.name }, ", ") + "}"; // whatever
}

Mistake. Why restrain to vectors ? your clients could have list, that would work the same for that implementation, and yet be forced into implementing a new version of that function ? Or they could want to yaml-ify only have of their vector. They’d have to copy that half. We saw it in the previous paragraph, the correct API here is ranges. We will assume a range is a pair of iterators, as chosen in stdlib algorithms pre C++20.

So it becomes:

template< typename IterType>
string ToYaml(IterType begin, IterType end)
{
  return "{" + join(begin, end, [](auto&& elem) { elem.name }, ", ") + "}";
}

Awesome. now you have duck-typability, because any collection that join supports, will work. Your ToYaml respects MOP (it forwards the limitations to join). But now  elem.name worked only on type S, and suddenly you have a struct T that you want to serialize too.

struct T { int id; };

template< typename IterType>
string ToYaml(IterType begin, IterType end)
{
  return "{" + join(begin, end, [](auto&& elem) { std::to_string(elem.id) }, ", ") + "}";
}

great but now you have a compile error because of redifinition. So…

template< typename IterType>
string SToYaml(IterType begin, IterType end)
{//... impl as before
}
template< typename IterType>
string TToYaml(IterType begin, IterType end)
{//... impl as before
}

?
NO ! that is not MOP compliant. Because now your implied interface is heterogeneous. which means your clients have to know what they are working on to chose the good function.
If they were methods of classes S and T that would be an equally problematic choice, because we don’t have UFCS, therefore the instance.ToYaml() syntax would break MOP as well. Who said you’re serializing things that have this method ? Now stuff that has free functions (e.g aggregates, fundamentals…) should be called by ToYaml(thing); and User Defined Types that can do it by themselves should do stuff.ToYaml() ? no ! it’s not canonical → not MOP. The solution, is to use SFINAE to split the definitions.

template< typename IterType, CONSTRAINT_HERE >
string ToYaml(IterType begin, IterType end)
{//... impl as before
}
template< typename IterType, CONSTRAINT_HERE >
string ToYaml(IterType begin, IterType end)
{//... impl as before
}

So what is the constraint ? it’s IterType::value_type is_same as S for the first case, and is_same as T for the second case. full expression:

#define SFINAE_IS_SAME(DependentTypeToCompare, ExpectedType)\
    enable_if_t< is_same_v<typename DependentTypeToCompare, ExpectedType> >* = nullptr

template <typename IterType, SFINAE_IS_SAME(IterType::value_type, S) >
string ToYaml(IterType begin, IterType end)
{ ... }

template <typename IterType, SFINAE_IS_SAME(IterType::value_type, T) >
string ToYaml(IterType begin, IterType end)
{ ... }

Now you’re good. you have a unified implied interface → MOP !

This is it

The reason why I chose the term opportunity, is because instead of designing ad-hoc for the problem at hand and nothing more, you are designing for 2 things, the problem at hand AND opening opportunities for other parts, or future parts of the system. When you make a function take a range and not a particular collection, you are opening opportunities for all sorts of clients.
This is yet another lego intervention; where imbrication (interoperability) is put on a pedestal, but at the level of the syntax and the typing system.
All these little practices together help me respect the “principle of maximum opportunity“. I found that applying them, together with other practices, like functional paradigm and reasoning about invariants (and all the stuff listed at the top of the article), plus a tiny bit of over-engineering at the corners, served me with happy surprises down the road of programming projects. Like having a long standing and forgotten TODO be naturally fixed by a refactoring that closed the completeness of a system. Or having enemy planted defense turrets, shoot at you out of the box even though you didn’t program that behavior explicitly (in my little game extreme carnage).
It enables unforeseen compatibility between parts of code you hadn’t consciously designed to work together. This really ups the mood when a requirement arises, and you realize you’ve already implemented it without knowing (or almost).
Another example of emerging feature happened when I was designing the separation of concerns between the main scene edition software, and real time rendering of viewports, in e-on software’s Vue suite (from Vue7). This is an equivalent refactoring as to apply MVC pattern. After weeks of pushing that paradigm, passing messages, decoupling life-times of objects, activating safe separation of threads… it got to a point where the rendering loop got so decoupled from the interface that it was free to get as late as heap would allow. And the visual response got laggish and delayed. In this case it was an undesired effect, but it’s a miracle to get such a separation to work at this level at all, while never putting any conscious design toward that end. Of course when it happened it was easily tamed with a condition variable to resynchronize the two; but the point to take here, is it’s better happening this way than the other (desiring it but having to work to get it).
Here is a great article that mentions the importance of MOP, in the development of Age Of Empires Definitive Edition: http://richg42.blogspot.com/2018/02/some-lessons-learned-while-developing.html
He doesn’t mention MOP, but that’s what he is talking about without having the language to put a word on it.
And yet, I noticed that elephants can’t cross shallow waters, while crocodile can. But this feels arbitrary and definitely smells of a if (entity.kind == crocodile) somewhere in the code: an anti canonical behavior. Or worse even, separated collections ?
Finally personally, the principle of maximum opportunity is a big part of the fun of doing this job. Because when you are rewarded with features that come from emergent behavior, it feels like Christmas.

Cryptocurrencies, not there yet.

Hello dear reader,

Today I wish to share with you my take on why cryptocurrencies are not ready.
This will sound like a rant, that’s because it is one.

Technology

Blockchain

First rant, the blockchain itself. You’ll hear a lot in the media that the opponents to cryptocurrencies have “doubts about virtual coins as legal tender” but are convinced “the blockchain is a great invention and here to stay”. That is a recurrent trope in the field.

The blockchain is not a great invention, it’s close to nothing: it is a linked list of Merkle trees.
Otherwise said, it is a combination of 2 basic programming concepts, one invented in 1955 and the other in 1979.

A normal programmer needs to “invent” or at least deal with, equivalent schemes on a daily basis. In mechanics, you could compare it to a “hinge”, that’s made of bronze rings in a steel stator, and standard screws. It is a simple element of a larger system. I don’t criticize simple, there is beauty in simplicity, and also I’m a proponent of “worse is better“. I’m just saying there is nothing marvellous or life changing just from the advent of the blockchain. Let’s not make it sound regularly like if it was a new god.

It was just waiting for someone to find a great application for it, and even a negative criticizing person like me, acknowledge that it has a great potential; more particularly in solving the electronic vote problem with safety.

Network of transaction verifiers

Serving as a ledger for transaction of money (or debt) is a great application indeed.
For cryptocurrencies, let us define some terms, the blocks in the chain, contain a bag of transactions (txs). In bitcoin they are “mined” and one is created every 10 minutes.
The genius of Satoshi Nakamoto however, lies not in the invention of this blockchain, -which is a simple piece of engineering as we saw-,
but rather lies in the invention of the Proof of Work: PoW.
It is a game between participants of the network that requires you to spend energy in the construction of new blocks. The biggest spender is awarded by the others, the right to include his work (the mined block) in the blockchain. This way, cheating is tremendously hard, because you’d have to spend way more energy that the others so that you can vote yourself the winner for a block. If you managed that, you could rewrite transactions and make yourself rich. Though, because the system requires already as much energy as you can afford for normal operation, cheating operations are infeasible.

The principal value we get from this is the possibility to create a network that doesn’t need trust of a central entity any more. You don’t have to nominate a judge or, a key keeper, to make sure nobody cheats your transaction inside a network.

That is a significant added value proposal, consider paypal: when an automatic bot flags your account as suspicious, your money is frozen. And you have to spend hours of misery with support tickets. With a decentralized network, the power is in your hands. That matters. I want it. And that, my friends, is the bitcoin dream.

Flaws

Unfortunately, this game engenders a series of flaws:

  • The more participants (miners) join the network, the more energy the system ask to be spent. It self regulates to be hard enough that a block takes 10 minutes to mine, whatever the evolution of the hardware, or how many miners rejoin (or leave). As a result bitcoin’s total network consumption amounts to a full nuclear power-plant today. (4 reactors of 1GW each)
    • mitigation: Don’t listen to idiots who claim that it will reach as much as a whole first world country’s consumption by 2020, that’s plain bullcrap. It is a purely economical game, the more miners join the party the less profitable it is for all of them. It regulates itself.
  • Only 7 transactions per second can be treated, because there is a limit of 1MB on the size of blocks. 1MB means about 4000 txs data can fit. This has increased to about 1.8MB (virtually) with the advent of segwit.
  • Because there is a market of fees, miners only accept transactions rewarding them the best fees. As a result, moving any amount of money requires you to hold at least the necessary fee, which means if you have 1$ left but tx fees are 5$, you just can’t, this is called dust amount. This is justified to avoid denial of service attacks.
  • Absolute transactions. They can’t be rolled back ever. If you send by mistake, you’re out of luck.

I believe those flaws are show stoppers. Because with that system, I still prefer to use banks. The tx fees are equivalent to banks, and that was my main reason to be infuriated against banks in the first place. I thought electronic transfers were costless, and they really are, though we get billed insane amounts just to move money, which should be free.
What I want:

  1. No judgement (oppressive tax over-watch agency; anti laundering invasive questions; account frozen because a regex don’t like your face)  – BTC pass
  2. No fees (fuck you banksters and paypal with your 4%, it’s too easy to rob people) – BTC fail
  3. No selling yourself (credit cards offer no tx fees because they can watch what you buy, which is data, that they sell to advertisers or other) – BTC pass
  4. No privacy risks (government banning pot ? or sex dolls ? or subscription to the underground Iranian revolt tribune, whatever your problem, you don’t want the nose of prude protestants in your business) – BTC pass
  5. Instant txs (on internet, I can get way more data than what a money transaction takes, in one click. The google logo at every search we do, is more costly to transfer than money. By a hundred.) – BTC fail
  6. Don’t ruin the planet uselessly. Don’t leak holder’s money from the pipes. – BTC fail

And I say “pass”, but it doesn’t, Monero is stronger at points 1-3-4. So I’d prefer to advocate Monero over BTC at any time.
To fix point 6, Proof of Stake (PoS) exists. Some coins like Decred want to remove the requirement of energy spending in the form of CPU mining, to a form of “heavy amounts of money immobilization” which gets dividends if you play by the rules, and gets nothing in return if you try to cheat.

Exchanges

Unfortunately, acquiring the tokens today, needs compatibility with the old wealth holding system: fiat currency. (Yen, Dollars, Euro..)
For this, exchanges exist. Bitfinex, Binance, Coinbase, Bitstamp, Bitflyer, BTCBox, Kraken, coincheck etc.

That leads me to another big rant: the exchanges are raided by governments, and now KYC measures are oppressing cryptocurrency holders around the world with the same kind of measures that have been put in airports due to terrorist fears. Tracking us, filing us like criminals, with identity registration, killing totally the anonymity we could otherwise have had. The tax agencies are also monitoring them to steal more from people, without even giving us a chance of getting compensated in case of trade loss like usual stocks benefit from. (looking at you Japan)

There seems to be hope in decentralized exchanges (dex), but for the moment I only know that it can work between cryptocurrencies, so fiat pairs needs to used pegged tokens like USDT which recentralizes the system ultimately.

Coming back to my whishlist, it exists a new buzzword in the field, tokens that don’t employ a simple blockchain, that could alleviate the tx fees and the speed issue. Those coins are called DAG based.
There are 3 of those today, as per my knowledge:

  • IOTA/mIOTA
  • Byteball
  • RaiBlock a.k.a nano

This is a great beacon of hope for me, thanks to the fact that all checks would be in the green in my wishlist.
WOULD, because IOTA programmers are incompetent idiots who create their own hash functions, whose network is a catastrophe where nobody can transact anything but if they contact some guy in a discord channel at the end of the world, to ask him to manually force their transaction on nodes that listen to him.
And that’s unfortunately a pattern we tend to notice too much in this field. It has been around for 10 years and I still have to be convinced of any serious progress. Bitcoin cash engineers modified 20 lines in the original bitcoin source, and still managed to get it wrong 2 times (making the early launches fall flat on their faces) before the real launch.

I believe the most in nano, it has a competent team and the most believable tech. It could even be bitcoin2. In fact, it’s the only currency I support. Check a video about it.
And a success story for it’s usage at an audiophile shop: https://www.reddit.com/r/CryptoCurrency/comments/7x6go5/vendor_accepting_nano_as_payment_over_the_last/

Human catastrophe

The field is .. not a field, it’s a minefield. Everybody here wants to steal your money.
You get scratch scams for your ledger, malware in wallets, hackers stealing your accounts, North Korea getting enough budget to run their army for a year from coincheck’s heist. (NK claim is is not sourced because it is my speculation only).
Cryptocurrency legit percentage usage volume just passed over illegal users only last year.
The DAO hack was quite something, Mt Gox one too.
AlphaBay, Silk road (in this affair, even the cops turned robber). Shall I continue ?
ICOs are scams 70% of the time (made up number, take with salt), why ? Because it’s too easy. Take a look: the useless token. This guy raised 100k USD.
Ripple, is even a case where the crypto token itself is a scam. They attributed themselves 90% of the whole monetary base. And people buying it today are trading the breadcrumbs between themselves like idiots.

I personally lost money from Nicehash hack and Bitgrail’s hack. The principal designer of Nicehash was a hacker who created a botnet and did years of prison because of it.

Sometimes I wonder if we should not stop cryptocurrencies altogether, just to give humanity a moment to cool down. Why so much greed ?

Usability flaws

Let’s talk concrete now. How does this all go in practice ?

Not so well. First there is the pizza problem. Cryptocurrencies were supposed to be money 2, but it costs so much to transfer that it only makes sense for saving amounts. So people have started re-branding them to “new gold”. Instead. That’s quite a downgrade of ambition.
The other problem, is the software we have to use. It’s hard to use a cryptocurrency today. I’m an engineer, and it takes me hours to move money around. As well as beads of sweat when you click transfer buttons, because there is no safety net.

Let’s take the example of Monero. I want to withdraw some XMR I bought from Kraken so that I can take the risk of custody.
Exchanges concentrate custody of funds of many people. Bitfinex has an account with more than a billion dollars worth of BTC on it. That attracts the bees.
Therefore, one should not leave funds on exchanges, traders have no choice that’s unfortunate, but holders must take their coins back locally, and bear the burden of securing their wallets.

Securing your wallet is a job. You need to be good at computer security to understand all the implications. I have an academic formation, a graduate degree of ENSEIRB, (that’s EIGSI in this image). I know about cryptography, I know about computer science. And yet, it’s still hard like hell to perform any operation.

The first thing you do, is google “how to do xx” You find a website that you have to trust, because who says it’s official ? A cryptocurrency is supposed to be decentralized, so there can’t be an “organisation” that owns official stuff, your neighbour is as official as the website. Though in practice it’s rarely the case, Monero has a portal called https://getmonero.org/. So that’s the first bug in the system, a trustless system that you have to trust. Hah, good job folks. So you go on reddit and check that people mention this link to get a sense of trust. Ok done, let’s proceed.

You get the wallet, and then it syncs the blockchain, and it shows an estimation of 30 days for syncing. Yes ladies and gentlemen, it was already bad enough in the network to need 6 confirmations (1 hour) before someone accepts your transaction, before you can spend anything you need a full month of software setup.

I wanted to accelerate that, because it seems you can: https://www.monero.how/tutorial-how-to-speed-up-initial-blockchain-sync
So you follow step by step, and it doesn’t go like the steps at all.

First step, is download the chain. Ah but wait, this will take 30 gigas ! Hem.. not so easy.
I can’t let my laptop on so much. So I opened an Amazon AWS virtual machine (EC2) to download far from my oversight. Except that the free instance of the amazon cloud virtual machines can have only 30 gigas of storage max. But you also have the system installed there, so free, maybe 25 gigs. I’m fucked ? Maybe, I piped curl (because wget doesn’t output on the stdout) into 7zip to compress on the fly of the download. In the end I got a 8Gig blockchain. Probably a mid-download network bug that cut my file, but I don’t know.

So then you try to use the import command, it fails in various ways until you realize that it can’t run while the deamon is running. Or that providing the file without –input-file will not be accepted. Well.. ok.

Then it imports. And it’s still importing 4 days later as I write this article. Because it is THAT slow. It’s been days and I still can’t make a single transaction.

Edit (11/2/18): The import terminated with lots of errors, but successfully made my blockheight progress to a million. Fortunately the db is not corrupted and I can now continue sync through the network.

VirtualBox_myLuBox_11_02_2018_15_50_23

I had to fight virtual box because the guest addition didn’t work in Lubuntu. Because of course, it’s recommended to use virtual machines, because of the malware risk.
Therefore to share stuff (between host and guest) I had to install an apache server on my host after realizing that IIS manager (inetmgr) didn’t get installed whatever option I chose in the “programs & features” of Windows. Then you realize that there is no more binary apache build and you need to get a “fanbuild”, and then the one you use doesn’t work because it requires visual studio runtime libraries, and then it’s slow and someone tells you it’s the IO cache that is off by default…

lightbulb

Edit2 (13/2/18): After using monderod (deamon) for 3 days, I was at 1.3 million blocks when today after rebooting, it ends up in a bus error:

moneroFail

One google search later: https://monero.stackexchange.com/a/4317

I could persist and try to restore one of the snapshots but It was not a full copy of the disk, so that won’t work. I could try to wait when I get access back to my desktop machine with a SSD (temporary move in another apartment now, so I only have my laptop), but 30 GiB is too much for the little free space we have in SSD machines.

In the end, even a computer scientist like me couldn’t yet move any coin, in a week of work (free time after putting daughter to sleep).

The only viable option at this point, would be to use mymonero (a third party who did this work) and pay them extortion fees to move your coins. No way.

I hereby declare Monero to be unusable, and I will sell my holdings. It’s fortunate that I only decided to move a dust amount to a paper wallet first, otherwise that would have been burnt coins.

I have success stories with decred and BTC but everytime, it takes hours of synchronization and painful fees, long waiting for confirmations, and putting up with exchanges crap. High withdraw fees, scandalous artificial market spreads (bitflyer applies 20% of spread on ALTs on spot market), daily limits, KYC crap, and that’s when they don’t exit scam.

Socrates moment

There were some words you didn’t understand in this paragraph ? If yes, that’s a massive philosophical problem. You see, cryptocurrencies claim to be trustless. Which means that the network alone is secure by itself. No need to trust an agency. Say a bank, today you trust that your bank will not run with your deposits. That’s the best you can do. With bitcoin, there is no such need, your deposit is only accessible to you, even when in custody of the network.
But to be sure of that, you need to understand enough about cryptography, computer science and mathematics (Chinese remainder theorem) to convince yourself that the system is indeed sound. If you don’t, then you need to trust somebody who says so (or form yourself). That’s a huge bug in the claim in my opinion. A system can’t be trustless when it’s not verifiable by most users.

Counter word

I want to finish with a final word of anti-FUD for my readers, bitcoin is not a scam, like Dimon from JPMorgan claimed, it is not a bad asset class, like W.Buffet claimed, and it is not a Ponzi scheme like some people like to claim as well.
Yes there are lots of scams going on everywhere (c.f. human catastrophe paragraph), but bitcoin itself is not a scam. I’d even consider it one of the best libre coins in that it is developed by the community and not one agency.
It is not a bad asset class, it’s just a very peculiar asset class in that it is very volatile and roller coaster-ish. When you get in, be ready for the ride that’s all.
And Ponzi scheme, the claim is that the value is only driven by people getting in, so while more people buy than people sell, the market rises. But it will collapse when there is no more people to get in. If that’s enough to make a ponzi scheme, then all of Wall street is just one big Ponzi scheme. No, this is just a normal market behaviour. It would be a Ponzi scheme, when there are money flows between holders, and referrers who introduce new recruits, that will themselves become referrers, and selling coins would flow the cash to the top of the pyramid. There is no such construction in bitcoin. That’s defamatory to make the claim than cryptocurrencies are Ponzi schemes. Please ignore those naysayers.

It is not a worthless piece of ones and zeroes. We hear this one a lot: it’s “virtual” therefore it has no value. Because it is electronic, it is worthless ? Total idiocy, fiat money is the same today, and worse. Fiat is worse because it can be spawned out of thin air during loan extension thanks to fractional reserve system. Fiat is worse because there is no more an underlying legal tender like in the past. It totally bites its own tail, the reserve is fiat itself nowadays. It’s a joke. Fiat is stored and transferred electrically as well, the numbers in a bank account are simply a record of the amount of debt the bank is obligated to you. In that sense, cryptocurrencies hold even more intrinsic value than fiat. Don’t let yourself be fooled by old geezers who shout “scam” everywhere without knowing even the beginning of the story.
I’ll add even more in this (because my last argument was whataboutism); txs on a ledger are just terms of a contract. A contract about a transfer of debt. What is different from a paper written contract and an electronic contract ? The medium. But can you say “I don’t respect this obligation because it’s written in a computer”. Go say that to a judge I doubt he’ll agree. When you click “I agree” on amazon, you think they can’t sue you at all if you infringe the contract ? Obviously, what matters is the content, not the medium. So virtual currencies have at least the same value than fiat. And obviously more, because you can’t counterfeit a bitcoin, but you can try and make a fake banknote, many Mafiosi certainly didn’t mind.

Heists

When an exchange is hacked and money stolen, it means nothing about cryptocurrencies, it doesn’t reveal a fundamental issues with them. In the case of coincheck and NEM, the NEM blockchain had no security issue at all, it’s only the private key of coincheck that has been found. It’s the same as having a robber getting in your house and steal some cash in your wallet. Is there a problem fundamentally with the banknotes ? No the problem was the security of your wallet, maybe the door lock had a flaw. Same here.
Banks had the same amount of problem in the past. Exchanges are banks and they get robbed. History have allowed banks to evolve into places that don’t hold cash, that has silent alarm buttons under desks, or traceable banknotes with paint bomb that explodes to mark them when a heist is detected. And the bills are under warranty against theft. Crypto is simply the wild wild west, and with time cryptobanks will develop good security measures that will be on par with current banks.
Decentralized exchanges will also help, as well as if fiat disappears eventually, there will be no need for central gateways anymore.

Revolution

Lots of people want to crush the crypto revolution because it threatens their reign. This is the case of the European Central Bank, and Europe’s agency for credit main executives who are spitting venom right now because they lived all their lives believing very strongly in a centralized system in which supervision provided the service of risk mitigation for the economy, by fine tuning interest rates (to prevent bubbles), reserve requirements, minting so that the monetary base follow the added value etc… They believe that this is important. And I agree, but that’s not what they are saying, they are saying “leave this power in our hands, we are your masters and we only should have the keys of money”. That’s wrong, all these services can be given back to the people through the means of cryptocurrency. Not bitcoin, bitcoin is incomplete, inefficient, and needs to die. But a system more evolved, bitcoin2 or whatever, can provide all of this and my checkboxes (wishlist). That’d be awesome.

Virtual currencies, or back to the future.

Hi there, today’s topic: the bitcoin versus capitalism.
Did you know ? A whole bunch of people believe that good old fiat money like we know it, might see its reign over the world reduced to only a share of the reign, or ideally, be scraped altogether. Those people are the early adopters of crypto-currencies.

It’s not just some crazy doctors’s experiment
emett

You don’t just put 80 billion dollars into an experiment (that’s more than the USA ministry of education’s yearly budget).
No it’s real and here to attempt some amount of change. And not just an AirBnB/uber-ish kind of change, rather a turn-the-continents-over-their-heads kind of change. Because it could very well sound the death knell of uncle Sam’s buck. And auntie Merkel’s Euro. And the rest.
Buuuuut I’m not here to speak of this. Rather, I’d serve you with some orthogonal ideas.

Indeed I postulate that a virtual crypto coin, say bitcoin (call it BTC), holds more hard value than fiat money we own today.
What’s that ? How can a bunch of bits have more “hard value” than the metal coins I have in my wallet ?
Well that’s because there is no amount of debt, that composes a BTC. One bitcoin, is exactly what its worth, just like a coin of gold is worth its own intrinsic value as a precious metal. However, the zinc and copper coins we use as cash to trade a fiat currency-like dollars, have no value per se, they represent a “IOU” (promissory note). The bank acknowledges it owes you 2 dollars when you possess a coin with written “2” on it. But the coin itself, is not 2 dollars. So what is ? Until first half of last century, real dollars’s value was kept as gold. So a bank needed to have as much gold piles in reserve as they issued coins and notes/bills.
That started to be a problem when the population of earth started to count in billion. And a little problem called world-war-2-in-need-for-big-bucks as well.
Behold, the fractional reserve banking system.
Now, by law, banks are authorized to create money out of thin air when, and only when, a loan is extended. I’ll repeat that slowly, getting money from a bank institution when financing something means new money is created out of nothing.
And because today, lots of money in circulation are deposits made by recipients of payments made by borrowers, the hard value of a dollar has reduced to about a cent (no joke). This means, that 99% of the money currently flowing this earth, is devoid of value, since it’s pure magic.

Well, this does not happen in BTC. Or not nearly as much. It is impossible to create new coins other than spawning a new block, that is valid, has a high proof of work, and is finished before other competitors. Knowing the difficulty is adjusted so that it takes 10 minutes to compute, and about 12.5 BTC are distributed to the contributors of the computation. This value of 12.5 halves every 4 years, and integrating over this decreasing exponential gives us the number of 21 million.
21 million is the maximum number of bitcoin that will ever exist.
This means multiple things:

  • BTC cannot create debt money
  • BTC is deflationary
  • Fractional reserve is inapplicable

So no fractional lending means that financing institutions will have to actually own the money they give out. Which will reduce the liquidity of the whole society. What do say I ? Reduce not it will, divide by 100 rather; since the reserve percentage imposed by law, has been lowered decade after decade, to a single digit value today.

That poses a serious problem to a system that Americans, I believe, tend to like a lot (Europeans too but they don’t want to admit it), and that is nothing less than capitalism itself.

Which I will not discuss here, but rather I will argue that mister crypto coin, announces a retrogression of the economic system.
Wait what ? How a super cutting edge technology full of jigowatts could be a retrogression ? And what the heck is this word anyway, retrogression !
Nevermind the word, but yes by my vision of things, the bitcoin and other cryptocurrencies are simply a suggestion to go back to before the abandon of the gold standard in 1933 by Roosevelt.

A friend once told me during discussions at the bar following the 2008 crisis, “without loans, the society is frozen”. I was a bit shocked, befuddled, taken aback.
It took me years to come to terms with this short string of words “without loans, the society is frozen”… I have thought about it from all angles during countless hours, but I finally came to conclusion that it has got to be true.
Indeed, having free loans means that we are compressing time. You see, time is money, and that’s not just a saying. We literally (mathematically) integrate effort over time to generate money. This is due to work. We work to create something, this has an added value, and that’s quantifiable with a price tag. Someone giving you money for what you built, is just trading time he spent to do some equivalent effort.
I often think of it this way, buying an apartment 200k means that it would take me 25 years to build it by myself. Using a bank loan, I can profit right now of 25 years of efforts. That’s very Einseiny, hypercubal even ! We are playing quantum physicists.

If loans were not free, it means there is no time compression, loans extended would amount to exactly their value of work, work that has been done in the past; not in the future. So it would become way harder to get one, because of lack of supply. And should you get one, the interests should mathematically be a hundred times higher than today, since we are effectively going from 1% of reserve to 100% of reserve.

In that way, a crypto-currency is a come back to 1933.
1933LA
Nice 🙂

Well, having money that represents hard value, and is not generated out of thin air, has an air of seduction to it, doesn’t it ?
Surely not to everyone. Big names of the capitalist game, like Buffet, Musk, Morgan, Tillerson and co, they didn’t become that rich just with money they’ve made and re-invested. The “invest & compose returns” snowball effect, is multipliable by a wonderful thing called “leverage”. And that is accessible only when you are lucky (through consumer sales), or use a loan. There are thousands of big sharks up there who will not sit and see their opportunities of leverage go down the sink. Governments first. They are the first ones who spend like idiots and thus needs tomorrow’s time, right now. What’s it gonna be with BTC ?

Deflation

This is the second angle of the consequences. A finite money supply means that if you consider this:
unity price = supply / amount of stuff in economy
When supply is a constant, and stuff increase, then price must decrease. That’s why BTC is deflationary by nature. Either we cool down the economy (and slow down the huge amount of waste we are doing anyway), either the prices will go down. It’s not a problem because it means that one BTC is going to be worth more. Deflation gives value to the money. That’s nice because people’s savings and deposits gain in value, for free. But that’t not nice because it becomes better to sit on cash than to bring it out to invest in stuff, because while it’s out there, the investment may bring less return than just sitting on it.
That’s another challenge that an intrinsically deflationary currency like BTC is facing.

I hope I’ve given you food for thoughts. be well !

Racial discrimination is ridiculous

Why is discrimination wrong ?

I think in the past decade, we are witnessing a resurgence of racial hatred around the world.

Greeks are lazy
brexit
terrorists
anti-japanese sentiments among chinese and koreans
anti-chinese sentiments in the world (c.f. modern purloin of south china sea islands)
anti-russian (c.f. crimea annexation)
anti-syrian migrants
Austria’s extreme right winning elections

Is this all just a feeling ?
Possibly. A cognitive bias called exposition bias, enabled by media over-coverage.
Unfortunately this could be the same issue than the self-fullfiling prohecy:
http://study.com/academy/lesson/self-fulfilling-prophecies-in-psychology-definition-examples.html
(A self-fulfilling prophecy is when a person unknowingly causes a prediction to come true, due to the simple fact that he or she expects it to come true)
Applied in this case I mean that the society is self-reinforcing hatred by propagating bullshit.

This has a tendency to easily get traction among mobs, why ? that’s another psychological effect covered in the book 1984.
If the internal society is unwell, or somehow ill-at-ease, the discomfort will have a propensity to make people look for culprit, re-inforce group cohesion, and point outside.
In 1984 it is the virtual enemy: EurAsia.
Today’s government are using this effect, point and accuse the outsider of being responsible for all internal issues.
Europe is responsible for Britain discomfort.
USA is responsible for North Koreans discomfort.
Japan is responsible for China’s suffering.
Terrorists are responsible for USA’s troubles.

Of course not. The true issue is yet again, a psychological one. On the matter of being depressed I believe.
Our societies do not fulfil individuals enough so that they are at peace enough with themselves.
The cause ? Not sure. But some ideas:
– too much individuality, some people have proposed that the way of living separated from our group in individual or couple apartment is part of the reason.
Personally I think it varies greatly between persons who have strong dependencies on others for social validation, and people who can live alone just great and are better this way. We all need our time alone for zenitude.
– too much materialism. That contender seems more prone to be the real cause.
Some trends such as minimalism, seems to be the way out of this for some people. Also depicted in the movie “Up in the air” with George Clooney. I Love it because I identify a lot with this.

We are rotting in an environment that’s not suited to our nature it seems, and that places us in a relative average discomfort. The economy is strongly linked to this. Economic depressions causes mental depressions as well. If this isn’t materialism at its best, I don’t know what is.

So there we have it, and internal mal-etre is the real cause for racism. But because it is unrelated, even if racial issues could be solved, like by closing the borders, it would not cure the primary cause for this mal-etre.
Therefore, it’s wrong. Not the “god say it’s wrong” kind of wrong, but the “1+1=3” kind of wrong.

Now let’s move on the pragmatic reasons why it’s wrong.
THe best way to realize it, is to live it. I have been the target of racial hatred as a white person in Japan, so rarely that I only remember one case. But still just this one case of getting insulted of “shitty foreigner” shocked me to the core. But that’s one thing, the pernicious issue is in the details, we are segregated as forgeiners in any country. You don’t have the right to vote, you need a special id-card that’s not the same than normal citizens. And in a homogeneous country like Japan you stand out a lot, so people behave according to cliches and prejudice, which is deeply annoying.
I am addressed to in English right out of the bat, or not even spoken to at all, with people using hand waving signs and expressions to communicate, rather than Japanese which I understand very well.
Some kids try to shout “hello” at me in the street, they mean well I’m sure but it’s totally wicked.
Take the opposite situation in Los Angeles:
An asian-looking guy walks down the pavement, no American would dare shout “konnichiwa!”.
Why ? Because you don’t suppose the asian-looking person is indeed a foreigner at first sight, you suppose they have Asian ancestry but are American citizens so you address them in English normally.
Shouthing “konnichiwa” would be derogatory and dangerously border line. The person could even be Korean or Chinese and the greeting would fall flat, even will become horribly negative because these countries don’t like each other.
It’s like shouting “guten tag” to an Italian because he looks caucasian. He’d be like “wtf” ??
Another quite sad observation is the discrimination in lodging, the owners of the apartments decide who can rent and who can’t, based on if they tolerate foreigners or not. And that’s not even illegal here.
I can tell you that getting the real estate agent telling you “you can’t live here” because the owner said so, feels infuriating. It’s like being told that you are a lesser piece of shit.
I also got police control, 2 times. That’s not nice to be assumed suspicious by default.

So all these little things are very upsetting and disturbing, and taints the experience of living abroad.
Realising how it feels, is the only true way that one can understand how racial discrimination is an absurdity and total wrongness all and by itself.

The third reason, is that no country is uniform. Each time some French say “I’m a good Gallic”, I’m saying to myself that he can’t be that stupid ? The country is made up from dozens of barbarians invasions from the South, the North, the East and watnot over the centuries. It’s the same everywhere. Japan was quite protected on its Island, but humanity could not spawn of of god’s ass in Africa and Japan simultaneously, so any human being anywhere today is just a human, who migrated from the source of origin anyway. Hitler thought Ayrians were superiors, so if they are, Donald Trump cannot be superior too ? Mathematically there could be only one superior. Since everyone claims his own superiority, in truth no one is.

To conclude,
When did it all already happen in the past ?
world war 2.
So where do you think this could be heading ?
should we be afraid ?

Light rendering (2)

Hi guys,

part1 : light rendering on maps

Allow me to post a follow up, on the implementation of light mapping in projet SERHuM.

I am doing final gathering right now for global illumination, it is totally broken still, but definitely we observe some results.

Capture3

On this image I set the sampling to 90% stratified to exhibit the bugs better. But we can see that the light that should be coming from the diffuse reflection of the wall on the right is definitely flowing somewhere at least. And we can see some beginning of truth in the indirect occlusion behind the cylinder.

Here is a schematic drawing I made in an attempt to explain how the final gather algorithm works:

lightmap_finalgather

The grey dots are the photons existing in a “floating cloud” which is called a photon map. Personally I decided to spawn them out of the lumels I previously lit in a first pass.

The second pass would be to create indirect lighting by going from the lumels again, in all directions (primary rays in red), gather k-nearest-neighbors (KNN) photons around the hit position, and tracing back to the starting lumel. If the secondary rays hit something in the way back, they are shadowed, if not, they can add some illumination. We use a double Lambert N dot L, one with the photon normal and one with the origin point normal.

Unfortunately, this algorithm is pretty slow because it costs O(lumels × samples × photon_density × average_radius) which is damn expensive. But I have hopes of being able to apply some tricks like cone tracing later on. First if this works at all, this will be a nice achievement in itself.

Let’s see an image computed with full random primary rays (or so I think):

Capture4

There is one thing going well in this image, and that we expect:
The darkening of the zone behind the cylinder. This is a soft shadow because the wall on the right is an area light, so that’s ok.
But the ground is very unexpected, because it’s dark, it has no reason to be darker than the left-side wall. It has a slightly steeper angle I concede, the left side wall is straight facing the right, so lambertian are closer to 100%, but even though the ground is tilted, it should get an average of 45°, (√2/2)² = 50% energy. Here we have almost zero on the left (where the lambertians are actually higher) and some noise on the right, where the lambertians should be close to zero because of steep incidence to the rays coming from the wall. Weird. That’s a bug somewhere.
Also the banding we see on the rooftop and the cylinder, that’s sign of some spurious self occlusions ?

to be continued…

PS bonus: a capture of the photon cloud with cluster coloring. (Cluster = one spatial cell for the KNN lookup)

photocloud

Windows and paths

Microsoft doesn’t get it…

11 years late to the party (in 1981), Microsoft arrives and lay a path identifier system mess.

C:\stuff\file.ext

So first thing they do different is to make use of backslash as path separator.
The world was using slash, so maybe they said to themselves “hey let’s be trendy and invent something similar but different”.
Thank you for the MeSs, MS.
They also decided it would be convenient if slashes were supported also, so they said they should be equivalent.
Here the true story : http://blogs.msdn.com/b/larryosterman/archive/2005/06/24/432386.aspx
They even knew it was bad, they fixed it, and then they kept it. …wait wat ?

Except slashes and backslashes ended up not being equivalent, and most times backslashes are mandatory, in the console if you want completion, in command line arguments you often get problems with forward slashes, and to locate network paths it never works with slashes, e.g. \\othermachine.
Of course, the drive letter is wrongness incarnated, first it limits them to 26, and if you want more, forces you to use mount points, thus creating a first inconsistency. If something is not simple and elegant it should be thrown away and refactored, right now.

How do you relate paths between two different drives ? you just can’t.

in Unix you would simply do:

/media/c/stuff/../../d/otherstuff

in windows:

c:/stuff/../../d:/otherstuff

except that’s forbidden. that’s right, this style of path is forbidden, they explicitly rule it off. good job again MS.

If this mess wasn’t enough, they decided that paths were historically limited to 260 letters and that was too small, and we should have a way of having more.
Instead of just doing so that we can have more (simply), they said, to use more, use the a special long path form, with this prefix “\\?\”
Hem.. they never learn do they ? if it’s not unified it’s just increased costs for support all over the world. And guess what, nobody bothered.
Not even them, they don’t support this form in 90% of their own API. Good job MS ! And the irony, is that even in this form there is a limit at 32767 characters, not even unlimited.

Of course, it doesn’t just end there, in 1993 they wanted to allow non latin characters, so they thought they will encode characters on 2 bytes each instead of one, and they implemented a crappy UCS16 which of course cannot support all of the world scripts since there are more than 65536 of them, and of course, it makes the old form and the UCS form binarily incompatible, and also endian sensitive. Good job MS !

Unix chose UTF-8 which has many advantages as listed here http://utf8everywhere.org/
In Unix, you don’t get bullshit long form prefix; you can make relative paths cross-drives; you don’t get special prefixes for network paths; you get binary compatibility for any language in the world, and of course, only forward slash is used as a separator. Which allows the backslash to be used as escape character like it should be, or regex escape. Escape in Windows is just… you guessed it.. a mess ! yes. You need to use quotes and you always need to try it thrice before getting it right.

That’s not even the end of it just yet, because the drive-first-mandatorily form was an inconvenience, they recognized the superiority of the Unix form, and in NT you can specify \Device\HarddiskVolume and network and even pipes in a unified fashion, way to go MS ! But… this is mostly internal and anything not-a-driver or working with NtQueryObject cannot use these forms… gg.

So once again, you get a system-over-system-over-system anti pattern, redundancy everywhere, incompatibility, emulation, legacy, and a stinky smell of crap everywhere. This, is MS’s world.