Security Incident Recap

In this article, I will share a recap of what occurred in the past two weeks, including lessons learned, stress dealt with, and wisdom gained. - CZ

What Happened

A group of hackers was able to gain control of a number of user accounts and made large withdrawal requests in such a way that they bypassed our pre-withdrawal risk management checks. Our post-withdrawal risk monitoring system caught it immediately, and suspended all subsequent withdrawals. While things are crystal clear in hindsight, at that moment, we weren’t 100% sure what exactly happened. Was it an actual user action?  A glitch in the system? Or maybe a hack? As we were still evaluating the situation at the time, we decided to proceed with caution. I put out a tweet saying the withdrawal servers are in unscheduled maintenance mode, while the team continued to investigate what happened. After confirming it was a hack, more questions followed:

- How much did the hackers withdraw?

- Were there previous withdrawals that we didn’t notice?

- How many other accounts did the hackers have?

- What other risks are involved?

- How did the hackers know our risk management rules so precisely? Do we have a mole?

- What do we need to do to get the withdrawal system online again?

While the team was investigating the above, there were further questions that needed to be answered:- How should we communicate?

- What would the community reaction be?

- How much reputational damage would we suffer?

In tough moments like these, we always choose to follow our first principles: Protect Users and Be Transparent.

Communication

After the initial incident, we decided to notify all our channels about the security incident. By then, we were relatively certain there was only a single affected transaction. All our other wallets were safe. We were cautious that the hackers might still have control of additional accounts that we were not yet aware of. Further withdrawals still posed a risk, and we needed to make a few significant changes to the system before we could re-enable withdrawals for our users. This security incident notification stated an estimation of one week of suspension on withdrawals.

In the world of technology, you can never accurately estimate how long changes might take. It is quite different when you compare it to repeated, predictable work. Regardless, our users and community needed an estimate, and once communicated, it became a target deadline for our team to deliver. I did not know how the community would react to a one-week withdrawal suspension, but luckily, being transparent paid off and we received tremendous support from our amazing community.

Lesson: During a crisis, constant and transparent communication is key.

Distractions

Before the AMA, I had been up all night and I was really feeling the effects. So, I took a 15-minute nap just before the AMA. Upon waking up, my team told me there was an interesting proposal from a Bitcoin Core developer. I read it for a few seconds. It involved something called a “reorg”. While I know it’s technically possible for a rollback in a 51% attack scenario, it never occurred to me that it is also technically possible to change one transaction and keep all other transactions intact, while hugely incentivizing the miners. The discussion was already pretty hot on Twitter, so I mentioned it in the AMA as something that was suggested. Little did I know, it was a taboo topic. Lesson learned.

AMA

We had already previously scheduled a video AMA for just a couple hours later. I believed it would be appropriate to keep it, as lots of people would have questions.  This turned out to be the right thing to do.

Seeing me live had put a lot of our community at ease. The livestream was analyzed to death, including a body language analysis, which I thought was a very good thing. It truly shows how the crowd will work as a hive mind on different aspects of analysis. The body language analysis results were very positive, which was reassuring.

Lesson: Get on a live video stream during crises. Your users deserve to know, not just what happened, but what you are doing to handle it, including allowing them to judge your mental state for themselves.

Mental State

I am not gonna deny it. My first reaction was: “F***!”, the second and third reactions were also the same. A few moments after that, I began to come to terms with it, “Well that sucks! What do we do now? Lots of people are waiting for me, some for instructions, some for information and some for reassurance. Lots to do, let’s just get on with it.”

When I checked in with the team, they were already a couple of steps ahead of me, implementing additional security measures to further ring-fence our systems and discussing all available options. The entire team was online. I have seen this mode before, it’s called “War-Mode”. Luckily, our team is accustomed to high pressured situations, and our urge to fight was stronger than ever. A few of them even gave me a pat on the back for planning to do the livestream AMA. A few variations of “Balls of Steel, Boss” came up a few times. They were cheering me on, I knew that was a good sign.

Funds

After 10 seconds of the “F***, F***, F***” state, I did a quick mental calculation. 7000 BTC, fine, I know we have more than that in our own BTC funds alone. There is enough.  A second calculation eased my mind, this was about the same as a quarterly burn we did about a year ago, not “such a big deal”.

Also, this was not the largest outlay of cash percentage-wise we have had to endure. Back in Sept 2017, when the Chinese government issued a letter banning ICOs and “recommending” projects to return money to investors. The news alone caused many tokens to drop below their ICO prices, and many project teams couldn't return the whole amount to users. Binance did help a number of projects raise money on our platform that were affected by this policy. So we did a quick calculation: if we were to help cover the losses for our users and for those projects, it would cost us roughly $6,000,000 USD. Putting that in perspective, while we only raised $15,000,000 two months prior, we spent a bunch of money and were barely cash flow neutral at the time. We decided to do it anyway. I was in a moving subway when the team called me, and we made that decision together in less than 5 minutes. That was more than 35% of all the cash we had at that time. The goodwill that that decision generated eventually brought us many users from China and all over the world, helping to fuel our growth. So, this time, this $40m represented a much smaller % of our cash reserves, plus we had the #SAFU fund that could fully cover it.

Thus, we announced that we would cover the entire loss in full.

Lesson: Money can always be earned later, do the right thing first.

Help Offered

We received tremendous community support, from people defending us, to people helping us by answering questions in the community, and platforms such as Twitter, Telegram, and Facebook. The Binance Angels (our volunteers) have been running at full steam on multiple communities, addressing questions and reassuring our users around the clock. Thank you, thank you; we thank you!

Many partners jumped in to help. Analytics teams started to help us track the stolen funds, e.g. Peck Shield, Whale Alert, etc. Exchanges and wallet services offered to block any deposits associated with the hacker addresses. Several of them may be perceived as our “competitors” by some people, but I am impressed at how the entire community came together and stood united in a time of need.

We also received numerous offers for help from law enforcement agencies around the world. This is a result of working with them closely in the past, usually helping them to solve cases.  Now, they offered their help to us in return.

Lesson: Being transparent makes it easier for others to help you.

Sales Guys

I received 40+ new leads from various security experts/consultants/companies offering to help. Though some clearly intended to help, many were simply trying to sell their services. While all help is fully appreciated, the timing was actually a little off. It would not be good for me to schedule 40 calls during a week when our system is partially down. Some even flat-out suggested that we give them full access to our servers so that they can help us do forensics. Of course, we politely declined. Moving on...

One Quarter in a Week

Our team pushed on, day and night. In places where we congregate in small temp “offices”, we had temp beds from Ikea rolled out. I won’t go into the details here, as we don’t disclose our security practices, but to bring the system back online within one week, all of our teams did more than a quarter’s worth of work in that one week.

A Blessing in Disguise

Speaking with various team members, and as correctly analyzed by community members, such as Gautam Chhugani, this incident may actually be a good thing for us in the long run. Security is a never-ending practice; there are always improvements to be made. We have implemented many of them in this last week and will continue to implement more in the future. Given this incident, Binance has actually become far more secure than before, not just in the affected areas, but as a whole.

Lessons Learned

We always aim to maintain constant and transparent communication with our community during a crisis. We believe this to be a strong contributing factor to the support we received from the community in return. One clear measure is the $BNB price: it dropped a bit on the initial news, but not nearly as much as one would have expected, and even before we resumed withdrawals, it has already made a strong comeback and hit all-time highs (in USD) again.

We hope this will be a new benchmark for how project teams communicate with their users, during both the good times and the tough, and we hope this will help make our industry healthier and stronger.

We Thank You!

-CZ

Updated on July 27, 2022.