2016-07-08 Security Notice event
2016-07-08 Security Notice
Last night we sent email notifications regarding a security incident that took place within our server infrastructure on 2016-07-08. While our team is working on the technical and forensics aspects of the incident response, we want to be fully transparent with you regarding our current status and help you protect your own infrastructure.
You’ll find answers to some of the questions you may have below.
Again, we apologize for the inconvenience and extra work this represents, and are committed to assist you through this process.
Chief Security Officer
Most importantly, what should I do now?
We strongly recommend that you immediately revoke or rotate any credentials in use in your Datadog account as described in our email.
For AWS users, Datadog supports two mechanisms of integration. As you update AWS integration credentials we strongly encourage the use of AWS IAM Role Delegation. This stronger method of AWS integration prevents the sharing of security credentials, such as access keys, between accounts.
Are the emails I received today from Datadog legitimate?
We sent two emails:
A password reset notice that was sent to all users with a stored password (Google Auth and SAML users aren’t affected)
A security notice that was sent to all admin users, instructing them to rotate / revoke credentials stored in Datadog
If you have any concerns about the legitimacy of any email you have received from Datadog know that you can reset your password by directly visiting our site at https://app.datadoghq.com.
What happened to my Datadog password?
Passwords are stored using bcrypt with a unique salt, but out of caution we have invalidated all stored Datadog passwords (Google Auth and SAML users aren’t affected). You can reset a new password at https://app.datadoghq.com.
What is the scope of the incident?
We have detected unauthorized activity associated with a handful of production infrastructure servers, including a database that stores user credentials. A user also has reported unsuccessful attempts to use AWS credentials shared with Datadog. To err on the side of caution, we are recommending revocation of all credentials shared with Datadog.
What about the Datadog agents running on my servers?
Any Datadog agents running on your servers are not affected by this incident. They were designed to never receive any data or code from our servers. They are also isolated from our own infrastructure, only ever communicating outbound from your instances to us via HTTPS. Our agents do not send local credentials to Datadog servers for storage.
What is the current status of the Datadog infrastructure?
Datadog is currently operational. We have rebuilt all identified compromised systems and additional infrastructure. Any known vulnerabilities have been mitigated.
Will I get a post-mortem? What will you do to make sure it doesn’t happen again?
We’re still piecing together the attack and we have brought in third party incident response and forensics experts. We expect forensics to continue well into next week. A post-mortem and longer term plans will follow.