Paxful is the place where the mainstream meets bitcoin.
We know how important it is that we give our users the most stable and secure experience possible. On April 15th Friday Paxful.com was down for seven hours. Fridays and Saturdays are our busiest days and the pain from our users was made clear. People depend on Paxful to literally feed their families and we take that very seriously. We worked around the clock for over seven hours and another eight hours after that trying to figure out exactly what had happened until we felt confident in securing our users.
We were chumming away, lattes going down and techno blasting on our new office Harmon Kardon sound system. Building up towards our big Roots launch, the affiliate program that would uplift every boat in the bitcoin space. We were all juiced, jazzed and zoned working away with those sly confident smiles developers only miss once they are gone. Alex was the first to notice that Paxful.com was down.
Good thing, Alex is our systems admin and no one dives deeper or watches more closely than he does. After scanning all monitors he determined that the server itself was down, this was no application or technical security issue. Our web host was the rock solid SoftLayer, owned by IBM they had a stellar reputation and we bet on it. Ray plowed through their live phone support until he learned that the main app server had indeed been rebooted. By this time seventeen minutes had passed and apparently there was another Paxful employee on the phone with SoftLayer at that very moment!
It took a month of back and forth with Softlayer before we got the name of this “Paxful employee” and the magick jack number he was calling from. Let’s call him Dren. A pseudonym stacked on a pseudonym but surely closer to the man than the name his mother gave him. Dren had managed to get them to reboot the server into rescue mode. We are not sure exactly what he said to SoftLayer because SoftLayer claims that their internal phone logging system went down at this exact moment but whatever he did it wasn’t anything dramatic as at this point we had come to realize that SoftLayers defense against social engineering via the phone was porous at best. A week earlier that had been a fruitless social engineering attempt made and we caught it early. SoftLayer assured us that every precaution would be taken to beef up phone support defenses for our account and we took their assurance at value, big mistake.
Ray was still on the phone with them racing ahead of the hacker and still trying to figure out exactly where their support was in trying to get our server back up. The danger was that if the server went up in rescue mode the hacker would have a much easier time to access it and it’s defenses would be much lower because of the nature of “rescue mode”, so Alex as in a race to beat the hacker to accessing the box as it came up. SoftLayers delayed responses did not help here. The nature of the circus became apparent as it turns out the SoftLayer rep working on the box was sitting next to the rep Alex was chatting with and yet they had no clue what was going on.
The dance continued for hours as we tried to get valid information from SoftLayer all the while trying to bring up the server without putting any user data in jeopardy. We decided to error on the cautious side and not bring the box up until we were absolutely sure that the social engineer did not have access to any user data, but first we needed to know just how far he got. SoftLayer was not giving us any information and without the ability to bring the box up we could not access the logs, rescue mode is a different OS and has different logs. Then something happened.
We got a contact request from DREN on skype. Several of us did actually all at the same time. This was the social engineer and he was asking the BTC equivalent of $20,000. Dren claimed he had our entire database and would cause us a public relations nightmare. He threw out some estimations of our revenue and a few other pieces of information to “prove” he was real. There were a few problems with Dren’s story.
- The revenue numbers Dren was throwing out were calculable from our public volume data found on CoinDance
- $20,000 was far to little to ask if one had a real dump of the database. Dren could not produce a single row of database data even when we offered double the bounty.
- The supposedly “sensitive” information he had amounted to little more than a few file paths on a staging server. We were not sure how he got that information but it certainly wasn’t what one would present if one had accessed a server.
- Dren’s tone was that of a confused child trying to impress grown ups. The more we challenged him the more flustered he got.
We began using the conversation with Dren as a diagnostic tool. Since we could not access the server or get any information from SoftLayer we pushed and prodded Dren into giving up more and more information. The more we got the more we realized that he had not gotten into out servers he had however most likely hacked the paxful skype account, read every conversation he could find and used it to socially engineer SoftLayer. A few more bait tests and it was 100% confirmed, Dren was a script kiddie who had nothing. Skype apparently uses a local hashing algorithm that makes it easy to brute force.
With the penetration vector confirmed and the level of access known we had the confidence to bring the servers back up. Our System admin worked for over four hours to make certain that it came up securely and all vectors were closed off. He brought each service back online only after confirming 100% security on each running process and configuration. As he worked we faced a torrent of angry and desperate twitter users.
Users were angry and just as their frustration and rage reached a fever pitch Alex for the server up and everything was running. We all breathed a sign of relief after seven tense hours. We did not realize it was 6AM in the morning on a Saturday night and despite wanting the pass out we still had one last thing to figure out.
With the spice flowing now we turned our attention to making absolutely sure that the hacker never even hit the ip of the server. We asked SoftLayer for a simple yes or no answer if any foreign ips hit our server. They stalled for a week even when we made doubly certain to remind them that the more time that passed the more vulnerable we were. Finally they claimed that they “had no such logs” a blatant and outright ridiculous lie considering that every router has internal logs especially in a network of SoftLayers caliber. What were they hiding that would make them tell such an infantile lie?
We had our attorney write them a letter and kept asking them for a simple yes or no answer. We got nothing. Our other task with them was to make certain that we were safe from future social engineering on the phone. Our Console was locked with 2FA but there was little we could do on for phone support except ask them to keep notes forbidding any phone support access unless full verification was passed. What passed as “full verification” for SoftLayer was pathetic. One only need answer either one out of three security questions, the last four digits of the credit card on file or the ticket number of an open support ticket. It became clear why they were stalling and refusing to answer us, their phone support security was a complete joke and they were liable.
If this was the first time we would understand but there was a social engineering effort made on us a month before and we caught it in time and asked them to fully lock down the account, nothing was done. That it happened again and how SoftLayer has handled the situation has left us with no confidence in their ability to stave off social engineering, which is a pity as they have proven to be an excellent host and quite generous with us thanks to their Catalyst program.
We will be beginning litigation against SoftLayer for putting our userbase as risk.
Social engineering doesn’t get as much respect as raw technical hacking but it can often be just as deadly if not worse. Fin Tech startups need to make doubly certain that their technical partners especially their web hosts are setup to defend against skilled social engineers. We will all be targeted more and more in the future and there is no telling the extent hackers will take things to get at the magical internet money. Be ever on guard.