I was having trouble sleeping one evening and I had gone to bed and then I woke up so I went downstairs to just futz around for a little bit and I turned on my computer and I was looking at my email and here was a message from my bank.
B of A saying that my account had dropped below $25. And at the time, it didn't trigger anything in me because I only, I knew I had about 30 something dollars in it. And it was a account that I used to keep money for equipment materials, that type of thing for my business.
But anyway, so I thought of that and just, yeah, okay, fine. And then I went back to sleep. And when I finally got up in the morning, all of a sudden I'm sitting there making my coffee and I'm going, well, why did my account go down below $25?
I haven't used that account in a couple of weeks. This is Tom. He's just found out that somebody's used his credit card without his approval. So obviously things are shot. So I immediately called the bank and I said, I don't know what's going on.
But I got a notice from B of A that I was overdrawn. So their fraud department said, OK, fine. And they started a deal and immediately notified me that my accounts were frozen and that I couldn't do anything. So this was kind of a frustrating thing. I'm sitting there saying to myself, and how in the hell could that happen?
These are true stories from the dark side of the internet. I'm Jack Recider. This is Darknet Diaries.
This is the story about a time when a major retail outlet got hacked. I'm not going to give the name of the store or even when this occurred because those details are important. This story is fascinating enough without it. This company is huge though. They have thousands of stores in the US and many more all across Europe and Asia. They do business both online and with physical stores all over the world. And of course, each of their physical stores have computers that are connected to the network.
The story starts out with an email. One of the credit card brands had found some cards on the black market being sold and they were linked to this store. The card brand emailed the store, letting them know some cards that are on the black market.
have a common purchase point of these retail stores. Specifically, they found 10 credit cards on the black market, whose last purchase was the store. Now, 10 credit cards found on the black market is not really that big of a deal, especially when credit card doms have tens of thousands of cards in them. But the store wanted to investigate anyways. So they called a consulting firm called Croll. Hello. Hey. And they asked Croll to investigate their network and try to find if there was any traces of malware on it.
And they got a team to help them out. Let's meet two of the members of that team.
My name is Courtney Dader. I'm a senior managing consultant. And my name is Matt Bromley. Both Courtney and Matt are incident responders. Their job is to go into a network that is breached or may have been breached and fined, isolate and fix the problems. Both of them are used to working on larger cases and specifically cases involving finance and retail. So doing an incident response for a global retail company is what they're good at. And these two got right to work looking for any signs of hackers in the network.
We usually start with two different approaches. Number one, we'll start to understand what's already been out there. So what type of data has been leaked and is it something that could actually be resulting from a breach? That's kind of number one. Number two is we'll also come in and drop in whatever tech we use. So the tool that we use primarily is carbon black, which is an endpoint monitoring tool.
and point monitoring and analysis tool. We use carbon black and we deployed that throughout the environment. So it's usually that two step approach is to quantify the data that's been exposed, drop in our tech, and we start to kind of get deployment throughout the entire environment. This particular customer has over 10,000 systems globally. They're acting in every major country that's out there. They've got
outlets in every single one, if you will. And we were deployed on a global scale to almost every system that they owned that was running Windows and that could handle the agent software. So basically everything Windows XP and above. The computers at each of these stores had antivirus running, but it wasn't picking anything up. And the endpoint monitoring software wasn't finding a lot either.
but the store's IT team had noticed one system was acting funny. So they took it offline and asked Matt and Courtney to look into it. So originally the company had identified that first system for us. It was actually already identified. They had pulled it offline. So that was somewhere where we were able to look. And this particular system, it looked like a bloody murder scene. When you just take a look at this system, it's just stuff everywhere. There's malware all over the place.
And we also found several directories full of nonsensical binary data files that just didn't have much of anything inside of them, but there was there was size to them.
And on Windows environments, very rarely do you come across just kind of blobs of data that don't have any structure or purpose or are in the wrong place, like these files were. And we came across thousands of them. And every single file, believe it or not, also had the same naming convention. It was a six digit date, so month, month, day year, followed by a host name,
followed by a seemingly random string of digits. These weird and unusual files were encrypted. So neither the company or the team could see what was in them. And based off of some of the information that we found in different logs, we were able to basically track
via different IP addresses where that account that was owned was also reaching out to. So from doing that, we were able to see where it was reaching out and where they were pushing malware to from that machine as well. And then in doing so, we also found additional malware as we kept going out. There was just more and more specific variations of the same malware,
So from there, we were able to get most of it, and then we got to a second major host machine, and we were able to then spawn out from that one as well. And then we got to a third one. And we identified basically, you know, upwards of about 1,200 compromised systems. The malware sits on these systems, scrapes all this data out of memory, and then it pushes that data over the wire to the central repository system.
So this particular malware was writing to an output file. So we knew that this malware was writing to a particular output file with this particular file extension. So we were going looking across a network for all of the files with the same similar naming patterns and similar file extensions. And we also looked at the different malware because some of them have a different naming convention in each of the different output files. So we made sure we were able to determine all the output files. We pulled them all together.
So at this point, the team had found over 1,000 computers, all in the network with the same malware. But here's the worst part. Want to guess what these computers were used for? They're the cash registers where you go buy something and you swipe your card or you insert your card these days, you insert pen and chip. Sometimes you still swipe depending, but the systems that are compromised are those particular, those particular systems.
malware that exists on 1200 computers that are cash registers is terrifying. So the team began studying this malware to try to understand what it's doing. The output files were encrypted using XOR, and the company couldn't decipher it. But Matt Courtney spent some time and eventually cracked the encryption.
They were able to see what was in those files and what they saw confirmed their fears. The malware was grabbing every credit card that was swiped on that register and putting it in this file. The way that they were scraping cards were they were scraping from memory, but they did it basically using 50
anything that was 15 to 16 digits, and usually it's followed by something along those lines of an equal sign or some sort of the limiter, and then it gives you the four-digit expiration number. So based off of that, that's where they were able to pull card data down from memory. And using that basic algorithm, they were able to get mostly positives, but occasionally you do get a few false positives and some of the things that are scraped.
This company is now in a nightmare scenario. So many of their cash registers have malware on it that's scraping every credit card that's being processed and then sending that data out of the network. As the credit card data leaks out of the company and into the wrong hands, the criminals who get these cards can use them for themselves. They do things like write the card data to a blank credit card and then withdraw cash from ATMs or use these cards to buy gift cards and sort of launder the money.
The thieves found a spring of seemingly endless money, and they were making some serious cash off this retail company. Our initial feeling was there's going to be a lot of data here. The initial thing that you think about it's not.
It's like an investigator's curse. You never want this number to be high, but one of the first things you think about is how much data has been stolen, how much data has actually been accessed. Because when you've got that much malware in an environment that big, the first thing you have to think is I've just found thousands of output files. What are the chances that I'm also looking at millions of credit card numbers? You never want to be in that target world, right? Those tens of millions of
numbers and that kind of stuff. You never want to be there, but unfortunately when you uncover a breach this big, one of the things that creeps into your mind is, you know, please, please, please don't be that, don't be that large. But wait, that was North America. We found the exact same type of infrastructure in Europe, as well as in Asia Pacific. We had these central pivot points.
that the attackers were using basic clearing houses. And malware was directly writing over the wire, right to these systems as they were going, thousands of systems at a time, simultaneously as well. And like I said before, the registers all had antivirus installed on it. But it wasn't picking up this malware in particular.
So this is actually a new variant that we've uncovered. It was not previously known. So there's a couple of little surprises about this malware. So first off, it was an unknown variant, but it's a derivative of tiny POS. There's a legitimate tiny POS, point of sales software, and then there's a tiny POS,
piece of malware. And TinyPOS is a piece of malware that is pretty, I don't want to say sophisticated, but for what it's able to do, it's pretty well developed. And I say that, you know, knowing I just complimented a malware family. But it's pretty well put together. But the kicker here is every single piece of card treatment malware that we came across on this case was less than six kilobytes in size.
As we kept adding more and more stores, it was definitely a point where we're like, are we going to have millions of cards here? I think, especially as we saw the output files go higher and higher number, I was like, oh, well, let's hope some of these overlap. Yeah, definitely no shit moment.
But wait, wait, hold on. It's worse than that. About 80% of the systems was the point of sales, the point of sale registers. The other 20% was the back of house systems as well, that the back of house systems that sit in the back of the store that no one has access to, aside from the store themselves. But those systems had malware on them as well. And those systems were also, also had data that was being scraped in them too.
delivering this kind of news to the client is never easy. The team had to call the client to tell them what they found. Yeah, I would say it was us, them, and lawyers. That was a fun chat.
I guess to give you a very brief perspective, there's always that moment of like, you know, when you receive that external alert or when your client receives that external alert, there's always that moment of like, is this bullshit, you know, or like how real is this? So when we had that call and we said, hey, FYI, we have uncovered this. We haven't covered this malware. We found what it's able to do. We want to cover all these output files. The very first reaction, because the output files were encoded,
They actually had a custom encoding that the malware was using. Because they were encoded, there was a little bit of like, well, that data is just garbage, right? There's nothing in there. And we're like, well, we actually decoded it. So there was definitely a little bit of an oh shit on their part as well, because it becomes real once you find out that kind of your greatest fear has come true, which is someone's been hanging out in your house for a while.
The lawyers, they do a very good job of letting us get everything out and then they process it and then they come back and ask questions. But there's always a little bit of hesitancy. They want to make sure you're correct, you know, because if you're
If you come out of the gate, you say, yeah, we have evidence of credit card of a credit card breach. There's a lot of wheels that start to turn. You're on a clock depending on the state. You know, some states you're on a clock bin about disclosure, that kind of stuff. You've got a file for protection with their credit card brands and everything like that. So they are they ask as many questions as possible to make sure you're 100% positive.
and what you see backs up. You know, technically you can back up what it is you're saying because they know that there's a lot of money about to be spent based on that finding, based on that opinion. So next, this is where Courtney and I then fall into the quantification mode.
where now, OK, we've got a breach. Now we need to understand how wide, how big, and how much data is at risk here. That's kind of the next step that you want to answer is is how much data is actually at risk when you have one of these breaches here and just how far back does it go? Doing some digital forensics, the team was able to see when the malware was originally installed and where it came from. And from this investigation, they found the hackers way in the network for about seven to eight months. They're in there.
They were in the network probably at least a solid month of just reconnaissance before they built their perfect piece of malware. And then even throughout time, we saw them make slight modifications to it as they kept going forward. They knew exactly where to go get the credit card on every system. Unfortunately,
The system that they first compromised was they first came into was no longer available. I think firstly, they came in with a fish only because there was very little exploitation of systems. Even though there was a very vulnerable environment, there was very little exploitation of those vulnerabilities. And that may be, again, because they didn't need to do that. So I don't want to rule that out immediately, but I don't know. Most of these cases, I always see start with some sort of a fish.
Especially to set up the infrastructure that these guys had, that the attackers had set up here, it makes you want to think it was likely a targeted approach. But you never know, you know, you never know until you find it. Fishing is when a hacker targets an employee to try to get them to click something they shouldn't click on. It could be an email with a malicious link. It could be a word document with macros enabled. And once the person clicks the malicious link,
that computer can become infected and then under control of the hacker. And when the hacker is in the network, they can move over to another machine to start setting up their malware. With simple cocktail nap and math, you're scraping, you know, 600 stores, five to 600 stores for a period of eight months. And that period of time includes the summer. It includes multiple sale weekends. It includes the buildup to Christmas.
And that kind of stuff, including all those various time periods, you could very easily get into the hundreds of thousands or more millions of cards easily with kind of those considerations and those factors. So the next step is let's get these output files parsed. Let's start to concatenate this data together. And let's start to dedupe it and see just how much data we've got exposed here.
Yeah, I want to say it took us at least two weeks to get through all of the credit cards and really be duped them and make sure that everything we had were actual card numbers. Something particular in this case that we ran into was credit cards that looked like credit card numbers but weren't actually valid card numbers. So that was something that we had to do a lot of
deduplication and verification with both on our end and with a little bit of help of the card brands to determine if what we were seeing were all actually card numbers. We started uncovering card data that was expired.
We're like, how are we finding so much credit card data that's expired? It's one thing if you find, let's say you have 20 numbers from the day and you find one is expired. You're like, okay, someone accidentally spiked an old card or something, right? But then you start to wonder, you're like, why am I seeing a significant percentage of cards that have already expired?
What was happening in this case is, if you remember earlier, I mentioned that the malware was on the back of house systems. The back of house systems were running SQL servers, and the SQL servers had historical, unencrypted track data that was being loaded into memory. And the malware was picking that up. They were picking up transactions from as long as four years ago.
So the attackers were effectively peaking back in time. They were looking at transactions from three to four years ago that they had no, they had no visibility to, which is another unique angle because most of this malware exists at the swipe or it exists to steal at the swipe. Now the team is ready to begin removing the malware from the network. They needed to understand every hole that was in the network and patch everyone so that the hackers could not get back in.
We didn't actually have to take any of the stores offline. Once we were able to really find those pivot points, taking those offline or at least making sure that we had process blocking in place was able to stop it for the most part across the network. Yeah, we ended up shutting down the clearinghouses, the central points that they were using. We ended up taking those off first and then waiting to see what would happen next.
When the first time we had kicked them out, we actually saw them re-enter through Asia. And within about three seconds of re-entering, they had re-compromised 40 different systems. I never want a company to actually debrief split.
Part of my job is to find that stuff and part of Courtney's job is to find that stuff. So our initial reaction was first off like, oh shit, you know, we knew this is going to happen. We didn't know it was going to happen this fast. So that's reaction one. Reaction two, usually what happens if you successfully kick an actor out 100%
Usually, you'll see a fishing campaign or something. They're trying to get back in at the beginning. To see them come through the network with that speed, the next thing is like, oh shit, there's another backdoor out there, which we got to go track down, which they ended up being a system we didn't have visibility to. Number three, there's that moment where you see how quickly the attack we're doing what they're doing. That speed's volumes to how long they've been in the network.
If you're watching an attacker, if you're seeing artifacts from an attacker very recently, and it looks like they're fumbling around in the dark looking for a light switch, then you're like, there's a very good chance this person hasn't been here that long. So watch someone re-compromise four dozen machines in 10 seconds. Okay, this person is come back home, they put their feet up on the couch, they know exactly where the remote is, and it's a very easy thing to slide right back into.
I think it was interesting to see them come back in so quick, but it was also interesting to see which tools they were immediately using because at that point we had the live response there. So we were able essentially to sit there and basically track what they were doing and see exactly
how they were moving. We had some of these records before, but it was just nice to actually sit there and be able to confirm like, oh, they came in through here, okay, while they ran an IP scanner, and then they hard-coded and were able to log in to all these IPs in the matter of three minutes.
The team had discovered that besides the malware, there were also back doors installed on many systems, which is how the hackers kept getting back in. They had added 350 back doors. However, they weren't pushing the malware to each, like, all 350 systems didn't have all the malware on it. It was almost as if they were just allowing themselves access back in in case they had ever got closed off.
But the team was able to find each and every backdoor and take every pivot point offline and stop any more credit cards from leaving the network. It was a good feeling to finally get this malware under control. There was about a month where every call got worse and worse. So there was definitely a moment of really is this ever going to end. You know, you can definitely get bad news fatigue.
after a while, but eventually once we started taking things offline, it then turned into positivity and we were able to actually deliver good news calls, which is, you know, hey, we've actually remediated things. So the attitude started to shift when once we had kind of figured out the way the global, the attackers global infrastructure was set up, but it was only, you know, it's only when you get to that point where you kind of mapped out the whole world that you can start to actually breathe a little bit.
Luckily, this malware was not ingrained to the point where it kind of became symbiotic with the environment, like some malware families do. But this one was pretty easy to delete. And I don't say that as a challenge. I say it was definitely a pain in the rear, but it was simple enough to delete and then disable the services to prevent it from running again. As the team was cleaning the malware off the network, they gave some suggestions to the company to improve their security.
Yeah, some password changes were necessary. Maybe something down the line of changing their network infrastructure so it's not so flat. That was definitely one of the things that enabled us malware to get as far as it did because every system could essentially reach every other system. And there was similar passwords shared.
administrator accounts, like privileged, privileged accounts that were accessible on many, many machines across the network. So I think this was a big learning point of how to properly secure your network and make sure encryption is in place to prevent this from happening again.
to try to trace this hack down to the person who is responsible is sometimes impossible. You can look for clues in the malware like the language that was used in writing it or the time zone that it's set to, but these things are just small clues that aren't very strong. Trying to figure out who did a hack is called attribution. I'm a firm believer that attribution doesn't really get you anywhere. Unless you're sitting in a political or an executive role and you've got to make decisions off of
who may be behind this keyboard and that kind of stuff. However, it's always interesting to know. The only thing I can say about this one is one thing we haven't mentioned yet. North America, there was one server that was treated as a clearing house. And then there was two additional systems that had backdoors on them. In Europe, it was very much the same thing. Europe and EMEA and whatnot was very much the same way. There was one or two systems that served as central pivot points
Asia, on the other hand, Asia had somewhere between 350 and 400 unique back doors installed at it. Almost every system got a back door. And a lot of the compromise itself actually started in Asia. It actually started in mostly Southeast Asia and that kind of stuff. And that doesn't lend any attribution whatsoever.
It's just when you're going after credit card data, it's a very interesting place to start if you catch my direct. Even if you pinpoint it that specifically, what are you going to do? You're a company that's headquartered in the United States. What recourse do you have? You've got to get your network back up. You've got to get to a point where you are not
You know, you got to a point where you're not having to fight fires every day. You don't really have time to, well, you wouldn't get higher a team to go after these guys or something like that. You know, I mean, good luck. So what was the final number of credit cards that were stolen? So that that number is to the best of my knowledge, still being sussed out. But I think after everything we had come across, we landed a little shy of a hundred thousand. That was all.
which was a very surprising number and a very relieving number as well. I don't know a lot about the current carding black market conditions, but it's safe to say these are probably too many cards for these hackers to try to scrape money out of themselves. So they're probably selling these cards in bulk somewhere. And the cards go anywhere from $10 to $100 each. So even if they got $10 per card, that means these hackers made a million dollars off this company.
So now the company has to do what it can to try to clean up the problem. Primarily then it falls on to the company to work with the banks and work with the credit card companies and get new cards out there. This breach was publicly announced and it hit the news. But the public's reaction to it wasn't a huge deal. It was in short, it was not as crazy as you'd think.
It was not that big of a deal. And I say that because you've got predecessors like Home Depot and Target and some of those huge major breaches, you've got predecessors like that, which received weeks, if not months of news.
And this one was not as prolific as that, you know, from kind of the world view. That and then on top of it with the whole network infrastructure right now and how often we're almost seeing these reported in the news and some of the larger breaches that we've recently seen, including Social Security numbers. I guess credit cards kind of a little bit fall to the back and I were always worried it gets stolen, but in the back of a lot of people's minds are like, oh, I'll just replace it. Get a new one.
Besides this being a major headache for this company and even a bigger headache for the credit card companies and banks, this also can severely impact the people whose cards got stolen. At the beginning of this episode, you started to hear from Tom. It's possible that Tom's card data was stolen and sold on a black market just like in the story you just heard. Someone used his card fraudulently and his bank was investigating to see what went wrong. So let's hear how his story pans out.
The morning that they did it was the 12th or 13th of December. So this effectively wiped out Christmas.
And I'm a licensed contractor and I receive some of my business through an outfit. And with my accounts frozen and nothing able to go in or come out, the first thing I found was that they stopped working for me and they said, well, your bill is overdue and your bill is overdrawn and we can't get any money. So you're stopped until something happens.
But luckily I had a financial backup on this. And so I was able to survive, but I could not work until this was finally taken care of. So it made a couple of months where things were very, very difficult. My
Main Bank had gone through and they said, OK, we have found the problem. And they had now put everything back the way it was supposed to be. And I was now able to do business with the account. But even having done that, it still took a couple of months to get things squared away. So it was a major interruption in my life.
Courtney and Matt gave a presentation at the Kaspersky SAS Summit earlier this year. In their talk, they went into detail about this new strain of malware. They also reported this new strain of malware to the antivirus companies so it can be detected in the future. Matt has since moved from crawl, has now working at silence, and has most recently been accepted as a SANS instructor teaching digital forensics and incident response.
been listening to Darknet Diaries. For show notes and links, check out darknetdiaries.com. If you want more InfoSec podcasts, there's one that does an episode almost every day. You do a daily wrap up of the news and interview some really smart people. It's called the Cyberwire, and I recommend it for your daily commute. A lot of you are asking how you can help with this show.
Right now, I'm just trying to grow the audience, and it's hard to get the word out, so you'd be a big help to me if you would tell others about this podcast. Think whose phone number you have of someone who might like this show, and text them right now to tell them about it, or post about it on social media, or tell your coworkers. These kinds of things make me super excited to make more episodes. This show is made entirely by me, Jack Resider. The music for this show, including this song, is made by Breakmaster, Cylinder.
Hey, one last thing. I made something you might like. I made a random password generator. Yeah, it's a website that creates some fresh new random passwords for you. Just in case you ever need to create a random password, I've got you covered. Oh, and there's an extra feature too. It has an API, which allows you to use it in your own programs. Anyways, if you want to check out the site, it's called password wolf.com. That's password wolf.com. See you there.
