68: Triton

en-us

June 23, 2020

Darknet Diaries

Importance of Reliable Safety Instrumented Systems in Preventing Accidents in OT Systems.: Companies should prioritize the safety of their OT systems by adopting reliable and resilient safety instrumented systems like Schneider Electric's Triconex to prevent accidents caused by cyber threats, changing workforce, and complacency.
Data breaches can have a major impact on our lives, and we may face unique kinds of hacks that affect our daily routines. Companies must ensure the safety of their OT systems to avoid disasters or accidents. The safety instrumented systems are an essential part of controlling physical aspects like valves and pumps. Saudi Arabia's massive petrochemical plant faced a severe shutdown when its Triconex safety systems failed. The plant produced 140 million barrels of products yearly, and its components go into making everyday-use items. Safety systems like these must be reliable and resilient to avoid accidents. Cyber-criminals, changing workforce, and complacency are some of the factors modifying operational integrity. Schneider Electric's Triconex offers excellent process safety solutions for a sound and cohesive business.
Importance of Regular Maintenance and Monitoring of Safety Systems in Industrial Plants: Regular inspection, limiting network access, and following proper protocol can prevent malfunctions and unauthorized changes to safety controllers, ensuring safety measures are functional and preventing potential disasters.
Proper maintenance and monitoring of safety systems are crucial in preventing disaster in industrial plants. Unauthorized access to safety controllers and leaving them in program mode can cause malfunctions and shut down the plant, resulting in financial losses and safety risks. Regular inspection and action on system alerts are necessary to prevent such incidents. It is also important to limit network access to safety controllers and use secure methods of remote access. Finally, following proper protocol and not neglecting system alerts can prevent unauthorized changes and ensure safety measures are functional.
The Importance of Incident Responders in Emergency Shutdown Systems: Proper incident response by skilled responders is crucial to preventing catastrophic results. Immediate on-site response is necessary to identify and mitigate security threats, and interview all involved parties to gather valuable information.
Proper functioning of emergency shutdown systems is crucial for preventing catastrophic results. Incident responders play an important role in investigating and identifying potential security threats and taking appropriate actions. Immediate on-site response is necessary to conduct incident response and forensics to mitigate the impact of security incidents. Incident responders are always ready to travel on short notice with go-bags to conduct investigations. It is important to have a skilled team of responders to handle incidents like these and to gather information by interviewing everyone involved.
Investigating a Plant Shutdown and Identifying the Culprit: Analyzing controllers and investigating inside/outside parties with special attention to unusual files is crucial when investigating plant shutdowns, even in hazardous environments.
Investigating the cause of a plant shutdown requires analyzing the actual controllers and identifying any changes made. Safety controllers are embedded systems with limited functionality, making it difficult to extract programs or perform diagnostics. Integrity verification commands can be used to compare what's on the controller to what's on the system. A discrepancy in IO points was identified as the cause of the shutdown. The investigation also involved looking into whether the shutdown was caused by an insider or an outside party, and analyzing engineering workstations for potential clues. The presence of unusual files in the system, such as a Python DLL in an HP folder, indicated that something was amiss. The investigation was challenging and carried inherent risks due to working in a noisy, hot, chemically hazardous environment.
Importance of a Strong Incident Response Plan in Plant Cybersecurity: Having a strong incident response plan and taking necessary actions to safeguard the plant and its employees from potential dangers is essential in ensuring plant cybersecurity. The use of network equipment with good logging can help in collecting valuable insights during the response process.
The safety controller in the sulfur recovery unit, responsible for shutting down plant operations in case of unsafe levels of H2S, had gone down due to the malware attack. Despite the risks, the management was hesitant to shut down the plant and start a thorough investigation. The incident response team finally traced the malware attack to an external party that exploited a computer inside the DMZ. The network equipment with good logging helped the response team collect valuable insights. This incident highlights how essential it is to have a strong incident response plan in place and take necessary actions to safeguard the plant and its employees from potential dangers.
Incident Response Team and the Importance of Securing Industrial Control Systems: The discovery of advanced malware on an engineering workstation in a petrochemical facility highlights the need for securing such workstations from remote attacks and the importance of proactive measures for threat detection and prevention in industrial control systems.
The incident response team successfully identified the attack on the system and its source, leading to a shift in their goals from initial triage to cleanup and prevention. The discovery of advanced malware on the engineering workstation and the deletion of these files by an unknown party highlights the importance of securing engineering workstations from remote attacks. Furthermore, the wider-scale targeting campaign meant that the security of other petrochemical and oil and gas facilities within the kingdom was also at risk. The investigation led FireEye to analyze the malware, which had the potential to cause serious harm if the target emergency shutdown systems were compromised. Overall, this incident illustrates the importance of securing industrial control systems and the need for a proactive approach to threat detection and prevention.
The Triton Malware: A Threat to Industrial Plants: The Triton malware can cause severe damage to industrial plants by instructing valves to operate unsafely, disabling safety systems, and ignoring emergency shutdowns, posing a significant risk to both safety and finances.
The Triton malware was a sophisticated program that could unleash catastrophic damage on industrial plants. The attackers had an understanding of the culture and sensitive operations of the plants, and they had a whole new level of expertise in IT and OT systems. The malware could instruct the valves to operate in an unsafe state, instruct safety systems not to shut down or even create an alert, and make the emergency shutdown system ignore unsafe operating levels. The Triton malware was a passive implant that would implant itself into the memory and wait for a certain packet to be activated. The potential damage payload associated with Triton could lead to major safety incidents and financial impacts to the plant.
The Dangers of Cyber-Attacks on Safety Systems in Industrial Plants: Cyber-attacks on safety systems in industrial plants have severe physical consequences and potential human casualties. International regulations are needed to prevent such attacks, as they are complex, time-consuming, and require high-level expertise.
A cyber-attack on safety systems in industrial plants can have severe physical consequences and potential human casualties. Targeting civilian-protecting systems is off-limits and not specifically regulated. Such attacks may result in prolonged shutdowns, in turn affecting the economy. These incidents highlight the need for international regulation of such attacks. The attackers behind such malware must have unrestricted access to the systems and a high skill set to hack both IT and OT environments. It takes years to execute such an attack, and it requires knowledge and understanding of the particular safety controllers and software. Such attacks are cyber-terrorism, and the people responsible have unlimited resources and time.
TRISIS malware targets industrial safety systems, posing a threat to human life.: The discovery of TRISIS highlights the importance of industrial security, a separate discipline from IT security, with tailored best practices to protect against targeted threats to industrial control systems.
The TRISIS malware, also known as Triton, was discovered by the threat intelligence group within Dragos, which investigates security threats related to industrial control systems. The malware specifically targets safety systems and has the potential to compromise human life. The discovery of this malware prompted Dragos CEO Robert Lee to inform the Department of Homeland Security, as it indicated that hackers somewhere in the world had broken into a chemical plant in Saudi Arabia and had the capability to cause a major terrorist attack. The incident underscores the importance of industrial security as a separate and distinct discipline from IT security, and the need to develop specific best practices tailored to the unique mission and threats of industrial control systems.
Moscow's Central Scientific Research Institute of Chemistry and Mechanics Suspected in Cyberattack: Research institutions with departments related to advanced informatics and critical infrastructure security can build cyber capabilities and cybersecurity is a community effort where sharing threat information is key.
Cybersecurity is a community effort, where companies often work together to benefit the community against adversaries. FireEye suspected that Moscow's Central Scientific Research Institute of Chemistry and Mechanics was behind the attack. Although it sounds illogical that a laboratory can build something with cyber capabilities, it isn't unusual. Such research institutions have departments related to advanced informatics and critical infrastructure security. There is evidence that some operations related to intrusions by the Triton team in known organizations were conducted from an IP address in Moscow. Cybersecurity companies like to publish threat information only when it is already going to be made public, enabling their customers and the community to stay informed.
State-Backed Attack on Industrial Networks: The Triton Attack: The Triton Attack, carried out by a state actor, highlights the need for advanced attack infrastructure and track-covering measures to prevent future attacks on key industrial networks and safety systems.
The Triton attack was likely carried out by a state actor as the complexity and sophistication points to a group with advanced attack infrastructure and motivation beyond financial gain. The attackers may have worked in collaboration with a research institute, possibly with Russian intelligence agencies. The attackers continued to refine and customize their attack over the years and targeted key industrial networks and safety systems. While the attackers did not hide their tracks well, attribution remains challenging. The attack can serve as a reminder for the need for proper attack infrastructure and track-covering measures to mitigate such incidents in the future.
Attribution in Cyber-Attacks is More Complex Than You Think: Attribution in cyber-attacks requires multiple factors and high confidence levels, which can be challenging in the private sector. Geo-political considerations like intelligence agencies, allies, and vendors must be taken into account.
Attribution in cyber-attacks is significantly more difficult than people make it out to be. A high-confidence level of attribution requires many components working together. Private sector high-confidence assessments would have been low or moderate-confidence assessments in the government. Further, national critical infrastructure and cyber-attacks have a tense situation between state players, so it is necessary to be cautious when pinpointing a country. Most intelligence requirements in the private sector relate to how to do better security and prioritize things. None of those things require true attribution of the attacker. Lastly, the discussion around attribution is more nuanced at a geopolitical level than what is generally seen from a cyber-security audience. It requires considering different intelligence agencies, military agencies, allies, and vendors of capabilities.
Clustering Intrusions for Effective Defense Recommendations.: Analyzing clustered intrusions is a powerful technique to deter threat actors and provide better defense recommendations. The latest threat activity, Xenotime, poses a serious risk by targeting human life and disrupting oil and gas infrastructure, making it imperative to use effective defense strategies.
Clustering on intrusions to form a group and analyzing them is an effective tool to trap an adversary and make defense recommendations. Xenotime is the only publicly known threat that has shown both the intent and capability to target human life, making them the most dangerous threat activity. Cyber-attacks on oil and gas infrastructure can destabilize a strategic regional or non-regional adversary. This attack could help state adversaries achieve political or economic goals, delay IPOs, and cause public perception issues. Attackers could also use this as training to gain combat experience. All reasonable analysis points to a state actor targeting Saudi Arabia to disrupt a portion of their oil and gas infrastructure.
The implications of the cyber-attack on industrial control systems: The Triconex attack serves as a blueprint for future attacks on industrial control systems, making it crucial for companies to have precautionary measures in place to prevent and detect these types of attacks. Publicizing good work and creating better regulations is key.
The cyber-attack on the Triconex safety controllers in Saudi Arabia is a serious concern for industrial companies worldwide. The attack reveals a blueprint for achieving future attacks, making it easier for other adversaries to carry out similar attacks. The attack on industrial control systems is not about vulnerabilities or malware, but about the future potential for attackers. Companies must prepare for this style of attack and have detective, prevention, and responsive capabilities in place. Negligence in this area can be detrimental to the safety of the community. The key is to find a balance between publicizing good work happening and keeping sensitive information private. The focus should be on creating better laws and regulations to regulate such operations.
Importance of Holding States Accountable for Cyber Attacks: Governments should conduct high-confidence assessment on cyber attacks like Triton, hold states accountable, and take these attacks off the table to prevent disruption of human life and infrastructure.
Governments should conduct a high-confidence assessment on cyber attacks like Triton, such attacks like Ukraine and NotPetya should be inexcusable. It is important to hold states accountable and take these style of attacks off the table using economic sanctions or others. Although state leaders may not find it important to know, technology clients need to know about the genesis of attacks to make informed decisions. It is embarrassing for nation's leadership not to understand technology in depth. Cyber attacks on operational technology are becoming more common, and they are purposeful and blatant attacks against civilians and infrastructure. The people who create such attacks may not comprehend the consequences of their actions and may even overlook the danger of disrupting human life. It is paramount to take action on cybercrimes.

Was this summary helpful?

0:00

So, at this point, every single one of my listeners has been the victim of some kind of data breach. Whether that's getting your personal data stolen from the Equifax breach or some other company that had info on you, but that got stolen. But how impacted are we when this happens?

0:16

At the least, you should change your passwords and tighten up your own personal security and stuff like that. But there's not much more you can do after that. So we're kind of stuck waiting for whoever stole our data to see what they do with it. And sometimes nothing happens, which is not impacted at all.

0:32

But I'm willing to bet, in the future, we'll all each be impacted by a different kind of hack, something that will certainly impact our daily lives in a major way, like one that might take out our electricity or water, or a hack that might cause a major disaster, like what if a dam got opened up and let out a bunch of water and flooded a whole city? That would have a big impact on our lives.

1:04

But these are true stories from the dark side of the internet. I'm Jack Recider. This is Darknet Diaries.

1:30

This story takes place in the Kingdom of Saudi Arabia. In the Middle East, Saudi Arabia has a massive amount of natural resources, primarily oil, which makes it a very rich country. In fact, the oil company, Saudi Aramco, is probably the most valuable company in the world because of the oil there. And in episode 30, I actually cover a hack that was done against Saudi Aramco called Shamoon. It came through and wiped out almost all the computers in the whole company. It was devastating.

1:56

But there's another massive company in Saudi Arabia. On the west coast of Saudi Arabia, something remarkable is happening. It's a petrochemical company. The world's largest integrated refinery and petrochemical single phase project. It produces 140 million barrels of products every year. It produces a wide range of high quality, high demand products.

2:18

They produce components that go into manufacturing, things we use, like clothes, to fertilizers, to packaging, to medical equipment, to electronics, to automobiles, and countless other items that make everyday life easier, safer, and more comfortable. I'm not going to say the name of the company. You can look that up yourself if you want. Where innovation, investment, and human potential are being exploited to the full to enriched life.

2:43

This chemical plant is huge. From a distance, it looks like a down-down skyline of a whole city. Huge tanks, towers, pipes going everywhere, lots of lights on at night. And each structure is a building with no walls. You can see right through it. Like, it's almost a skeleton-like, very industrial. And it's a massive plant with lots of chemicals, oil, and people all working together to make petrol-based products that you and I use. But in 2017, something big happened there.

3:14

In June 2017, a TRICONX controller shut down. Redefining process safety. These are the emergency shutdown systems. Market leading TRICONX safety systems have, for example, run for more than 600 million hours without failure on demand and are still going strong. Safety systems like this have to be extremely robust and resilient and never fail. But today, technology is only part of the safety equation.

3:42

Production complexity, aging systems, changing workforce, cybercrime and complacency are just a few of the factors introducing new threats to operational integrity. Your culture is enriched. Your people are safe. Your business is sound. Triconics process safety by Schneider Electric.

4:03

Okay, hang on a second. In order to understand what happened at this plant, we need to learn a little bit more about what OT is. You probably already know what IT is, right? Information technology. It's where computers store, manipulate, and transfer information. OT is operational technology. And this is the hardware and software that's used to control physical things in the world like valves and pumps and other machinery. Think about all the electronics that control a factory, a plant, or a utility company.

4:32

A chemical and petrol plant like this has a ton of OT systems. There are electrical devices that open valves, poor chemicals, release gases and pump fluids. But an important component of all this is the safety instrumented systems, or SIS. So many of the chemicals at the plant are toxic and must be handled very carefully. So these SIS or safety systems will monitor the environment very closely and trigger a shutdown if anything becomes dangerous.

5:02

And those safety systems that aren't responsible for conducting an emergency shutdown are the Tricon X controllers. In June 2017, something had gone terribly wrong. One of the emergency shutdown systems stopped working at malfunctioned. And when the emergency shutdown device malfunctions, then if there was a real emergency at the plant, this could result in a disaster. This is a big problem, like when the brakes go out on your car.

5:31

But when this system malfunctioned, it triggered an alert on another system, which alerted the engineers to go shut the plant down and inspect this controller. The manufacturer of the tri-connect system came out, and they examined it, but didn't find anything wrong with it. The plant was able to get back online pretty quick. And that's because they weren't looking in the right place for the problem.

5:53

Fast forward, two months. It's August 4th, 2017. It's 7.43 pm on a Friday night. 6. Of the tri-connects safety systems had malfunction and tripped an alarm. When the safety systems fail like this, it automatically causes a shutdown at the plant. Because if you don't have properly operating safety systems, you have nothing protecting you in case something goes wrong.

6:18

Those systems that had problems were in charge of issuing a shutdown if either the sulfur recovery unit or the burner management systems had detected a dangerous condition. This is a big chemical plant, and there are many technicians and engineers who work there and can troubleshoot this kind of issue, but it's 8 p.m. on a Friday night. It's the weekend.

6:39

So the crew was minimal. There's also a lot of vendors who work there who could also troubleshoot this equipment, but their staff is also minimal too because it's the night and on a weekend.

6:51

Troubleshooting began on these tri-connects systems. Logs showed that some configuration changes had been pushed to the controllers. Now, to make a change on the tri-connects controller, yeah, you need to use a computer to interact with it, but someone had to physically be present at the controller to make the change. Specifically, there's a key that needs to be inserted into the controller, and you have to turn that key to the mode program.

7:16

Once the key is in that setting, someone back in the control room can push a configuration change to that controller. Well, it just so happened that someone had left six of these controllers in the program state. That's not right. It's 8 p.m. on a Friday night. No authorized changes were approved for those controllers at that time of night. The key should not have been left on that setting.

7:40

But I guess it was just laziness on the plant operators. I mean, it takes 10 minutes to go from the control room all the way to the controller, just to put the key in and switch it to program, then you need to go all the way back to the control room, make the changes you need to make, and then when you're done, hopefully remember to go all the way back to the controller and turn the key back to the run mode. So it looks like a few of these were just accidentally left in the program state, which was bad practice.

8:05

And actually, operators had been seeing alerts on a daily basis that the key was in the wrong state, but once a day they would just clear those alerts and ignore them. I'm not sure if it was just laziness of the people monitoring the alerts or the engineers or both, because typically you don't want anyone to be able to make remote changes to these safety controllers. You want to cut these things off from the network entirely for safety reasons.

8:30

But when that key was in the program state, it meant it was now waiting for a configuration change from over the network.

8:38

But something went wrong when the config changes were pushed to these controllers. Whatever configuration was sent, it caused a failure state on the units. It didn't like whatever it was getting and caused a reboot of these systems. This is what triggered the alerts and caused the plant shutdown. This was similar to the outage two months ago, but that one was just one controller. This time it was six at the same time.

9:03

But what's more suspicious is that because this was a weekend and at night, there were no planned changes to these controllers at that time. So whatever config changes were attempted, they were completely unauthorized.

9:20

As the on-site crew investigated, they found the computer in the operations room, which was pushing these configurations. And when they investigated further, they found this computer had an unauthorized RDP session opened on it. This is really scary. To connect the dots here, some unknown person has gained remote access to a computer in the operations room.

9:43

And that computer had just pushed a config change to six of these safety systems, which caused the plant to shut down. Something very fishy was going on here. The on-site crew continued to troubleshoot for days, and even weeks, but weren't getting anywhere further with this investigation. It was just above their skill level. So they called for additional help. My name is Julian Germanus. I'm an industrial incident responder.

10:12

Julian was working as an OT incident responder in Saudi Arabia at the time. And he was told to hop on a conference call and listen to their problem to see if he had any input. The first I was told is that we needed to get it on a phone call to provide some guidance to the plants as though having mechanical problems that had resulted in a shutdown. So they only mentioned the one shutdown in August. So all we'd really heard about it was mechanical issues. They just want to have

10:38

security analyst on the phone to make sure that there's nothing wrong. Don't worry about it. It's not a big deal. Just join the call. See, at this point, the plant didn't even know if this was a security incident or a mechanical failure. But when I probed a little bit further to say, what's actually happening here? And they started saying that, well, it looks like the emergency shutdown systems have kicked in and shut the plant down and they don't know why.

11:03

And they're seeing some, like, potentially weird logins and a tapping on, like, a Friday night. And I just, I was, like, almost double take. Like, what are you talking about? This is probably the most serious thing I've ever heard about in my career. Can we get on a plane now?

11:20

Julian added everything up quickly. An unknown remote attacker had attempted to make configuration changes to an emergency shutdown system of this plant. Why would someone do that? Why would someone want to mess with the last line of defense like that? Without a properly functioning emergency shutdown system, catastrophic results could occur. So Julian immediately wanted to travel to the site

11:45

So he assembled a team. NASA is also an OT incident responder based in the kingdom of Saudi Arabia. And they called NASA up and said, hey, get ready. You're going on a trip. My bags were ready. We're used into this kind of traveling. I just picked up one of my ready bags and

12:07

This is sometimes the life of an incident responder. People have to always have a few bags ready to go at any time. Like having a 3-day go bag and a 70-go bag are suggested.

12:22

when you're dealing with big incidents like this, it's best to have someone get on site as soon as possible and help conduct the incident response and forensics. So Julian and NASA grabbed their go bags and jumped on the earliest flight to the plant. It was an overnight flight, which means they were up all night getting there. So we arrived there the next day. It was August hot. It was still early in the morning, but it was super hot. So we're waiting a line to get in through the security checkpoint and get our access granted.

12:52

So by the time we made it, the system just decided to malfunction and shut down. I guess one of the funny things we were joking about at the times when we went through the security checkpoint, it was about the time that we handed over our IDs and they started looking at who we were that the system just shut down. So we were kind of joking at the fact that the IT compromise is so bad that these guys are monitoring the security desk and blocking people from getting in.

13:19

So it was quite entertaining at the time. So the security guard could not grant us access so we waited there for another hour waiting for them to figure out how to restart the system and the grant us access. They finally got in. It's not just two of them actually. I think four of them showed up on site to help conduct this incident response. They break into two teams of two people each and start interviewing everyone just to get a lay of the land.

13:44

So I guess from the investigation standpoint, we really wanted to start at the systems or impact it. So what caused the actual shutdown was the safety controllers. So obviously the engineers have already done some reliability and some mechanical testing on the devices and pulled things like diagnostics logs and other certain artifacts from these devices.

14:04

So after analyzing the actual controllers and identifying this, we wanted to figure out what, if anything, could actually change from the controllers. And you've got to understand that these controllers aren't like Windows or Linux machines. They're embedded systems. The functionality that you can actually get from these devices is relatively limited, especially depending on the configuration.

14:25

Pulling these logs is really plugging in a serial cable and waiting five, ten minutes until it actually completes downloading the logs and things like that. It's not a basic process. The other thing you can't really do is actually pull the program back off the controllers and say this is what's on there. What you can do is you can jump onto the engineering software tri-station and issue an integrity verification.

14:50

command. So this command basically takes what program and logic files that the engineers worked on within the system and then pushes them obviously to the safety controller and does like a comparison of what's running on the controller versus what's in the system.

15:07

So what came back after that was actually a number of IO points. There was discrepancies between the IO points, which is basically the inputs and the outputs that go to the safety systems that would end up shutting down the plan.

15:20

Keep in mind, while they're down there working on these controllers, they're in the plant where the products are being created. It's loud, hot, and they have to wear safety gear. So taking a step back from that, we wanted to go, okay, well, if this is occurring, how is it occurring, who's doing this? So again, when we arrived, we really want to show whether this was an insider that was doing this. Maybe it could have been one of the operators that just gained access to the engineering workstation.

15:49

Or was this somebody coming in from the IT network? Could have been some kind of contractor. There was a number of plants, projects that were going on at the moment with different vendors and things going on. So you have potentially a number of untrusted parties wandering around that could have gained access to these systems. Realistically, at this point in time, the last thought on our mind was, this is a remote attack. We were really thinking that it could have just been either somebody messing with the systems, somebody doing something they shouldn't be doing, or a malicious internal party realistically.

16:18

So what we started doing there was really investigating the engineering workstations, which involved taking triage, artefacts from the devices, a number of images and things like that. So one of the things we were working with was pretty handy was obviously a pretty confirmed timeline. So we knew exactly when the control was shut down and resulted in the plant shutdown.

16:42

So if you're doing an investigation, this is very handy. You know that I can just focus on the lead up to this event and then really narrow down my search on what's occurring in that timeframe. I remember the process engineer was sitting next to me and I just looked at him and I was like, by any chance, do you have any kind of HP printers here? It's unusual in these environments. And he was like, no, why? And I was like, there is a folder called HP in here.

17:10

And there is a python deal. This is where it kind of clicked in my head that this is something that's going on. To be honest, the first thing I start thinking about is I'm in this plant. And initially, when we went there, we knew that it's possibly an unsafe state. But you're just sitting there in a place where you're not sure what was going on. And to be honest with you, it was scary.

17:37

It's it's not something where you know an email is gonna go down or

17:42

these systems, especially when you work in this field, this is a get drilled in your head. You're not supposed to go there until you get all these safety trainings. And one of the safety trainings that drill in your head is H2S, H2S, H2S this, H2S that, it's poisonous. And it just kind of goes to the back of your head that, you know, if something happens, you need to do this and you need to do that. And they give you the real scenarios. It's not something that, you know, it's

18:10

It's instant death in some cases. H2S is hydrogen sulfide. It's very poisonous, corrosive, and flammable. The safety controller that they are troubleshooting was part of the sulfur recovery unit. This system was in charge of shutting down the plant if there were unsafe levels of H2S detected, but this safety system itself had gone down.

18:39

So if there were unsafe levels of H2S, there was no safety system to shut things down to protect the people and the equipment in this plant. Knowing that you're in this unsafe condition, I remember I just walked outside and I was like, maybe I shouldn't be breathing this air. You're really scared. It's not an easy

19:01

Thing even I remember when I discussed it with my boss when we came back and I was like, yeah, this is a really dangerous situation

19:09

I mean, at this stage, when we're being engaged, as we mentioned, it was a couple of weeks after the actual outage had occurred. So, management's already done the difficult discussions about, do I start this plant back up, or do we need to do further investigations, or what do we do? And obviously, they come to the conclusion that leaving the plant in a downstate is extremely expensive. We've already had to pay for the outage, which is obviously a week of something to get back up and running.

19:34

So, they want to start the plant back up. And even when you've detected these kind of systemally, the malware and stuff within the plant, and you're providing reports saying you have an advanced adversary in your plant, they're going to be hesitant to even shut the plant down. So, you know how you're dealing with the hot environment when you're doing the incident response. You know that it could

19:55

there could be some pretty hairy situations if the attackers choose to do some kind of, if they're still active with any environments or if they've triggered some kind of back doors or time bombs in the environment for when the communications are severed.

20:07

Whoa, this is a lot to think about while on site. Without having any sleep the night before, but Julian and NASA had work to do still. We wanted to confirm whether or not this was an insider realistically. That was our main goal. So what we initially did is we identified, you know, we identified the malfunctioning systems, such as the controllers.

20:28

We traced it back to the engineering workstations, which then led to the investigation that found the triconics tools, the Triloc.exe and the library python files.

20:40

They figured out which of the engineering computers had that remote desktop connection to it and examined it. And they found that computer and immediately took a snapshot of that system, copying everything off it, all files, of course. But on top of that, all the event logs on that system and everything that was in memory and all running processes and all open connections to that computer. And yes, someone had access this computer remotely, but what did they do once they got in?

21:06

Julian and Nasser discovered two files on this computer that were the smoking gun, trilog.exe and library.zip. This was malware, very dangerous malware. These files were used to interact with those safety controllers. And this was the program that was used to push configuration changes to those safety systems. And inside that zip file were the binary files that were sent to the controller.

21:33

This would be extremely useful to analyze more in-depth later. But for now, they're still trying to track down who connected to this computer to put these files here. So from there, what we did was luckily able to trace a lot of the activity through the DMZ firewalls. Luckily, the plant were capturing both successful and failed connection attempts through the plant DMZ.

21:56

So leveraging these communications will help to trace a number of sessions that overlaps with artifacts being created on the engineering workstations within the system journals, the NTFS journal. So we can see the sessions coming through a DMZ choke point through the jump box and then DMZ from the perimeter VPN.

22:18

So we did track this to an external party that was logging in from the VPN through to the DMZ and then through the engineering workstation and leveraging these attack tools. The network equipment at the plant had some pretty good logging turned on, so it made it easy for them to connect the dots. The incident response team determined that someone had connected from the outside, the internet, the world, and exploited a computer inside the DMZ, a separate part of the network inside this chemical plant.

22:46

And it was supposed to be separated from the inside of the network, but the attackers found a hole in the DMZ which let them slip through into the internal network, which is how they got to those engineering workstations. And that's how they got TriLog.exe and the library.zip file onto that computer. And once the attacker was on that engineering workstation, they got a list of safety controllers and did a multicast ping on all those controllers to see if any of them were in the program state.

23:15

And that's how they found these six controllers were ready to receive a new configuration.

23:21

These two files that were on the engineering workstation had some advanced malware, something that Julian and NASA were totally blown away by, something that the makers of the tri-connects controller, Schneider Electronic, had also never seen before, and they were flabbergasted by it. Collecting these hacker tools was a fantastic find for the security teams to investigate further, but when they looked at the engineering workstation again later, these tools were suddenly gone. Somebody had deleted them. Yeah, I mean, obviously, you know, they're still active.

23:51

I mean, we kind of thought that they may have taken a break after the shutdown had occurred, and come a couple of weeks later, the toolkit's still there. It seems like they probably haven't done much, but seeing it being affiliated like that was a... I keep saying it, we got lucky. We were lucky we imaged that machine when we imaged it, and we found what was causing the outage.

24:14

This incident response team is good at industrial control systems and OT. They came in collecting enough information, and they determined the problem. So I mean, at that point, realistically, we had kind of achieved our goal. Our goal was to realistically do an initial triage and find out, is this just a malfunctioning controller, or is this something more malicious? And our consideration at that point was that we had a pretty advanced actor that was potentially interacting with the controllers.

24:44

This is obviously excluding a lot of the stuff that we did find within the environments, including other malware strains and things like that. At this point, our goal kind of shifted. Our goal wasn't now initially the incident. We had escalated to a cleanup crew, basically an external party to come in, do a full scoping exercise, and eradicate the threat from the environment. That was how we handled that. Our goal from there really shifted to the kingdom.

25:13

We ourselves had 170 plus plants that have a number of them have, you know, not our electric controllers that we needed to assess to make sure that we aren't currently being compromised or impacted. And we were protected ourselves against that state. We also looked at communicating with other potentially impacted organizations, other petrochemical facilities, other oil and gas facilities within the kingdom, because obviously it was a wide scale.

25:41

targeting campaign. It wasn't obviously just the victim that was being impacted. From there we were also, I mean, NASA was doing a huge amount of communication with the Saudi government to ensure that appropriate information was shared within the intelligence circles to be distributed to appropriate teams to make sure that they can track what's going on and be across everything that was going on. So our responsibilities didn't end at the victim and the initial triage if anything grew from there.

26:16

Julian and Nasser got out of there. A new team came in to take a look at the problems in the DMZ and the insecure engineering workstations and of course went through and made sure that none of the controllers were in that program state anymore. Also, it should never be allowed for someone to come into this network from the internet and to be able to gain control of a safety system in the plant. This is a design flaw of the network.

26:37

Those engineering workstations that had the ability to push configurations to the controllers should be totally disconnected from the network so that a remote attacker could never gain access to them. This should make it so that the only people who can make changes are the people who are on site and authorized to do so. Something big had happened here. Something extremely serious and potentially really dangerous. Why would someone hack into this place and target the emergency shutdown systems?

27:06

After the break, we'll try to unravel that mystery as best we can. FireEye is a company that is known for investigating cybersecurity threats, and FireEye was called down to clean up and investigate this problem. My name is Marina Kratafel, and I've been specializing on the security of industrial control systems for almost a decade by now.

27:29

This is Marina Krotofil. As a member of the FireEye team, Marina was investigating the incident. She knows her stuff when it comes to attacks and exploitation of embedded systems. She focused on this malware analysis. She's here to tell us what was in the public FireEye reports, as well as her independent analysis of this.

27:47

So the attackers seems to also understand overall the culture and how the plants work. So they were trying to frighten Saturday. This is a weekdays of in Saudi Arabia. And they were basically targeting their sensitive operations like injection of the implant to this weekdays of and on the later hour.

28:14

Okay, good point. They had to know this plant inside and out because let's face it, IT and OT are very different animals. A typical hacker is not going to know how to work a triconic safety system to take control of it or know how to program it. That takes a whole new level of expertise.

28:29

I remember there was one evening I so once we were still like studying the codes and you know like you're still just trying to understand what's the malware is exactly doing what are the intent you're just at the very beginning and they were like function names and one of them had like write X. So at the end it was X.

28:53

And you know, like, X for me was the first thought, what I was had in my head, it's external. So do you want to write in some external memory? Then I started talking to some guys who had access to the controller. And I received a photo of the

29:11

And when she looked at the photos of this device, she saw that the safety device also had the ability to control the valves. What this meant was if this malware was writing to the external memory, it could instruct the valves to operate in an unsafe state, which could cause damage. And at the same time, the malware could instruct the safety systems not to shut down or even create an alert.

29:37

This meant the attackers could unleash a catastrophic blow to this plant.

29:51

like I felt like I discovered so important and then later on when I like first analyzed the code I realized that this is not external but extended so if you want to write more than 20 like a large chunk of code then you would evoke specific function which allows you to write more so it's not was external but extended but at that time I saw that I would have a heart attack

30:21

they realized when the plant shut down, it was a mistake. The hackers accidentally tripped some kind of emergency shutdown system while fumbling around with these systems, which makes you wonder, what was their objective? And FireEye came up with three potential attack scenarios. Attack option one, the attackers could force this plant to shut down by triggering the emergency shutdown systems, basically a false positive. But by shutting down the plant, it could mean a financial impact to the plant.

30:49

And then there's attack option two. The attackers could reprogram the safety system so the plant could continue to operate in an unsafe state which could cause destruction to the plant or even a disaster. And then there's attack option three and this one is the most scary. The attackers could make the emergency shutdown system ignore unsafe operating levels and then somehow cause the plant to operate in an unsafe state.

31:15

So like in this scenario, the attackers might be able to control the valve for hydrogen sulfide, H2S, and somehow pump out high amounts of this dangerous gas, and then tell the emergency shutdown system to ignore the dangerous levels of H2S. If you just breathe too much of this stuff in, you can lose your sense of smell, fall unconscious, or die.

31:37

To top it off, hydrogen sulfide is extremely combustible, so one little spark, and this could cause a major explosion, which would almost certainly result in casualties. As the team at FireEye investigated this, they decided to give it a name. Since the file was called Trilog.exe, and this was targeting the TriConnect systems, they called the malware Triton.

32:01

But this malware wasn't made by someone who was sloppy or unskilled. Marina found it to be a pretty sophisticated program.

32:20

Right, so the job let's start with the job. So try to do this as such as a passive implant and why I call it passive because it does nothing. It sits in the memory, it's once you inject it in the memory, it sits in the memory and it's expect a certain packet to be activated.

32:35

This malware was very stealthy. As Marina said, it would implant itself into the memory. That is volatile memory, like RAM, where the system would reboot and it would be gone. But these safety systems would often go over 10 years without a reboot. So hiding out in the memory was fine.

32:52

Now, once it was hidden in the memory, it was designed to act normal. And engineers could interact with it just fine without knowing there were any problems with this thing. What's more is that this malware had to rewrite the firmware in order to be successful, and this was not possible to do remotely. As a user accessing it through the engineering workstation, you typically needed to bring like a flash drive to the system and then plug a console cable into it and upgrade the firmware while like physically standing next to the system.

33:18

But this malware found an unknown bug in the controller, a zero-day, which allowed it to elevate its privileges to right into the firmware of the system. So again, for someone to have such an advanced knowledge of this particular safety controller, running this particular version of software, and to be able to craft a zero-day to exploit it, this is just top-level stuff. I mean, if you think about who could have made this, it first of all had to be someone who had a lot of time, because this attack took years to execute,

33:48

And it had to be someone who has a very high skill set who can hack both IT and OT environments. And then for them to develop this malware, which they probably had full unrestricted access to these tri-connects controllers in a lab or something, so they could build this on in practice with. Basically, the attackers had unlimited resources to carry this attack out with. Okay. Why would the attacker want to get into the safety system?

34:15

Exactly. And this is where we're really getting into the large discussion also with human cost of cyber operations and ethics and so on. So safety systems are there. So even if the attacker will try to like engineer damage scenario and

34:36

executed using the main control system like DCS, really bad consequences like explosion and toxic releases will be always prevented by the safety systems. By targeting safety system and potentially preventing it from executing its function, the attacker would allow such terrible incidents like explosions and toxic releases.

35:04

And so that you would really have a cyber attack with a very dramatic physical consequences. But because people work in those plants and also even in the nights.

35:16

This may also result in casualties. So you basically deny because safety systems are meant to save life. This is the right of every plea to be saved in safe working conditions. So they specifically target systems which protect civilian people.

35:37

and this is already off limits so you know like you should not be targeting those systems in the event you do not even have like war conditions so i've been working a lot with international humanitarian

35:51

international institute for humanitarian law and with the international organization of Red Cross and all of these questions and you see like targeting civilian protection system is like not permitted and it's off limits but currently like these operations are not really

36:09

specifically regulated, and this is what actually encourage more active discussions. How should we regulate such operation on the international level? So yes, it's very upsetting that because it's such an attack may result in human casualties.

36:30

But that's also a really bad damage. So the reason why I see what they would go that once you want to take a specific refinery for a very long, like take it down for a very prolonged time, you would go for such an attack. So this would be really something very dramatic. But again, this is connected also with human casualties.

36:54

Whoa, I can't believe somebody would be insane enough to attempt something like this. This is straight up terrorism, cyber terrorism.

37:05

Now while FireEye was investigating this to try to figure out what was the purpose of this attack and how it worked and who did it, words started to get out because at this point it's months after the attack and many teams have been involved. There was the internal team and then the team Julian Nasser were on and then the Schneider Electric team and also there were other vendors on site troubleshooting this and now FireEye. And someone within all these teams started leaking information about this attack.

37:30

First, somehow the US government became aware of this. The Department of Defense began tracking this, but what also happened is that someone uploaded this malware to VirusTotal. VirusTotal is an amazing website. Anyone can upload a file to it, and when you do, it gets ran through like 70 different virus scans to see if it's known malware, and then tell you information about that. So someone uploaded these files to VirusTotal, and it just came back as unknown.

37:58

And this was probably a mistake for whoever uploaded it, because when malware like this gets uploaded to VirusTotal, the premium users of the site get to see a copy of this malware. So when it was uploaded there, it pretty much landed in the hands of all the premium users of the site. And at that point, the world was not aware of this attack. But if whoever did this attack was a premium member of VirusTotal, now they knew their cover was blown.

38:27

Another company comes into picture here, Drago's. They also investigate security threats related to industrial control systems. I sat down with their CEO to try to get to the bottom of this. So my name is Robert Lee. I'm the CEO and co-founder over at Drago's. Now, Rob, Rob, Rob used to work with the NSA before starting Drago's. That's correct. So I led the, I built and led the ICS threat discovery mission, NASA security agency.

38:52

After that, they moved me into offensive operations of the United States government. I didn't like that. I don't really have a desire to do offense. I saw a gap in the private sector around industrial security, and I saw this belief that was essentially taking IT security best practices and copy and pacing them in the ICS, not actually thinking about the difference in mission, different threats and similar.

39:18

and to be perfectly blonde with you, I would really like my son to have lights and water when he grows up. And so Adam was set to be a trying to get this right. I jumped him and created Drago's.

39:29

So there's a threat intelligence group within Drago's, which is looking at what's going on in the world to see what threats there are out there against industrial control systems. We ended up finding this malware. I don't think they found this malware through virus total, but this is a company who has their finger on the pulse for threats related to industrial control systems. And when something new like this shows up in the world, they're probably going to find it pretty quick. And so when we found it, we had never heard it before. We never seen it before. We didn't know about what it

39:58

happened in Saudi Arabia at the time. We analyzed it, and we started applying it to the set of intrusions that we were tracking. And so we've made this assessment, yep, we have enough now that this is a real set that we would be tracking. And here's this SIS or safety system target malware, and we're going to call it Traces. Yeah, okay. So they didn't know FireEye had already named this malware Triton. So Dragos called this Traces.

40:23

Just so you know, Triton and Tricist are referring to the same malware. At that point, we ended up feeling very uncomfortable about what we were looking at. And we knew very clearly just from what we could assess and do the malware analysis.

40:39

that we were looking at an adversary that was either already deploying or going to be deploying malware to target safety systems and potentially compromise even like. Now, Rob is extremely experienced on the security of industrial control systems. I know this because I actually took a class with Rob at Sands once and he just kind of blew my mind with his next level of understanding of things.

41:00

He's been involved with some of the world's biggest industrial control system hacks ever. He was there for black energy, the attack on Ukraine's power grid and has responded to hundreds of serious incidents in industrial plants, utilities, dams, you name it. But as he was understanding what he was looking at with Triton, this hit him hard like nothing else has to be extremely candid and transparent. I let out an audible fuck and lighten like sit back in my chair.

41:27

went board a glass of whiskey, sat there realizing that I had to draft this email to the Liberal Home Security, understanding what could come after if it went forward.

41:37

Rob has a history of working with the US government and feels like something like this is important enough to inform the Department of Homeland Security. That hackers somewhere in the world have broken into a chemical plant in Saudi Arabia and had the capability to cause a major terrorist attack. And sitting there reading the report of the first ever SIS targeted malware, the first time in human history that somebody

42:01

tangibly went after human life from a cyber attack and knowing what was going to happen next. It's a lot to take in because I also thought about it from the industry perspective. I thought about all of the conversations that were going to take place, the years of my life that I would then be talking about this and trying to educate groups and talk into engineering and operations, security. Those are not fun situations. I think everyone thinks these are fun situations. These are not fun situations.

42:27

Yeah, it does sound exciting to be part of this, but Rob is right. This stuff can get dark and scary real quick, and the burden it brings can really bring you down because it's so intense. And so we elected, we don't always tell governments about what we do. I think it's very important for us to try to keep our customers out of the media and out of government channels a lot of times, but we thought that this was so concerning that the US government needed to know. And so I passed the information on to the Department of Homeland Security.

42:56

and said, look, this is a very, very significant. And little did I know, I don't think they leaked it. I don't think there's any badness happening here. But there's a lot of contractors and people inside of DHS. And so one way or another, it made its way to fire. And a fire executive ended up calling me up going, hey,

43:19

We see that you're tracking this. We saw your analysis and stuff. That's great and wonderful. FYI, we're already involved. I was like, oh, okay, cool. You want to partner together on this or analyze it. And they said they couldn't, which makes sense from NDAs and similar. I said, okay, well, we're not going to publish on this. We're going to report it to our customers. But whenever you guys publish it, let us know and we'll publish our analysis as well.

43:44

And so we, I think a lot of people view cybersecurity teams always be competitive, but behind the scenes, a lot of your cybersecurity companies work together for the benefit of the community, because we all hate the adversary kind of just quickly. So anyways, FireEye ended up going forward and deciding to publish this late December. We don't, we take a stance at our firm that we never publish about threats

44:08

and their capabilities unless it's already going to be made public, because we want our customers and the community to have the information as much as possible ahead of like the New York Times articles or some more. Okay, so back to FireEye. After all, FireEye had as close to a full picture as possible with all the extra data they collected. And after analyzing the code and looking for clues and understanding its capabilities, they started to form an idea of who might be behind this.

44:33

I think Iran was initially suspected by everybody because it was a logical target, but it was quickly ruled out. I think fire has never confirmed it was Iran, but in the mass media it was

44:49

frequently speculated that it could be iron because it was a logical target, but there was no evidence and FII did not confirm that. Yes, and then there was another report that which FII has attributed activities to this National Research Institute of Mechanics of chemistry and mechanics in Moscow.

45:16

Oh, what? The Central Scientific Research Institute of Chemistry and Mechanics? Is it suspected behind this? Let me look this up.

45:26

Okay, so they're based in Moscow, Russia. But they literally seem to be a regular research institute publishing reports about thermovision, gas dynamics, high energy substances. So in my opinion, they don't sound like a hacker group who would be intent on blowing up a chemical plant in Saudi Arabia. It just doesn't make sense. But, hmm.

45:49

Wait a minute. Do you remember Stuxnet, the hack against the nuclear enrichment facility in Iran? Do you remember where we think Stuxnet was created in the Idaho National Lab, or the Oak Ridge National Lab, which are both ran by the Department of Energy and studies science and physics?

46:10

I mean, the story goes is that somebody from the NSA or CIA went to these labs to find people who were skilled enough to develop an exploit for a centrifuge. So maybe someone went to this scientific institute in Moscow to get their help in developing the OT part of this attack.

46:29

So it's not really unusual that something what is built as a lab also has a cyber capabilities. It sounds unlogical, but it does. And it shows that previously we have not really articulated this or never really look.

46:49

into the structure of such research institutions in depth. But it's not a very unusual combination. And they have a couple of departments which are related to the advanced informatics and security of critical infrastructure. And so what evidence is there to point that this research institute in Moscow may have done this?

47:16

right so fire has laid down the facts pretty well actually so this IP address which is from which they observed intrusion being conducted or at least some operations related to intrusions of the titan team like in a normal organization were conducted from that IP address there have been also that IP address was used to

47:39

money to the activities related to publications on Triton. I've also read in the FireEye report that the same IP of that Research Institute was doing reconnaissance on some other plants and was seen engaging in other suspicious activity. And also a little bit of funny, so Nick Carr was really very vocal. I've also seen this like

48:06

tweeting about this incident, and in the library.zip, there was one of the files, like the calculation of the CRC code, was written by Alexander Kotov. So they directly took that file and just used it. And then there was a block post, but this Alexander Kotov, where he described how he needed to write this file, how he developed it. And later on, when they found

48:34

this Department of Advanced Informatics from this Research Institute, they have a group photo and there is one of the members of this group, looked like this Alexander Carter. Like, it's like the idea to work there. And he posted this two pictures for the tweet from October 24, 2018, which if I look at the photos, it could be him, so it's just a fun fact.

49:03

And of course Russia has some very skilled hackers who work on behalf of the government. Hackers within the FSB, or GRU, which are intelligence agencies in Russia.

49:13

It's possible that they might have been teaming up with this research institute, which then makes this multi-disciplinary attack. I mean, it makes sense that if one team got into the plant and got access to the engineering workstation, then the engineers from the research institute could take over the keyboard and go from there. It seems like they didn't have a proper attack infrastructure in place to make sure that the attribution will never be done, including this IP address.

49:43

Which is, you see, on one hand, it makes sense to move intrusion team to the engineers. On the other hand, you're still better off to contact an operation from the established governmental institutions because you have better attack infrastructure. Maybe they need to work on that. Good point. If it was this research team, they didn't hide their tracks very well, which is something a more seasoned government hacking group would have done better at.

50:09

Now, once FireEye published a report on this, Dragos also published a report, and in their report, they didn't identify any specific group that did this, but instead they created a name for the threat actor and called them Xeno Time. When you look at what Xenotime was capable of doing, what they were, what they did is they compromised this company back in 2014, and they

50:32

beeline straight for the industrial networks. They compromised their SMS, two-factor authentication. They went directly into the industrial networks after compromising the company. After getting into the industrial networks, they went and profiled the best of our knowledge, that safety system, and then they left.

50:50

And they didn't come back until 2017 with a purpose-made capability on a highly proprietary safety system. Oh, wow. Okay, yeah. So when the attackers have the capability to spend years fine-tuning their attack, this pretty much rules out any hacktivism groups.

51:06

Simply because the sophistication here is just too high for some teenagers or ragtag group of hackers to do. And see, while trying to figure out who did it is impossible, we can take pretty good guesses at who didn't do it and try to eliminate certain groups. So next, we can try to look at this attack through the lens of a cyber criminal, someone who would be motivated by financial gain. Yeah. And so one of the things we think about with cyber crime, and again, I don't think it's fair to ever eliminate fully.

51:34

But one of the reasons, cheaply, that you would start and think it's not cyber criminal related, regardless of the sophistication and aberration, is the impact. And what were they trying to achieve? Usually you think a lot about what's the criminal aspect of this. And there was no financial motivation. There was no intellectual property. They were stealing that they could then sell off to somebody else. There was no return on investment.

51:60

to a criminal enterprise easily sussed out. You can always try to connect a million things all over there, shorting the oil market, but straight away kind of analysis, it would not rise to an assessment around this being criminal really. As you look at this case, there's not enough to support that it was hackableism. There's not enough to support that it's criminal really. And there's not enough to support that it was a terrorist action or non-state actor. The overwhelming support, the overwhelming

52:28

Um, sort of evidence, classification to our offices would be a state actor. Okay. A state actor is a group of hackers who work on behalf of a government organization. And when I think about state actors, the first group that comes to my mind is the NSA because they're totally capable of pulling something like this off. And that's what NSA stands for, right? Nation state actor. This is a good question. Like would be the NSA, which, which I, I think would, um, fail all reason that

52:57

a strong U.S. ally and the NSA are going after to cause physical events and try to kill people. It's definitely not in anything that we've ever seen them do before. But sort of, let me talk about the attribution in general, my general plot time. So a number of folks, if I were I came out, a number of folks came out and have attributed this to the Russian government. And I am not

53:23

saying that these are incompetent folks or their analysis is bad or that they're not supporting their assessments. I'm not ever trying to dismiss other people's assessments. My assessment of the situation and my knowledge of it and working with my intelligence team and some really wonderful professionals is that attribution is significantly more difficult than people make it out to be.

53:45

It's significantly easier to do than the naysayers with position, or you can't get to absolution. That's not true either. But to get to a high confidence level of attribution is incredibly difficult. And my own biases from having worked in the National Security Agency with intelligence professionals is that high confidence level of attribution isn't just related to the forensics and instant responses in intrusion or tracking adversaries or doing OSIN.

54:12

Hell for us, high confidence would have been, I've got screenshots of the person or I've got camera feeds and intrusion data and signals intelligence and maybe human intelligence. It's like so many components working together to get to a high confidence level of assessment. So a lot of the private sector, high confidence assessments I see,

54:32

really would have been low or moderate confidence assessments in the government. And I've never been able to break that. And I don't try to, again, I'm not going to downplay anybody or similar, but when you're talking about national critical infrastructure and cyber attacks upon it, which is really, really tense situation between state players.

54:52

The last thing I want to do is have my firm as an example come out and go, oh, we are basically positive at its Russia. I'm like, wow, that's going to be used diplomatically, potentially militarily. That's going to feed into broader assessments. You're going to be real careful when you're talking natural infrastructure and state tension. But the other reason we push back, there's two other reasons that we push back. The first,

55:17

is that what most people want, not all, but what most intelligence requirements in the private sector relate to is how to do better security. How do I prioritize things?

55:28

How do I look to better have security controls? What type of behaviors in the environment should I be detecting? What's my response plan B? None of those things require true attribution of it was Vladimir in Russia. That's not a valuable return on investment in trying to get the defensive recommendations.

55:49

So our customers, and largely our wider ISA security community, most of the time don't care about attribution outside of a talking point to executives. And even them, it's really just a talking point. They're not actually using that information, but it's a high cost to try to even get that information. And I would argue, you probably really can't get high compliments as often as you would like. And then the last thing, without being too worried, the last consideration around this, and again, not fair to put anybody down,

56:17

But we in InfoSec generally treat attribution as this binary thing. It was Russia or it wasn't. It was China or it wasn't. But these state players are not so black and white. Russia has a variety of intelligence

56:36

agencies and military agencies. When we say Russia, we mean SBR, we mean GRU. What elements are we talking about? Inside of that, there's the aspect that they have their own supply chain and non-state actors like hard defense industrial base that they're using. They might be having vendors of their own capabilities, maybe somebody making exploits for them.

56:57

They have allies, Russia, China, North Korea, or Iran, teaming up at any given point on different operations, just like we would do in the UK, Australia, and others. This discussion around attribution is way more nuanced at a geopolitical level than I generally see from a cyber security audience. And so to just come out, though, it's Russia, I think, is

57:21

It is not a position that I could comfortably take because of what that means and impact, what little value it has the customer and how nuanced the real answer around that solution might be. Okay, but at the same time you've identified a group called XenoTime. So how do you identify a group behind this without knowing who the group is? Yeah, great question. And so clustering on intrusions to form a group

57:51

kind of diamond analysis or pill chain analysis, you're ever going to do it, is an effective tool to trapping an adversary in the methods and tools and infrastructure they use to make those defensive recommendations. So if you're going to get to its Russia, you actually have to go through individual intrusions, you analyze and intrusion. You're probably analyzing hundreds or thousands of pieces of elements that have intrusions, if not tens of thousands, to slaping it down to a set.

58:20

And then once you have a set of intrusions and characteristics and similar, then you can start looking at victimology and infrastructure patterns and capability patterns and similar to then get to attribution. So it's actually not in the other way where you, if it's Russia and then want to follow them, you're first actually creating sets of intrusions that you then follow. And if you go and put the additional work into it, you can try to make assessments around true attribution. So you're still doing

58:49

attribution. You're attributing this intrusion or this attack that you saw to a set, but I'm not making a substance about who that set is. I'm saying if this actor, this is a xenotime, we can tell that they've targeted the other entities, we can follow them, we can track them, we can learn from them. I'm just not going that additional step to put in the analysis time and resources to try to get to true attribution.

59:12

One of the first lines in this report that I'm reading on Drago's website is Zeno Time is easily the most dangerous threat activity publicly known. Can you kind of back that up? They're the only threat publicly that we know of that has shown both the intent and the capability

59:35

to go after human life. I don't think you can measure anything else other than that. I think it's very fair to say there's threats that cause a lot of intellectual property laws and economic damage and similar, but there is nothing so sacred as human life. And for an adversary to specifically intend and be capable of targeting that, that puts them in a special league of their own of a particularly dangerous and honestly awful threat.

01:00:02

I mean, so the next question I logically have is why would somebody want to actually kill people at this plant?

01:00:11

There is a wide variety of motives that can go into it. I don't want to speculate. I'll give you some examples, but it shouldn't be seen as assessments. This is just speculation of what could happen. So first and foremost, if you are a state actor that is competitive with the oil and gas industry in Saudi Arabia, which there are numerous.

01:00:32

the loss of life in those plants could not only have an immediate impact on production, could have an immediate impact on morale of the workers and similar going back to those plants. It could have a public perception issue inside the kingdom they have to deal with. But a lot of these companies are stock owned and publicly traded. And so you have impacts on actual wealth and capitalization and future operations and similar. And so what you're basically doing

01:01:02

is with a single cyber attack, you have an ability to help destabilize a strategic regional or non-regional adversary. And so if you are a state adversary that particularly doesn't like Saudi Arabia

01:01:19

or their wealth and oil and gas. This is a very effective attack to achieve, especially because Saudi Aramco, even though they weren't the victim, was getting ready to do their IPO at the time. They ended up delaying it. We don't know if it was related to the attack, but not related. They ended up delaying their IPO until later on, and these types of attacks definitely make investors and others very, very concerned. The other aspect about it,

01:01:46

I mean, there's so many different motives. You'd have a motive of simply using this attack, even though it wasn't a training exercise, but using it as training too for your own team on, cool, can we go achieve these attacks? How could we make this scalable? What's the next level of it? You have to get combat experience, if you will, not to overplay it, but you have to get experience as the adversary being able to do these types of things.

01:02:09

All reasonable analysis points to a state actor targeting Saudi Arabia to disrupt a portion of the rolling gas infrastructure. Why they did that is a very difficult intelligence requirement to have that really is inside the realm of state intelligence agencies, not something that a private sector intelligence agency could really reasonably get to. It's like a step beyond attribution is understanding why. Is this a story that we should be freaking out about?

01:02:39

Because this could potentially target people in the US or places like that. And the whole infrastructure is like, ah, I share people's concern. And I completely find it reasonable when people are concerned. But I always try to downplay the faith of it. So what's the hype of it and what's the reality? The hype of it would be to assume that this is some highly scalable attack that immediately could target oil and gas companies or electric companies around

01:03:09

the world like all at the same time or similar. In the same way that attacks an electric system aren't hype, but thinking that there's one grid that you could take down all at once is hype. And so on this one, how seriously do I take this? I take this so seriously that when I talk to board of directors, I talk to security teams in the oil and gas industry. This is one of the first things I highlight and I tell them very clearly, if you do not have detective

01:03:37

prevention and responsive capabilities around the style of attack loop scene, not taking indicators of prices because the indicators will change, but the style of the TTP is the behavior of the attack. If you're not prepared to try to prevent the attack in response to this, you are doing the disservice to your community.

01:03:55

And what I mean by that is this is the absolute best document case we've ever had of what really could have happened from a cyber attack to lose life in the community. And if people aren't taking that seriously in these industrial operations and industrial environments, I think they're being negligent. Do I think the public should be freaking out about it? No.

01:04:21

and the work that I see out of these infrastructure companies is that so much work is happening that's not public, that they never get credit for. We comment, oh, electric utility or whatever, it's not taking security seriously. That's not true. There are some that aren't, and they need to do better for sure. But there are so much good work happening, and you just don't come out and publicize it.

01:04:42

And so we have to find a balance there. But does this attack and this adversary concern me? Absolutely. And what really concerns me is these attacks and industrial control systems aren't about the malware. It's not about the vulnerability. It's about a blueprint of how to go achieve future attacks. And so you're revealing knowledge and insight that other adversaries could pick up and use. This is how

01:05:07

the realm of only state adversary activity gets into non-state actors hands. Is once a state actor figures out how to do it and do it and publicize it, you get other people trying to do those things in the future. So the butterfly effect here is that when people start doing these types of attacks, they start to become more common. They start to become easier. And we want to prevent that because these are particularly damaging style of attacks.

01:05:36

For me at least, this whole attack puts me in deep thought. There are hundreds of industrial plants around Saudi Arabia and the world that have these same tri-connect safety controllers. And it sounds like these hackers were in the network for years before accidentally tripping an alarm. So it just makes me wonder how many other industrial networks might these attackers be in right now, lying in weights, waiting for the need to pull the trigger.

01:06:04

And it also makes me wonder how many other plants might have had a mysterious shutdown and didn't have the capability or care to look deeper for this malware, and instead they just started the plant back up.

01:06:17

spooky stuff. On one hand, I want to know more, but on the other hand, I'm kind of afraid to look. Sometimes you have to let it go because it consumes you so much, you know, that, yeah, sometimes you have to let it go. And that's what exactly what I did with Stratton. I don't think about this anymore, so I'm more concentrated right now and working with, for example,

01:06:37

Red Cross and with people who involved in social humanitarian law, so that I'm helping them with my technical knowledge, with my technical inputs to explain them the possible consequences of such attacks and cyber operations on the critical infrastructure, so that they could create better laws and regulate how to regulate such operations. So this is my main focus right now.

01:06:58

Yeah, I think when we look at the attribution side of it, where I will say the private sector may not need to go the distance and try to come up with a high economic assessment, I do think government should. So is it important for clients of, you know, Dregus' technology to know that Russia did this? No. But if Russia did do it, then the US and others do actually need to know that. And it does need a way into discussions between states. It could lay way into economic sanctions or others, like

01:07:28

This attack was a very purposeful and blatant attack against civilians and civilian infrastructure. And state leaders around the world need to take this attack, attacks like Ukraine, attack like not Petra, and actually

01:07:46

take these style of attacks off the table and penalize the states that do these types of facts. They should be inexcusable. So whereas on the attribution subject, I don't want to go the distance because I don't see the value in trying to pin it to any given state. The various intelligence agencies around the world need to and they need to get it right and there needs to be action followed through.

01:08:08

And I've seen the way our nation's leadership interviews people like Mark Zuckerberg. Our leaders simply don't understand technology enough to know what to do about this. And it's embarrassing. Technology defines our current time. There's no excuse for our leaders to not understand technology more in depth at this point.

01:08:26

Maybe this was all just a test, or practice. Since the attackers didn't actually cause damage to the plant other than an accidental shutdown, because I wonder about the people who were behind this. Did they know this was a mission to kill people? Or were they told this was just a test and that no human lives would be lost during this test?

01:08:47

When you look at the code long enough, the malware, you start to really think about that person who wrote it because it was a human who typed out that code. Marina thinks a lot about whatever person wrote this malware. Oh, because I spent so much time with his activities and because, you know, it's a very typical research, intensive research work.

01:09:15

that, to which I can relate. And I actually talked to many guys about that, you know, like everybody who is investigating an incident and spend a lot of time, you know, like you start already seeing the incident and can feel more that person, the pain, the frustration, you know, but they sometimes also kind of want to see the person.

01:09:37

Yeah, and I think it's probably my personal opinion. They probably even did not really clearly understood the consequences and what exactly they are doing. Or maybe, as you say, if it was a gesture test and they knew they'd never go into disrupt anything, though they did not feel like they were doing really something dangerous because I would not be ever comfortable to conduct an operation which may impact life of civilians.

01:10:06

There is a lot to think about regarding this incident. These kind of attacks on operational technology are slowly becoming more common. We've seen Stuxnet try to disable a nuclear enrichment facility and we've seen attacks on the Ukraine's energy grid. And now we see Triton going after the emergency shutdown systems of a chemical plant.

01:10:28

It's chilling, for sure. And I just hope that whoever created this is not crazy enough to intentionally cause a disaster.

01:10:38

There is an update to this story. On October 23, 2020, the US published an article saying, quote, today the Department of the Treasury's Office of Foreign Assets Control, OFAC, designated a Russian government research institution that is connected to the destructive Triton malware. On August 2017, a petrochemical facility in the Middle East was the target of a cyber attack involving the Triton malware.

01:11:05

This cyber attack was supported by the Central Scientific Research Institute of Chemistry and Mechanics, a Russian government-controlled research institution that is responsible for building customized tools that enabled the attack. In 2019, the attackers behind the trident malware were also reported to be scanning and probing at least 20 electrical utilities in the United States for vulnerabilities.

01:11:28

As a result of today's designation, all property and interests in property of the research institution that are in or come within the possession of US persons are blocked, and US persons are generally prohibited from engaging in transactions with them. Additionally, any entities 50% or more owned by one or more designated persons are also blocked.

01:11:50

Moreover, none US persons who engage in certain transactions with the research institution may themselves be exposed to sanctions." End quote. So there you go. But while this is still an allegation that this research institution was behind this, and nothing has been actually proven in court, it seems like Marina and the team at FireEye were right to have blamed them.

01:12:11

And now with these sanctions in place, I hope it's enough to deter them from doing any further dangerous activities like this.

01:12:27

A big thank you to our guests for coming on the show and sharing the story with us. Julian and Nasser's initial investigation was pivotal to everything that followed, and both of them now work for Drago's with Rob. Marina Crota Phil's research on the team at FireEye was eye-opening to the world, and Rob Lee's report really does have an impact and hopefully saves lives in the future. Keep up the great work on helping us stay safe from major catastrophic events like this.

01:12:52

The show was created by me, The Crimson Bear, Jack Recider, original music created by The Salty Jackal, Garrett Tiedemann, editing help this episode by the Stardust kitten Damien, and our theme music is by the Sonic Panda Breakmaster cylinder. And even though when my dad has a computer problem and he calls me up to help him, I remind him about how he used to nag on me to get off the computer when I was in high school. And if I did, wouldn't be able to help him now, this is Dark Knight Diaries.

Was this transcript helpful?

Recent Episodes

71: Information Monopoly

Darknet Diaries

The hosts recreate one of the greatest hacks in history to inject information into North Korea's totalitarian regime system. Guests Yeonmi Park and Alex Gladstein share their stories about this hack.

August 04, 2020

70: Ghost Exodus

Darknet Diaries

Guest storyteller Ghost Exodus describes his experiences as a hacker who documents online activities on YouTube. IT Pro TV is thanked for sponsoring this episode and offering free training with promo code DARKNET25.

July 21, 2020

69: Human Hacker

Darknet Diaries

Security consultant Christopher Hadnagy shares his experiences as a social engineer before the term existed, discussing techniques to manipulate humans with their stories from his long career in penetration testing.

July 07, 2020

67: The Big House

Darknet Diaries

John Strand is a penetration tester. He’s paid to break into computer networks and buildings to test their security. In this episode we listen to stories he has from doing this type of work. Thanks to John Strand for coming on the show and telling your story. Sponsors Support for this episode comes from LastPass. LastPass is a great password manager but it can do so much more. It can setup 2FA for your company, or use it to monitor what your users are doing in the network. Visit LastPass.com/Darknet to start your 14 day free trial. Support for this episode comes from Blinkist. They offer thousands of condensed non-fiction books, so you can get through books in about 15 minutes. Check out Blinkist.com/DARKNET to start your 7 day free trial and get 25% off when you sign up. Sources How a Hacker’s Mom Broke Into a Prison—and the Warden’s Computer Video: How not to suck at pen testing John Strand Video: I Had My Mom Break Into Prison Learn more about your ad choices. Visit podcastchoices.com/adchoices

June 09, 2020

Related Episodes

Ep 30: Shamoon

Darknet Diaries

In 2012, Saudi Aramco was hit by a devastating virus that destroyed thousands of computers. Recovering efforts were made possible thanks to Chris Kubecka's guest appearance in an unknown podcast.

January 22, 2019

57: MS08-067

Darknet Diaries

Hear what goes on internally when Microsoft discovers a major vulnerability within Windows. Guest Thanks to John Lambert for sharing this story with us. Sponsors Support for this episode comes from ProCircular. Use the team at ProCircular to conduct security assessments, penetration testing, SIEM monitoring, help with patches, or do incident response. Visit www.procircular.com/ to learn more. This episode was sponsored by IT Pro TV. Get 65 hours of free training by visiting ITPro.tv/darknet. And use promo code DARKNET25. Support for this episode comes from Blinkist. They offer thousands of condensed non-fiction books, so you can get through books in about 15 minutes. Check out Blinkist.com/DARKNET to start your 7 day free trial and get 25% off when you sign up. Sources https://blogs.technet.microsoft.com/johnla/2015/09/26/the-inside-story-behind-ms08-067/ https://www.justice.gov/opa/pr/payment-processor-scareware-cybercrime-ring-sentenced-48-months-prison https://www.nytimes.com/2019/06/29/opinion/sunday/conficker-worm-ukraine.html https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-0601 https://www.wired.com/story/nsa-windows-10-vulnerability-disclosure/ Book: Worm Attribution Darknet Diaries is created by Jack Rhysider. Episode artwork by odibagas. Theme music created by Breakmaster Cylinder. Theme song available for listen and download at bandcamp. Or listen to it on Spotify. Learn more about your ad choices. Visit podcastchoices.com/adchoices

January 21, 2020

150: mobman 2

Darknet Diaries

Re-examining Episode 20 of Darknet Diaries, exploring doubts about Greg aka 'mobman' claiming to create sub7 malware.

October 01, 2024

135: The D.R. Incident

Darknet Diaries

Omar Avilez worked on a cyber security incident response team when his country experienced a major ransomware attack; he shares the procedures they went through. Breakmaster Cylinder's new album is out.

July 04, 2023

Ask this episodeAI Anything

Hi! You're chatting with Darknet Diaries AI.

I can answer your questions from this episode and play episode clips relevant to your question.

You can ask a direct question or get started with below questions -

What are the consequences of data breaches in daily life?

How did the Triconex system failure impact Saudi Arabia's plant?

What role do incident responders play in cybersecurity?

What measures can governments take against cyber attacks?

Why is operational technology targeted by cybercriminals?