The time I fought off a hacker ... live

You see it in TV shows and movies where someone is fighting a hacker. They go back and forth, turning each other's lights off, maybe trade witty banter as the camera goes back and forth, and it's funny. But in the real world, it's rare that you are there as the hacker is there and get a chance to actively fight them. One time in my career, I did. Here's the story.

Many years ago, I was working a helpdesk and pseudo-information security job. I say pseudo-information security, because I was self-taught and had no real experience. My responsibilities included removing viruses from computers and eventually that morphed in to more security-specific things. This was one of my first "incident response" tasks at this job. I did a pretty bad job at it looking back, but I eventually found what was going on.

I was working on my regular day-to-day duties doing helpdesk things when I was told a server was acting weird when trying to RDP in. "RDP in" is slang for logging in to the desktop using remote desktop. This server was at a remote location, so I had to drive to it. I hopped in my car and drove to the location. I was told I couldn't take the server offline.

When I got to the server room, I sat down at the Windows server with a keyboard and mouse. It was an Active Directory server. It served that site, and while it being offline wouldn't stop logins, it would make them slower as requests would have to round-trip several physical miles, across a line that was much slower than even today's DSL. Obviously not ideal, especially for people who were experiencing slow login times as it was.

Active Directory, or AD, is a service that is part of Windows Server and enables organizations to have a centralized login service. AD is built on LDAP, so you'll find similar base schemas to LDAP in AD, but extends LDAP and integrates kerberos for authentication against this store of users and passwords. The schema can be extended to support Exchange and other services. At one point, I was working on integrating fingerprint authentication with AD. We used AD to manage our desktops and allow employees to log in. An Active Directory server acting up would slow things down when employees try to login.

Clear as mud? Let me try to simplify that. Active Directory is a part of Windows that a lot of organizations use so that employees have one password regardless of which computer they go to. It can also be the same password for email. It's a really powerful service.

I'm at the server. I notice that the anti-virus agent is disabled. Odd, but I'm not immediately concerned. If something crashed, that'd explain the slowness and could explain the disabled anti-virus agent. I re-enabled the AV. It almost immediately disabled. I'm raising an eyebrow now. Maybe it's malware? Some malware was known to disable AV agents. But if it was malware, how'd it get there? I open task manager to see what's running. Nothing out of the norm, but CPU usage is definitely pegged at 99%. I note it and keep looking around. I try to enable the AV agent a couple times more, and it gets disabled each time.

Having 99% CPU utilization is bad. That's why things are slow and acting weird when trying to RDP in to it. But what's causing it? I open Internet Explorer. I don't really know why I did, but I did. Nothing strange there for settings. I opened Windows Explorer and it closed. It closed. I open it again in case I accidentally clicked the X with the mouse button or something. It closed again. Windows Explorer doesn't close by itself. That is not normal behavior. Now I know for sure something is up.

I call my boss to let him know what I have at this point: yes, it is more than just a crashed processing taking all the CPU. After a few minutes getting my thoughts together about what I just saw, I get a plan for how I'm going to track this down. We have anti-virus that keeps disabling itself, we have Windows Explorer mysteriously closing itself, and we have 99% CPU usage.

Back to that 99% CPU usage. I opened task manager again and ordered the process list so that the highest process usage was first. I mentally added up the usage and nothing added up to 99%. How could that be? Down in the bottom of task manager it clearly says 99% but no processes add up to 99%. So what's using the CPU so heavily? I closed task manager and opened it again. I added them up again. Nothing added up to the 99% usage I was seeing in the bottom corner of task manager. My eyes got wide and my mouth was probably hanging open - this was a rootkit.

If we are being dramatic here, the room seemed to close in and all I saw was the screen. I sat there, stunned. I could hear the clock on the wall tick the seconds by and for what seemed like forever while also being an instant, I snapped back.

Hey, I said being dramatic.

This was a rootkit.

But I did sit there stunned. I stared at the screen as my mind raced. How did this happen? How long? How do I find the rootkit? I wasn't allowed to take the server offline, so I did what I thought was the next best thing and put it in safe mode with networking. This would allow things to function while I tried to see if this rootkit loaded in safe mode so that I could try to find it. What I had thought was something disabling AV was someone. And I made them angry.

A rootkit is software that hides things. It can hide a backdoor, it can hide other tools. It typically doesn't do anything other than hide other software. So let's sum up what I have so far:

• AV gets disabled
• CPU usage is constantly at 99%
• Windows and browsers get closed
• This appears to be a rootkit as CPU usage doesn't add up

Clearly someone is in this server and actively doing things. I have no idea yet how, how long, or what they've done.

And oh did I make them angry. This is where I made a big mistake. When you reboot in to safe mode with networking, things still talk to the internet. The attacker was still there, still doing things. I'd open a window, and it'd close. I'd open a web browser, and it'd close. I was fighting against someone who was actively on my server. It got more aggressive while networking was on in safe mode. So I took the server off the network. I had no choice.

After taking the server off the network, I dug through the process list in task manager more closely. I don't remember the name of the process, but I found one process that didn't look normal. It had maybe 1-2% process usage, minimal RAM usage, but the name of the process didn't look normal. I had an explorer window open on C:\. When I killed that process, another folder appeared. In this folder were several files. I grabbed them off the server, deleted the folder, and brought the networking back up on the server. I watched things as best as I could and process usage sat around normal. AV worked again. Windows weren't closed. Phew.

I still had no clue what the attacker did on the server, and taking the server offline to image it was a non-starter. I grabbed the USB drive that had the folder of the tools the attacker left and headed back to the main office. When I got there, I explained what happened. The server was still acting normally so it seemed to be okay now. I uploaded what I had to VirusTotal and nothing got detected. That's ... not good.

It wouldn't be for a few months that a new zero day would get announced with new malware that had rootkit capabilities. I reuploaded the files to VirusTotal and all the engines at the time flagged it as this new malware. I don't know if we were the first or one of the first, but that was a wild ride. Looking back, I'd certainly do things differently, but it worked out.

And that's the time I fought a hacker, live.