Texas Cyber Summit - OpenSOC After Action Report

I recently had an opportunity to attend the second annual Texas Cyber Summit from the 10th through the 12th. In preparing for the conference I noticed that the OpenSOC Network Defense Capture-the-Flag was going to be running. While I had not had any direct experience with the OpenSOC CTF I was familiar with it after BSidesSATX 2018 hosted them. That year was my first information security conference and I wanted to hit the gamut of talks so I made the decision to skip checking out the CTF. Shortly after the conference, I learned just what the CTF was all about from their website and regretted not devoting some of my time to it. This time around, my schedule for the Summit was already filled in with all the talks I wanted to sit in on but I knew I had to at least sit down and try the OpenSOC event when it opened. A few hours in and I knew there was no way I was going to leave…

Background

OpenSOC is put on by the great people at Recon InfoSec. For those who aren’t familiar with OpenSOC it is the brain child of Recon’s CTO Eric Capuano who saw a need for more realistic blue team training and exercises to prepare participants to defend their organizations. The OpenSOC team have the ability to customize the scenarios and build them out in a way that replicates the tactics, techniques and procedures (TTPs) of known threat groups.

This is the biggest advantage that OpenSOC has over any other CTF I have participated in: the attention to detail in deploying an environment where a participant can build and hone the skills needed at their job. Sure, playing with steganography, cryptography, doing some SQL injection or whatever else found in most offensive security based CTFs is fun, but does the average attendee get to use any of that knowledge where it matters most? If someone is a pentester or red teamer they might, however, the vast majority of cybersecurity professionals are serving in some form of blue team role. How many of the blue teamers can say they ever needed the knowledge of that niche, exotic stegonagraphic technique they learned about at a CTF? Conversely, would it benefit them to know what it looks like if APT34 gains access to and moves laterally in their environments? Would they know what to look for? Would they know the best way to respond?

That’s where OpenSOC comes in:

OpenSOC is a Digital Forensics, Incident Response (DFIR), and Threat Hunting challenge meant to teach and test practical incident response skills in an environment that’s as close to “the real thing” as it gets. This isn’t just another CTF. We’ve built this platform to train real-world responders to handle real-world situations.

How It Works

OpenSOC is a slightly unusual format from CTFs most people have likely participated in. Questions are asked in sequential order regarding a specific “intrusion”. Most, but not all, of the scenarios worked their way through the entire cyber kill chain of initial compromise, post-exploitation enumeration, lateral movement, and data exfiltration. The questions were generally targeted at the key details of the intrusion:what user account was compromised, what directory some malware was running out of, etc. In general, the scenarios started easy and got progressively harder. If a participant hits a wall and is having a hard time answering the question they may have the option to take a hint and lose points in exchange for being pointed in the right direction.

My team only needed a hint once and unfortunately the hint wasn’t specific enough to help us (luckily, we figured out the answer anyway). I really like this format, regardless of how frustrating it is to get stuck, because it mimics real life. If a threat hunter or incident responder is working a real life issue they don’t get a “skip” button, they have to work with what they know which is often far less than they would prefer.

Tools Don’t Make the Security Professional

A lot of security professionals get wrapped up in the tools they use. Endgame vs Carbon Black, Elastic vs Splunk, etc. Someone that wants to be a flexible and well-rounded professionals needs to be more concerned with the concepts behind employing their tools rather than the tools themselves. Speaking for my team’s experience, we had never used ANY of the tools in OpenSOC and we still managed to succeed. Just because OpenSOC chooses certain tools doesn’t mean someone that works with a different tool set will get little out of the experience. There will undoubtedly be a learning curve but once someone has gotten the basics down the methodology of employment should be the same.

Specifically, OpenSOC competitors rely primarily on Moloch for PCAP indexing and searching, Graylog for host log aggregation, and Kolide/OSQuery for querying information from the live hosts. Within the first few hours we had nailed down how to work with these tools given to us: Wireshark-like queries in Moloch, Lucene syntax for Graylog, and SQL-like statements with Kolide. Kolide was by far the software my team struggled with the most, though. Using SQL-like statements against the hosts was not something we were accustomed to and I’m confident we knocked over a host or two with some overly broad queries. Ultimately, we knew what information we wanted from Kolide and how to leverage it we just weren’t comfortable making it do what we wanted.

Dave Kennedy gave the keynote on the first day of the Summit and said that threat hunting is an art. Artists tend to have their “thing”; for some it might be oil painting and for others it might be sculpting. Sit them down with a different set of tools they aren’t comfortable with and tell them to make art and it’s probably going to come out pretty horrendously. Eventually, though, the artistic mind is what matters most and they will probably be able to deliver something worthwhile. I’m being fairly reductive to get the point across: as security professionals we are “artists” and we have a preference in tools but it’s our “artistic minds” that matter most. Hone that mind and the rest doesn’t matter nearly as much.

Red Team Perspective

My buddy and teammate Sam gave his perspective as a Red Teamer on participating in the OpenSOC CTF. I would be remiss if I didn’t give him a special shoutout for going outside his comfort zone and helping us crush it. Checkout his writeup at his blog.

Did I mention he did all of this while also giving his own presentation?

Conclusion

If it isn’t already plainly obvious I will admit I am gushing over OpenSOC. I had not had that much fun doing a CTF in a long time. While I was completely drained at the end of each day the sense of accomplishment kept me motivated for the next day. Having done Threat Hunting for the Air Force it was a big confidence boost to validate my abilities and knowledge in a controlled environment against my peers.

As such, that brings the conversation back to the tagline of this post, training like you fight. This is a mantra often repeated in the military or competitive sports to represent the fact that the best way to prepare for the real thing is to treat every practice as if it was real. In a lot of ways cybersecurity is a competitive sport. As professionals we are competing against adversaries, whether real or emulated, to secure our organizations. If an organization’s idea of training is to pit their people against the little league baseball team down the road but their real competition is the St. Louis Cardinals can anyone honestly be surprised when they lose? Why, then, do we expect information security professionals to “train” on the humdrum of false positives and end users with too much time on their hands? OpenSOC helps fill that void and gives a bored detection team the shot in the arm to reinvigorate their love of the game, so to speak.

Additionally, I believe we need more organizations and people providing these services and experiences. There is too much demand for this quality and style of event to leave it on the shoulders of one organization. It might be too cliche to say but a rising tide really does lift all boats. The more trained and versed we as blue team security professionals are in identifying and responding to real world events the better the industry and our respective organizations will be. My hope is that the success and recognition that Recon InfoSec and OpenSOC has garnered will spur other motivated, community-oriented individuals and entities to enter this space and continue to grow and evolve this type of training a la CTF.

Credits

Thanks to Eric Capuano, Whitney Champion, @JoshOps, and the rest of the Recon Team that I am undoubtedly missing. As someone that administers a humble homelab I can only imagine the missed meals, late night coffee and even later night stiff drink it takes to make this juggernaut run.

Thank you to the Texas Cyber Summit for hosting OpenSOC and succesfully putting the physical infrastructure in place to let over eighty people hammer at this CTF in addition to the other competitions and events taking place simultaneously.

Finally, a giant thank you to my team: Sam, Ashley, Joe and Christian! Without them there is no way I would have been as successful in this CTF. I tried not to make this post too self serving but if it wasn’t obvous my team was able to secure first place in the CTF and ultimately earn two Texas Cyber Summit “Black Badges” for our efforts. At the end of the day, it wasn’t about winning for us but about proving to ourselves that we had the chops to hold our own in the subset of cybersecurity that we work in.