Tag Archives: architecture

Fixing a Security Program

I’m still unsettled by how many security programs are so fundamentally broken. Even those managed and staffed by people with impressive credentials. But when I talk to some of these individuals, I discover the key issue. Many seem to think the root cause is bad tools. This is like believing the only thing keeping you from writing the Next Great American novel is that you don’t have John Steinbeck’s pen or Dorothy Parker’s typewriter.

In reality, most of the problems found in security programs are caused inferior processes, inadequate policies, non-existent documentation  and insufficient standards. If buying the best tools actually fixed security problems, wouldn’t we already be done? The truth is that too many employed in this field are in love with the mystique of security work. They don’t understand the business side, the drudgery, the grunt work necessary to build a successful program.

For those people, here’s my simple guide.  I’ve broken it down to the following essential tasks:

  1. Find your crap. Everything. Inventory and categorize your organization’s physical and digital assets according to risk. If you don’t have classification standards, then you must create them.
  2. Document your crap. Build run books. Make sure you have diagrams of networks and distributed applications. Create procedure documents such as IR plans. Establish SLOs and KPIs. Create policies and procedures governing the management of your digital assets.
  3. Assess your crap. Examine current state, identify any issues with the deployment or limitations with the product(s). Determine the actual requirements and analyze whether or not the tool actually meets the organization’s needs. This step can be interesting or depressing, depending upon whether or not you’re responsible for the next step.
  4. Fix your crap. Make changes to follow “best practices.” Work with vendors to understand the level-of-effort involved in configuring their products to better meet your needs. The temptation will be to replace the broken tools, but these aren’t $5 screwdrivers. Your organization made a significant investment of time and money and if you want to skip this step by replacing a tool, be prepared to provide facts and figures to back up your recommendation. Only after you’ve done this, can you go to step 6.
  5. Monitor your crap. If someone else knows your crap is down or compromised before you do, then you’ve failed. The goal isn’t to be the Oracle of Delphi or some fully omniscient being, but simply more proactive. And you don’t need to have all the logs. Identify the logs that are critical and relevant and start there: Active Directory, firewalls, VPN, IDS/IPS.
  6. Replace the crap that doesn’t work. But don’t make the same mistakes. Identify requirements, design the solution carefully, build out a test environment. Make sure to involve necessary stakeholders. And don’t waste time arguing about frameworks, just use an organized method and document what you do.

Now you have the foundation of any decent information security program. This process isn’t easy and it’s definitely not very sexy. But it will be more effective for your organization than installing new tools every 12 months.

 

Tagged , , , , , , , ,

The Question of Technical Debt

Not too long ago, I came across an interesting blog post by the former CTO of Etsy, Kellan Elliott-McCrea, which made me rethink my understanding and approach to the concept of technical debt. In it, he opined that technical debt doesn’t really exist and it’s an overused term. While specifically referencing code in his discussion, he makes some valid points that can be applied to information security and IT infrastructure.

In the post, he credits Peter Norvig with the quote, “All code is liability.” This echoes Nicholas Carr’s belief in the increased risk that arises from infrastructure technology due to the decreased advantage as it becomes more pervasive and non-proprietary.

When a resource becomes essential to competition but inconsequential to strategy, the risks it creates become more important than the advantages it provides. Think of electricity. Today, no company builds its business strategy around its electricity usage, but even a brief lapse in supply can be devastating…..

Over the years, I’ve collected a fair amount of “war stories” about less than optimal application deployments and infrastructure configurations. Too often, I’ve seen things that make me want to curl up in a fetal position beneath my desk. Web developers failing to close connections or set timeouts to back-end databases, causing horrible latency. STP misconfigurations resulting in network core meltdowns. Data centers built under bathrooms or network hub sites using window unit air conditioners. Critical production equipment that’s end-of-life or not even under support. But is this really technical debt or just the way of doing business in our modern world?

Life is messy and always a “development” project. Maybe the main reason DevOps has gathered such momentum in the IT world is because it reflects the constantly evolving, always shifting, nature of existence. In the real world, there is no greenfield. Every enterprise struggles to find the time and resources for ongoing maintenance, upgrades and improvements. As Elliott-McCrea so beautifully expresses, maybe our need to label this state of affairs as atypical is a cop-out. By turning this daily challenge into something momentous, we make it worse. We accuse the previous leadership and engineering staff  of incompetence. We come to believe that the problem will be fully eradicated through the addition of the latest miracle product. Or we invite some high-priced process junkies in to provide recommendations which often result in inertia.

We end up pathologizing something which is normal, often casting an earlier team as bumbling. A characterization that easily returns to haunt us.

When we take it a step further and turn these conflations into a judgement on the intellect, professionalism, and hygiene of whomever came before us we inure ourselves to the lessons those people learned. Quickly we find ourselves in a situation where we’re undertaking major engineering projects without having correctly diagnosed what caused the issues we’re trying to solve (making recapitulating those issues likely) and having discarded the iteratively won knowledge that had allowed our organization to survive to date.

Maybe it’s time to drop the “blame game” by information security teams when evaluating our infrastructures and applications. Stop crying about technical debt, because there are legitimate explanations for the technology decisions made in our organizations and it’s generally not because someone was inept. We need to realize that IT environments aren’t static and will always be changing and growing. We must transform with them.

Tagged , , , , , ,

Why You’re Probably Not Ready for SDN

While it may seem as though I spend all my time inventing witty vendor snark to post in social media,  it doesn’t pay the bills. So I have a day-job as a Sr. Security Architect. But after coming up through the ranks in IT infrastructure, I often consider myself “architect first, security second.” I’m that rare thing,  an IT generalist. I actually spend quite a bit of time trying to stay current on all technology and SDN is one of many topics of interest for me. Especially since vendors are now trying to spin it as a security solution.

Software-defined networking (SDN) is still discussed as if it’s the secret sauce of the Internet. This despite Gartner placing it at the bottom of its Networking Hype Cycle due to “SDN fatigue” and the technology’s failure, thus far, to gain much traction in the enterprise.

 However, the magical SDN unicorn still manages to rear its head in strategy meetings under the new guise of hyper-convergence and the software-defined data center (SDDC). This is probably due to IT leadership’s continued yearning for cost savings, improved security and the achievement of a truly agile organization. But is SDN, with its added complexity and startling licensing costs, really the answer?
You can read the rest of the article here. And yes, there’s a registration wall.
Tagged , , , , , , ,

Security Vs. Virtualization

Recently, I wrote an article for Network Computing about the challenges of achieving visibility in the virtualized data center.

Security professionals crave data. Application logs, packet captures, audit trails; we’re vampiric in our need for information about what’s occurring in our organizations. Seeking omniscience, we believe that more data will help us detect intrusions, feeding the inner control freak that still believes prevention is within our grasp. We want to see it all, but the ugly reality is that most of us fight the feeling that we’re flying blind.

In the past, we begged for SPAN ports from the network team, frustrated with packet loss. Then we bought expensive security appliances that used prevention techniques and promised line-rate performance, but were often disappointed when they didn’t “fail open,” creating additional points of failure and impacting our credibility with infrastructure teams.

So we upgraded to taps and network packet brokers, hoping this would offer increased flexibility and insight for our security tools while easing fears of unplanned outages. We even created full network visibility layers in the infrastructure, thinking we were finally ahead of the game.

Then we came face-to-face with the nemesis of visibility: virtualization. It became clear that security architecture would need to evolve in order to keep up.

You can read and comment on the rest of the article here.

Tagged , , , , , , ,

Is Your Security Architecture Default-Open or Default-Closed?

One of the most significant failures I see in organizations is an essential misalignment between Operations and Security over the default network state. Is it default-open or default-closed? And I’m talking about more than the configuration of fail-open or fail-closed on your security controls.

Every organization must make a philosophical choice regarding its default security state and the risk it’s willing to accept. For example, you may want to take a draconian approach, i.e. shooting first, asking questions later. This means you generally validate an event as benign before resuming normal operations after receiving notification of an incident.

But what if the security control detecting the incident negatively impacts operations through enforcement? If your business uptime is too critical to risk unnecessary outages, you may decide to continue operating until a determination is made that an event is actually malicious.

Both choices can be valid, depending upon your risk appetite. But you must make a choice, socializing that decision within your organization. Otherwise, you’re left with confusion and conflict over how to proceed during an incident.

baby_meme

Tagged , , , , ,

Being the Security Asshole

Yes, I have become a security asshole. The one who says “no” to a technology. But I say it because of risk, and not just security risk.  And I’m angry, because my “no” is a last resort after many struggles with developer, engineering and operations teams in organizations that struggle to get the basics right.

I try to work with teams to build a design. I bring my own architecture documents and diagrams, which include Powerpoint presentations with talking points. I create strategy road maps explaining my vision for the security architecture in an organization. I detail our team’s progress and explain how we want to align with the rest of enterprise strategy and architecture. I stress that our team exists to support the business.

What do I get in return? Diagrams so crude, they could be drawn in crayon or made with Legos. They usually don’t even have IP addresses or port numbers. I have to argue with sysadmins about whether Telnet is still an acceptable protocol in 2015. I’m subjected to rehashed Kool-Aid about how some product is going to rescue the organization even though I found significant vulnerabilities during the assessment, which the vendor doesn’t want to fix.

And if this means that you hate me, fine. I’ll be the asshole. I’ll embrace it. But at least I can have a clear conscience, because I’ve done my best to safeguard the organization that’s paying me.

Tagged , , , , ,

Tootsie Roll Pop Security

Recently, it occurred to me that the security of most organizations is like a Tootsie Roll Pop. Hard and crunchy on the outside, soft and chewy on this inside. One bite and you easily get to the yummy center.

How many licks does it take to get to the crown jewels of your organization: your data?

Tagged , , , ,

The Security Policy’s Bad Reputation

I had a disturbing conversation with a colleague last night. He told me that he didn’t believe in compliance-only, checkbox security, so why should he waste time on policies and standards? I almost blew a gasket, but because he’s pretty junior, I thought it best to educate him. The following is a summary of what I told him.

Security policies and standards are a foundational set of requirements for your engineering, development and operations teams. Without these boundaries, the entire IT organization floats aimlessly, buying solutions and implementing controls without rhyme or reason. Generally, only oblivious technologists design solutions without referencing policies and most engineers are begging for this guidance from their security teams.  Engineers aren’t mind readers, they just want us to tell them what we want: in writing.  Without policies and standards, the result is reactive inefficiency, because the security team becomes a chokepoint for every implementation.

Security policies help keep organizations ahead of the risk curve. It means that risk has been evaluated to some degree and a decision made (by someone) regarding the level an organization is willing to accept. Any security organization that wants to achieve some level of maturity will spend the cycles to develop its policies or suffer the consequences.

Developing policies and standards isn’t an easy process. Often the right stakeholders haven’t participated in the discussion, the documents are badly written, outdated or compiled by consultants with no organizational context. Moreover, policy debates often degenerate into arguments over semantics, but the how of getting this done isn’t as important as simply getting it done.

Ultimately, when security professionals don’t create and maintain policies and standards, they have abdicated their responsibility to the organization that employs them.

Tagged , , , , ,

Security and Ugly Babies

Recently a colleague confessed his frustration to me over the resistance he’s been encountering in a new job as a security architect. He’s been attempting to address security gaps with the operations team, but he’s being treated like a Bond villain and the paranoia and opposition are wearing him down.

It’s a familiar story for those of us in this field. We’re brought into an organization to defend against breaches and engineer solutions to reduce risk, but along the way often discover an architecture held together by bubble gum and shoestring. We point it out, because it’s part of our role, our vocation to protect and serve. Our “reward” is that we usually end up an object of disdain and fear. We become an outcast in the playground, dirt kicked in the face by the rest of IT, as we wonder what went wrong.

We forget that in most cases the infrastructure we criticize isn’t just cabling, silicon and metal. It represents the output of hundreds, sometimes thousands of hours from a team of people. Most of whom want to do good work, but are hampered by tight budgets and limited resources. Maybe they aren’t the best and brightest in their field, but that doesn’t necessarily mean that they don’t care.  I don’t think anyone starts their day by saying, “I’m going to do the worst job possible today.”

Then the security team arrives on the scene, the perpetual critic, we don’t actually build anything. All we do is tell the rest of IT that their baby is ugly. That they should get a new one. Why are we surprised that they’re defensive and hostile? No one wants to hear that their hard work and long hours have resulted in shit.

What we fail to realize is this is our baby too and our feedback would be better received if we were less of a critic and more of an ally.

Tagged , , , ,
%d bloggers like this: