May 15 2018
Leave a comment
Blog Posts

Your Pets Don’t Belong in the Cloud

At too many organizations, I’ve seen a dangerous pattern when trying to migrate to public Infrastructure as a Service (IaaS) i.e. Cloud. It’s often approached like a colo or a data center hosting service and the result is eventual failure in the initiative due to massive cost overruns and terrible performance. Essentially, this can be attributed to inexperience on the side of the organization and a cloud provider business model based on consumption. The end result is usually layoffs and reorgs while senior leadership shakes its head, “But it worked for Netflix!”

Based on my experience with various public and hybrid cloud initiatives, I can offer the following advice.

  1. Treat public cloud like an application platform, not traditional infrastructure. That means you should have reference models and Infrastructure-as-Code (IaC) templates for the deployment of architecture and application components that have undergone security and peer reviews in advance. Practice “policy as code” by working with cloud engineers to build security requirements into IaC.
  2. Use public cloud like an ephemeral ecosystem with immutable components. Translation: your “pets” don’t belong there, only cattle. Deploy resources to meet demand and establish expiration dates. Don’t attempt to migrate your monolithic application without significant refactoring to make it cloud-friendly. If you need to change a configuration or resize, then redeploy. Identify validation points in your cloud supply chain, where you can catch vulnerable systems/components prior to deploy, because it reduces your attack surface AND it’s cheaper. You should also have monitoring in place (AWS Config or a 3rd party app) that catches any deviation and  automatically remediates. You want cloud infrastructure that is standardized, secure and repeatable.
  3. Become an expert in understanding the cost of services in public cloud. Remember, it’s a consumption model and the cloud provider isn’t going to lose any sleep over customers hemorrhaging money due to bad design.
  4. Hybrid cloud doesn’t mean creating inefficient design patterns based on dependencies between public cloud and on-premise infrastructure. You don’t do this with traditional data centers, why would you do it with hybrid could?
  5. Hire experienced automation engineers/developers to lead your cloud migration and train staff who believe in the initiative. Send the saboteurs home early on or you’ll have organizational chaos.

If software ate the world, it burped out the Cloud. If you don’t approach this initiative with the right architecture, processes and people, there aren’t enough fancy tools in the world to help you clean up the result: organizational indigestion.

burping_cloud

 

Tagged , , , , , , , ,
Jan 12 2018
1 Comment
Blog Posts

The Five Stages of Cloud Grief

Over the last five years as a security architect, I’ve been at organizations in various phases of cloud adoption. During that time, I’ve noticed that the most significant barrier isn’t technical. In many cases, public cloud is actually a step up from an organization’s on-premise technical debt.

One of the main obstacles to migration is emotional and can derail a cloud strategy faster than any technical roadblock. This is because our organizations are still filled with carbon units that have messy emotions who can quietly sabotage the initiative.

The emotional trajectory of an organization attempting to move to the public cloud can be illustrated through the Five Stages of Cloud Grief, which I’ve based on the Kubler-Ross Grief Cycle.

  1. Denial – Senior Leadership tells the IT organization they’re spending too much money and that they need to move everything to the cloud, because it’s cheaper. The CIO curls into fetal position under his desk. Infrastructure staff eventually hear about the new strategy and run screaming to the data center, grabbing onto random servers and switches. Other staff hug each other and cry tears of joy hoping that they can finally get new services deployed before they retire.
  2. Anger – IT staff shows up at all-hands meeting with torches and pitchforks demanding the CIO’s blood and demanding to know if there will be layoffs. The security team predicts a compliance apocalypse. Administrative staff distracts them with free donuts and pizza.
  3. Depression – CISO tells everyone cloud isn’t secure and violates all policies. Quietly packs a “go” bag and stocks bomb shelter with supplies. Infrastructure staff are forced to take cloud training, but continue to miss project timeline milestones while they refresh their resumes and LinkedIn pages.
  4. Bargaining – After senior leadership sets a final “drop dead” date for cloud migration, IT staff complain that they don’t have enough resources. New “cloud ready” staff is hired and enter the IT Sanctum Sanctorum like the Visigoths invading Rome. Information Security team presents threat intelligence report that shows $THREAT_ACTOR_DU_JOUR has pwned public cloud.
  5. Acceptance – 75% of cloud migration goal is met, but since there wasn’t a technical strategy or design, the Opex is higher and senior leadership starts wearing diapers in preparation for the monthly bill. Most of the “cloud ready” staff has moved on to the next job out of frustration and the only people left don’t actually understand how anything works.

AWS_consumption

Tagged , , , , , , , ,
Oct 13 2017
Leave a comment
Blog Posts, Security Resources

Moving Appsec To the Left

After spending the last year in a Product Security Architect role for a software company, I learned an important lesson:

Most application security efforts are misguided and ineffective.

While many security people have a good understanding of how to find application vulnerabilities and exploit them, they often don’t understand how software development teams work, especially in Agile/DevOps organizations. This leads to inefficiencies and a flawed program. If you really want to build secure applications, you have to meet developers where they are, by understanding how to embed security into their processes.

In some very mature Agile organizations, application security teams have started adding automated validation and testing points into their DevOps pipelines as DevSecOps (or SecDevOps, there seems to be a religious war over the proper terminology) to enforce the release of secure code. This is a huge improvement, because it ensures that you can eliminate the manual “gates” that block rapid deployment. My personal experience with this model is that it’s a work-in-progress, but a necessary aspirational goal for any application security program. Ultimately, if you can’t integrate your security testing into a CI/CD pipeline, the development process will circumvent security validation and introduce software vulnerabilities into your applications.

However, this is only one part of the effort. In Agile software development, there’s an expression, “shifting to the left,” which means moving validation to earlier parts of the development process.  While I could explain this in detail, DevSecOps.org already has an excellent post on the topic. In my role, I partnered with development teams by acting as a product manager and treating security as a customer feature, because this seemed more effective than the typical tactic of adding a bunch of non-functional requirements into a product backlog.

A common question I would receive from scrum teams is whether a security requirement should be written as a user story or simply as acceptance criteria. The short answer is, “it depends.” If the requirement translates into direct functional requirements for a user, i.e. for a public service, it is better suited as a user story with its own acceptance criteria. If the requirement is concerned with a back-end service or feature, this is better expressed as acceptance criteria in existing stories. One technique I found useful was to create a set of user security stories derived from the OWASP Application Security Verification Standard (ASVS) version 3.0.1 that could be used as a template to populate backlogs and referenced during sprint planning. I’m not talking about “evil user stories,” because I don’t find those particularly useful when working with a group of developers.

Another area where product teams struggle is whether a release should have a dedicated sprint for security or add the requirements as acceptance criteria to user stories throughout the release cycle. I recommend having a security sprint for all new or major releases due to the inclusion of time-intensive tasks such as manual penetration testing, architectural risk assessments and threat modeling. But this should be a collaborative process with a product team and  I met regularly with product owners to assist with sprint planning and backlog grooming. I also found it useful to add a security topic to the MVS (minimum viable solution) contract.

I don’t pretend to have all the answers when it comes to improving software security, but spending time in the trenches with product development teams was an enlightening experience. The biggest takeaway: security teams have to grok the DevOps principle of collaboration if we want more secure software. To further this aim, I’m posting the set of user security stories and acceptance criteria I created here. Hopefully, it will be the starting point for a useful dialogue with your own development teams.

Tagged , , , , ,
Aug 23 2017
Leave a comment
Blog Posts

Infrastructure-as-Code Is Still *CODE*

After working in a DevOps environment for over a year, I’ve become an automation acolyte. The future is here and I’ve seen the benefits when you get it right: improved efficiency, better control and fewer errors. However, I’ve also seen the dark side with Infrastructure-as-Code (IaC). Bad things happen because people forget that it’s still code and it should be subject to the same types of security controls you use in the rest of your SDLC.

That means including automated or manual reviews, threat modeling and architectural risk assessments. Remember, you’re not only looking for mistakes in provisioning your infrastructure or opportunities for cost control. Some of this code might introduce vulnerabilities that could be exploited by attackers. Are you storing credentials in the code? Are you calling scripts or homegrown libraries and has that code been reviewed? Do you have version control in place? Are you using open source tools that haven’t been updated recently? Are your security groups overly permissive?

IaC is CODE. Why aren’t you treating it that way?

devops_borat

Tagged , , , , , ,
Aug 18 2017
Leave a comment
Blog Posts

NTP Rules of the Road

There’s nothing more infuriating than watching organizations screw up foundational protocols and NTP seems to be one of the most commonly misconfigured. For some reason, people seem to think the goal is to have “perfect” time, when what is really needed is consistent organizational time. You need everything within a network to be synchronized for troubleshooting and incident management purposes. Otherwise, you’re going to waste a lot of energy identifying root causes and attacks.

It’s recommended to use a public stratum one server to synchronize with a few external systems or devices at your network perimeter, but this should only be configured if you don’t have your own stratum zero GPS with a stratum one server attached. I can’t tell you how many times I’ve seen a network team go to the trouble to set this up and the systems people still point everything to ntp.org.

Everything inside a network should cascade from those perimeter devices, which is usually a router, Active Directory system or stratum one server.  This design reduces the possibility of internal time drift, the load on public NTP servers and your firewalls, and the organizational risk of opening up unnecessary ports to allow outgoing traffic to the Internet. Over the last few years, some serious vulnerabilities have been identified in the protocol and it can also be used as a data exfiltration port by attackers.

In addition to the IETF’s draft on NTP “best practices,” the SEI also has an excellent guidance document.

While it’s not realistic to have your own stratum zero device in the cloud, within AWS, it is recommended to use the designated NTP pool specified in their documentation.

Oh, and for the love of all that is holy, please use UTC. I cannot understand why I’m still having this argument with people.

Tagged , , , , ,
Jul 27 2017
1 Comment
Blog Posts

Security Group Poop

One of the most critical elements of an organization’s security posture in AWS, is the configuration of security groups. In some of my architectural reviews, I often see rules that are confusing, overly-permissive and without any clear business justification for the access allowed. Basically, the result is a big, steaming pile of security turds.
While I understand many shops don’t have dedicated network or infrastructure engineers to help configure their VPCs, AWS has created some excellent documentation to make it a bit easier to deploy services there. You can and should plow through the entirety of this information. But for those with short attention spans or very little time, I’ll point out some key principles and “best practices” that you must grasp when configuring security groups.
  • A VPC automatically comes with a default security group and each instance created in that VPC will be associated with it, unless you create a new security group.
  • “Allow” rules are explicit, “deny” rules are implicit. With no rules, the default behavior is “deny.” If you want to authorize ingress or egress access you add a rule, if you remove a rule, you’re revoking access.
  • The default rule for a security group denies all inbound traffic and permits all outbound traffic. It is a “best practice” to remove this default rule, replacing it with more granular rules that allow outbound traffic specifically needed for the functionality of the systems and services in the VPC.
  • Security groups are stateful. This means that if you allow inbound traffic to an instance on a specific port, the return traffic is automatically allowed, regardless of outbound rules.
  • The use-cases requiring inbound and outbound rules for application functionality would be:
    • ELB/ALBs – If the default outbound rule has been removed from the security group containing an ELB/ALB, an outbound rule must be configured to forward traffic to the instances hosting the service(s) being load balanced.
    • If the instance must forward traffic to a system/service outside the configured security group.
AWS documentation, including security group templates, covering multiple use-cases:
Security groups are more effective when layered with Network ACLs, providing an additional control to help protect your resources in the event of a misconfiguration. But there are some important differences to keep in mind according to AWS:
Security Group
Network ACL
Operates at the instance level (first layer of defense)
Operates at the subnet level (second layer of defense)
Supports allow rules only
Supports allow rules and deny rules
Is stateful: Return traffic is automatically allowed, regardless of any rules
Is stateless: Return traffic must be explicitly allowed by rules
We evaluate all rules before deciding whether to allow traffic
We process rules in number order when deciding whether to allow traffic
Applies to an instance only if someone specifies the security group when launching the instance, or associates the security group with the instance later on
Automatically applies to all instances in the subnets it’s associated with (backup layer of defense, so you don’t have to rely on someone specifying the security group)
Additionally, the AWS Security Best Practices document, makes the following recommendations:
  • Always use security groups: They provide stateful firewalls for Amazon EC2 instances at the hypervisor level. You can apply multiple security groups to a single instance, and to a single ENI.
  • Augment security groups with Network ACLs: They are stateless but they provide fast and efficient controls. Network ACLs are not instance-specific so they can provide another layer of control in addition to security groups. You can apply separation of duties to ACLs management and security group management.
  • For large-scale deployments, design network security in layers. Instead of creating a single layer of network security protection, apply network security at external, DMZ, and internal layers. 

For those who believe the purchase of some vendor magic beans (i.e. a product) will instantly fix the problem, get ready for disappointment. You’re not going to be able to configure that tool properly for enforcement until you comprehend how security groups work and what the rules should be for your environment.

aws_poop

Tagged , , , , ,
Jun 12 2017
Leave a comment
Blog Posts

Apple, I Wish I knew How to Quit You

My baptism into the world of computers occurred the Saturday my dad brought home an Apple IIc when I was a kid. He was always fascinated by electronic gadgets and this seemed to be the latest device that would sit in a room (and eventually gather dust) next to his ham radio and photography equipment.  The damn thing didn’t do much, but it still felt magical every time I had the opportunity to play games on it; that funky beige box with a square hunk of plastic on a leash moving an arrow around a screen. At the time, it didn’t feel like anything more than a fancy toy to me. I certainly had no idea that this interaction would foreshadow my own love-hate relationship with the entire technology industry.

Years later, I turned into an Apple acolyte while working in the IT department of an East Coast university. I was an MCSE and had become thoroughly exhausted battling Windows drivers and the BSOD circle of Hell.  After I moved to Unix administration, the first version of OS X felt like a relief after long days of command line combat. While it had an underlying CLI that worked similarly to the Unix platforms I was used to, Apple provided a more stable desktop that was painless for me to use. Unlike Windows, the hardware and the OS generally seemed to work. Sure, I didn’t have all the software packages I needed, but virtualization solved that problem. While I lost some of my expertise in building and fixing personal computers, I no longer seemed to have any interest in tinkering with them anymore. During a mostly effortless, decade-long relationship with Apple, I would jokingly tell friends, “Buy a Mac, it will make you stupid.”  I became an evangelist, “Technology is just a tool. Don’t love the hammer, love what you can make with it.”

But as often happens in romances, this one has hit a rocky patch. Lately, trusting the quality of Apple products feels akin to believing in Santa Claus, honest politicians or a truly benevolent God.

The relationship started to flounder about 6 months ago, when I was unlucky enough to have the hard drive fail in my 2013 27″ iMac. It had been a workhorse, but the warranty had expired a few months before and I was in the unwelcome position of choosing between options ranging from bad to worse. I could try to replace it myself, but after looking at the IFIXIT teardown, that idea was quickly nixed. The remaining choices were to pay Apple or buy a new system between release cycles.  After running the cost/benefit analysis in my head the last option made the most sense, since the warranties on both of my laptops were also expired. With some trepidation, I purchased the newly redesigned 2016 MacBook Pro with an external LG monitor. And here’s where my struggles began.

Within a few days of setting up the laptop, I noticed some flakiness with USB-C connections. Sometimes external hard drives would drop after the system was idle. In clamshell mode, I often couldn’t get the external monitor to wake from sleep. I researched the issues online, making recommended tweaks to the configuration. I upgraded the firmware on my LG monitor, even though the older MacBook Pro provided by my workplace never had any issues. Then I did what any good technology disciple would do and opened a case with Apple, dutifully sending in my diagnostic data, trusting that Support would cradle me in their beneficence. The case was escalated to Engineering with confirmation that it was a known timing issue to be addressed by a future software or firmware patch. In the meantime, I refrained from using the laptop in clamshell mode and got used to resetting the monitor connection by unplugging it or rebooting the laptop. I also bought a dock that solved my problems with external hard drives. Then I waited.

Like the rest of the faithful, I watched last week’s WWDC with bated breath. I was euphoric after seeing the new iMac Pro and already imagining my next purchase. But a few days later, after having to reset my external monitor connection four times in a 12-hour period, I emailed the Apple Support person I had been working with, Steve*, for a status update on the fix for the issue with my MacBook Pro. I happened to be in Whole Foods when he called me back and the poetic justice of arguing over a $3000.00 laptop in the overpriced organic produce aisle wasn’t lost on me.

Support Steve: Engineering is still working on it. 

Mrs. Y.: What does that mean? Do you know how long this case has been open? I think dinosaurs still roamed the earth.

Support Steve: The case is still open with Engineering.

Mrs. Y: If you’re releasing a new MacBook Pro, why can’t you fix mine? Is this a known issue with the new laptops as well? If not, can’t you just replace mine?

Support Steve: That isn’t an option.

Mrs. Y: Please escalate this case to a supervisor.

Support Steve: There’s no supervisor to escalate to. Escalation goes to engineering, I only have a supervisor for administrative purposes.

Mrs. Y: I want to escalate this case to the person you report to, because I’m not happy with how it’s being handled.

Then the line went dead.

While Support Steve never raised his voice, for the record, a polite asshole is still an asshole.

This experience has done more than sour me on Apple, it’s left me in a full-fledged, technology-induced existential crisis. How could this happen to me, one of the faithful? I follow all the rules, I always run supported configurations and patch my systems. I buy Apple Care!

I even considered going the Hackintosh route. But I don’t know if I can go back to futzing with hardware, praying that a software update won’t make my system unusable. And Apple’s slick hardware still maintains an unnatural, cult-like hold over me.

So, like a disaffected Catholic, I continue to hold out hope that Apple will restore my faith, because it’s supposed to “Think Different” and be the billion-dollar company that cares about its customers.  But I can’t even get anyone to respond on the Apple Support Twitter account, leaving me in despair over why my pleas fall on deaf ears.

brokeback_apple

*I’m not making this up, the Apple Support person’s name is actually “Steve.” But maybe that’s what they call all their support people as some kind of homage to the co-founder.

Tagged , , ,
May 13 2017
Leave a comment
Blog Posts

When Security Pros WannaCry

Once again the Internet is set to DEFCON level:OH SHIT due the latest ransomware, WannaCry. I’ll refrain from any further analysis of the malware, since it’s already been discussed ad nauseam by every major security vendor. But I will offer the following thoughts.

WTF?! Why is the industry still so bad at dealing with malware? This attack paralyzed organizations like the NHS and impacted carbon units (you know, those things who pay us) in almost 100 countries. But even as the Internet was melting down, organizations were still sluggish to test and apply this patch after it was released.

“In healthcare and other sectors we tend to be very slow to address these vulnerabilities,” says Lee Kim, the director of privacy and security at the Healthcare Information and Management Systems Society.

According to Brian Krebs, Microsoft released a patch for the vulnerability in March 2017, “…but organizations running older, unsupported versions of Windows (such as Windows XP) were unable to apply the update because Microsoft no longer supplies security patches for those versions of Windows.” Woah Nelly, ORGS ARE STILL RUNNING CRITICAL SYSTEMS ON WINDOWS XP?! That OS was released in 2001 and most people don’t even drive cars that old.

And what about all those NextGen security products that are supposed to address zero days? Where was that super-fantastic, heuristic, machine learning AI when we needed it?

The depressing thing about fighting malware is that the most effective solutions are the same as they were a decade ago:

  1. Make sure you’re running an endpoint security product with updated signatures, formerly referred to as antivirus.  Do these programs negatively impact system performance? Oh yeah. Are they foolproof? Hell no. But like a screen door, they filter out the majority of attacks.
  2. Patch and update your devices like it’s 1999.* If you’re running Windows, install the official patch (MS17-010), which closes the affected SMB Server vulnerability used by the attack. Microsoft even released a patch for those unsupported versions of Windows. 

*That’s another Prince reference, in case you missed it.

doves_cry_malware

Tagged ,
Feb 24 2017
Leave a comment
Blog Posts

Chicken Little Security

It’s been one of those weeks in information security. The kind that makes me think about raising sheep in New Zealand, because they won’t argue with me about APTs and attribution. In addition to the Java/SMTP/FTP vulnerability that has vendors scrambling, I’ve suffered through trying to explain the following:

While I could probably break each of these down and explain how the sky really isn’t falling, I think Val Smith said it best recently:

Are you able to get an accurate inventory of your network?
Can you rebuild any system, anywhere, in less than a day?
Can you push software and configuration changes, including patches, remotely?
Do you have tested backups?
Do you have enough IT/DevOps to keep your environment stable?
Do you have a tested IR plan?
Do you have proven data sources (logs, netflow, full pcap, endpoint telemetry)?

If you answered no to any of those questions, you probably shouldn’t be too worried about SHA collisions. 

Here endeth the rant.

sha-asif

Tagged , ,
Feb 22 2017
Leave a comment
Blog Posts

Fear and Loathing in DC

Lately it takes a very compelling request to get Mrs. Y to leave the Sanctum Sanctorum and give a talk, but what better topic is there than digital defense? I love the smell of FUD in the morning, whipping people up into a frenzied paranoia, then watching them rush out of the room to get prepaid cell phones and put duct tape over their web cams.

In all seriousness, no matter which side of the political fence you inhabit, no one can argue that government surveillance is at an all-time high. I can’t even get the Security SOC Puppets together in the same room anymore, because they’re demanding a Faraday cage on their contract rider. So I’m happy to offer my perspective and some guidance to help the general public (i.e. nerd-challenged) protect themselves from snooping and digital attacks.

Special thanks to the the former (recovering) attorney and activist who organized the event.

If you don’t trust Slideshare, you can download the presentation here.

Tagged , , ,
← Older posts
Newer posts →
%d bloggers like this: