Tag Archives: networking

The Question of Technical Debt

Not too long ago, I came across an interesting blog post by the former CTO of Etsy, Kellan Elliott-McCrea, which made me rethink my understanding and approach to the concept of technical debt. In it, he opined that technical debt doesn’t really exist and it’s an overused term. While specifically referencing code in his discussion, he makes some valid points that can be applied to information security and IT infrastructure.

In the post, he credits Peter Norvig with the quote, “All code is liability.” This echoes Nicholas Carr’s belief in the increased risk that arises from infrastructure technology due to the decreased advantage as it becomes more pervasive and non-proprietary.

When a resource becomes essential to competition but inconsequential to strategy, the risks it creates become more important than the advantages it provides. Think of electricity. Today, no company builds its business strategy around its electricity usage, but even a brief lapse in supply can be devastating…..

Over the years, I’ve collected a fair amount of “war stories” about less than optimal application deployments and infrastructure configurations. Too often, I’ve seen things that make me want to curl up in a fetal position beneath my desk. Web developers failing to close connections or set timeouts to back-end databases, causing horrible latency. STP misconfigurations resulting in network core meltdowns. Data centers built under bathrooms or network hub sites using window unit air conditioners. Critical production equipment that’s end-of-life or not even under support. But is this really technical debt or just the way of doing business in our modern world?

Life is messy and always a “development” project. Maybe the main reason DevOps has gathered such momentum in the IT world is because it reflects the constantly evolving, always shifting, nature of existence. In the real world, there is no greenfield. Every enterprise struggles to find the time and resources for ongoing maintenance, upgrades and improvements. As Elliott-McCrea so beautifully expresses, maybe our need to label this state of affairs as atypical is a cop-out. By turning this daily challenge into something momentous, we make it worse. We accuse the previous leadership and engineering staff  of incompetence. We come to believe that the problem will be fully eradicated through the addition of the latest miracle product. Or we invite some high-priced process junkies in to provide recommendations which often result in inertia.

We end up pathologizing something which is normal, often casting an earlier team as bumbling. A characterization that easily returns to haunt us.

When we take it a step further and turn these conflations into a judgement on the intellect, professionalism, and hygiene of whomever came before us we inure ourselves to the lessons those people learned. Quickly we find ourselves in a situation where we’re undertaking major engineering projects without having correctly diagnosed what caused the issues we’re trying to solve (making recapitulating those issues likely) and having discarded the iteratively won knowledge that had allowed our organization to survive to date.

Maybe it’s time to drop the “blame game” by information security teams when evaluating our infrastructures and applications. Stop crying about technical debt, because there are legitimate explanations for the technology decisions made in our organizations and it’s generally not because someone was inept. We need to realize that IT environments aren’t static and will always be changing and growing. We must transform with them.

Tagged , , , , , ,

Why You’re Probably Not Ready for SDN

While it may seem as though I spend all my time inventing witty vendor snark to post in social media,  it doesn’t pay the bills. So I have a day-job as a Sr. Security Architect. But after coming up through the ranks in IT infrastructure, I often consider myself “architect first, security second.” I’m that rare thing,  an IT generalist. I actually spend quite a bit of time trying to stay current on all technology and SDN is one of many topics of interest for me. Especially since vendors are now trying to spin it as a security solution.

Software-defined networking (SDN) is still discussed as if it’s the secret sauce of the Internet. This despite Gartner placing it at the bottom of its Networking Hype Cycle due to “SDN fatigue” and the technology’s failure, thus far, to gain much traction in the enterprise.

 However, the magical SDN unicorn still manages to rear its head in strategy meetings under the new guise of hyper-convergence and the software-defined data center (SDDC). This is probably due to IT leadership’s continued yearning for cost savings, improved security and the achievement of a truly agile organization. But is SDN, with its added complexity and startling licensing costs, really the answer?
You can read the rest of the article here. And yes, there’s a registration wall.
Tagged , , , , , , ,

The Physical Layer: Networking’s Fifth Circle of Hell

Recently, Pope Francis made headlines when he called unrestrained capitalism the “dung of the devil.*” I’m betting he’s never been a network engineer dealing with negotiation issues.** Problems at the physical layer always seem to be the most painful, mostly because you think, “Dammit, I should have known better!”

These thoughts come after the third in a series of attempts to insert a Vendor’s network packet broker bypass module inline at my $dayjob. Vendor requires and only supports manual configuration of the GigE LX fiber ports, but we couldn’t get this to work. It was infuriating, because we seemed to have link, but the dB loss was too high.

I should mention that in most of my previous jobs, the network team always managed the perimeter router. But not in this shop. Unbeknownst to me, the demarcation point, a port on a router, was set to auto-negotiate. This wasn’t identified in the diagram and was only discovered after lots of yelling by a cast of thousands, including multiple representatives from the Vendor, in a server room late at night. This issue was finally resolved by opening a case with the service provider and requesting that they hard set the uplink port on the managed router.

The situation was infuriating. I woke up at 5:00 AM the next morning with a bad case of nerd-rage. I needed to find a target for my anger, so I went to Google to help me direct this justified (so I thought) apoplexy.  I started reading through the IEEE standards on Ethernet and 1000BASE-X, even finding an “interpretation” document. This just made me more irate. Auto-Negotiation
The Auto-Negotiation algorithm of Clause 28, while the preferred method for the determination of half or full duplex operation, is not currently defined for fiber MAUs.
Manual configuration, while not recommended for copper-based MAUs, is the only practical choice for fiber implementations. Connecting incompatible DTE/MAU combinations such as a full duplex mode DTE to a half duplex mode MAU, or a full duplex mode station (DTE and MAU) to a half duplex network, can lead to severe network performance degradation, increased collisions, late collisions, CRC errors, and undetected data corruption.

Point – Vendor User Configuration with Auto-Negotiation

Rather than disabling Auto-Negotiation, the following behavior is suggested in order to improve interoperability with other Auto-Negotiation devices. When a device is configured for one specific mode of operation (e.g. 1000BASE-X Full Duplex), it is recommended to continue using Auto-Negotiation but only advertise the specifically selected ability or abilities. This can be done by the Management agent only setting the bits in the advertisement registers that correspond to the selected abilities.

Point – Mrs. Y. The Vendor doesn’t support this option.

Interpretation for IEEE std 802.3-2002 #1-11/02 (1000BASE-X Auto-negotiation)

The standard clearly states in the second paragraph of subclause Control register (Register 0), “When manual configuration is in effect at a local device, manual configuration should also be effected for the link partner to ensure predictable configuration.”
Hence, if this recommendation is not followed and a link has manual configuration in effect at a local device but not at the link partner, or the reverse, the resultant link configuration can not be predicted.

Point – Vendor

Final Decision: I will direct my righteous indignation at the IEEE. I spent over two hours going through reams of documentation to find this information and I’m still confused by which is the “best practice.” Is it any wonder that vendors and network professionals still argue about the standards?


*Devil Dung would be a great name for a punk rock band.

** And please don’t flame me over referencing Pope Francis. He seems like a cool guy with a good sense of humor. I don’t think he’d mind.

Tagged , , , , ,
%d bloggers like this: