As some of you may know, I have spent a fair amount of my time in the last few years designing and improving multi-tennant hosting environments. Each revision attempts to learn from the mistakes of the previous iterations, as well as bundle in new features and “advancements” from each of the different vendors in the stack.

New offerings on the storage fronts, developments in the server space in the form of the boom of virtualisation, and the simple existence of the network amongst the fact that none of these technologies changed the existing/fundamental laws of networking.

Software-Defined Networking has sprung up as a way of providing both advancements in our current architectures and providing agility in changes needed in the future, but what is truly needed is a true abstraction of the entire data centre model that encompassed all of compute, storage, security and networking. The ability to define all of the requirements of your existing data centre and have them deployed and rolled out across which ever stack you are using (Private / Public / Hybrid / Tomorrows Favourite buzz.), in a consistent and definable manner.

Merging the requirements of each of the existing silos and describing them in a uniform, consistent and repeatable manner is a great step towards blending the cultures of these teams and beginning to solve real problems instead drawing lines in the sand. Continue creating more awesome!

Platform as a Service offerings are already delivering on a model of consistent application delivery (assuming your code fits within certain criteria), but PaaS is not always appropriate for every situation. I have several large “end customers”  who do not have staging or lab environments that appropriately mirror their production environment because the current glue is far to complicated to recreate. If the entire infrastructure was abstracted to the point that they could roll out a complete mirror at the click of a button this would be a no-brainer.

Thankfully, there are many smart people thinking about just these problems, and what it means to the industry. Tom Hollingsworth wrote these words over at Network Computing discussing some of these initiatives, and the guys from GestaltIT / Tech Field Day have joined with the team from SDNCentral to bring together the Software Defined Data Center Symposium in September. It’s still early on in the buzz around SDDC but, like Cloud and SDN before it, over the next 12 months I expect we will hear far more about this as a topic and a trend.

For those of you not local to the Bay Area, the event is also going to be streamed live. For me this means tuning in from 1am Sydney time, but I plan on being online for the entire event. I had the good fortune to be present for the Software Defined Networking Symposium back in 2011, and I learnt an amazing amount from the event, and I expect the SDDC Symposium to be just as informative.

Registrations have now opened, and tickets can be booked and event details can be found at http://sddc13.eventbrite.com/.

It truly is an exciting time to be in this industry!

I dont doubt that vendors have a tight line to walk when it comes maintaining their brand integrity. To build up a skill set in the market the certification teams put in many weeks developing a program that is relevant, useful and achieves the goals required. Followed by countless hours reviewing each of the certifications regularly to ensure integrity. There is the added benefit that these certifications build of community of loyal followers – The Cisco and VMWare certification programs are evidence to this.

I personally have been involved in the development, technical review, and exam rewrite process and I can attest to the effort that the certifications teams go to to ensure the validity and integrity of their offerings. Weeding out sources of brain dumps and NDA violations and other activities that threaten this integrity work becomes an on going commitment that requires many hours of dedication.

On the other side of the line is the requirement to have the partners representing the vendor to maintain a level of skills and customer satisfaction. This ensures that when the brand is represented in the market that it will be delivered by the most skilled people capable of delivering the best outcome from the products on offer. This ensures that the vendor brand is not tarnished by substandard implementation or post-sales response.

The easy option is usually to mandate that partners levels be broken into tiers and that each partner must maintain a minimum set of certifications as well as some minimum dollar commitment. And I dont doubt that this is effective in the short term. It’s an easy metric to graph and report on. Sadly it also has the exact opposite effect than the original desired business goal.

The problem usually comes about because various partners start requiring all new staff to get X, Y and Z certification. The number of people obtaining these certifications purely to get a job, and not to truly understand and learn the concepts increases. With this too is the growth of brain dumps and other practices that reduce the integrity such as brute forcing exams or engineers sitting in countless classes to achieve continued education requirements.

As a result meeting minimum number of certified people becomes the goal, and not the education of the suitable engineers capable of delivering the best outcome for both the customer and the vendor. Just as we saw with the CCIE program prior to v4 or the MCSE program in the late nineties and early 2000s – The market gets flooded with sub-par certifications and integrity is diluted or lost.

And in the end, everyone loses.

There is one thing that anybody who has been in a room with me longer than 5 minutes can tell you – I am not a smart guy! I have lots of smart friends. I am not one of them. Sometimes I feel like I’ve done more stupid things, more often than I would like to admit, and its only that I have been stupid enough often enough that I have eventually learned “dont do that!“.

A couple of things have happened over the past few weeks that made me think about “The Smartest Guy in the Room”, and I thought I would share a few incomplete thoughts on the matter.

As a Consultant

Back on June 18, Matthew Norwood (who I would nominate for the award of “Nicest man in Network Blogging and Puppeteering”) wrote this blog post about consulting. In this post he talks about how as a consultant sometimes you have to accept that you are not the smartest guy in the room. At first this may seem odd, especially because usually our customers are paying us lots of money because we are experts. Funnily enough, on consulting engagements my job is more “I know where to find how to do that thing you are expert in”.

The most recent example of this was a few weeks back I was working on a Proof of Concept lab with a customer to see how we could integrate a series Juniper MX routers with their existing ASR9000 MPLS topology. Thankfully these sorts of things are fairly standard offerings on both of these platforms but this particular customer had their entire network built on Multicast over MPLS Point-to-Multipoint LSPs. Now when you consider that most people consider Multicast a “corner case” in networking, then these additional requirements made it a corner case inside of a corner case! On top of the fact that the customer engineers I was working with did this day-in and day-out, there was no pretending I was the expert. Instead my job became “I know where and how to make the Juniper MX do what you want”, and that was how I proceeded. And I can tell you this was one of the most enjoyable engagements I have had in quite some time.

As an IT Guy

It’s been stated many times that when you look at a group of IT guys, that most likely many of them were the “nerd” or “geek” growing up and were used to being the smartest guy in the room. As such, we often get interested in all sorts of technologies and ideas and run off on different tangents. I work with a couple of amazing guys who seem to tackle any odd-ball concept that comes to mind – like rigging a relay in circuit with the door control system so they can control the door with their pebble watch or simply building 3D printers to allow them to make all the toys they dream of. And often times they love to do this towards the end of the day/week when we are wrapping up units of work, and I know many late nights have been spent back in the office hacking away on something crazy “just because it can be done”.

Sadly, occasionally we take on tasks that are not our core competencies. This isn’t to say that we are unable to do the work, but simply it is not our strength and it can take much longer to resolve than it would a suitably trained / skilled engineer. A recent example of this was an incident that recently caused our team of network engineers to remember what it means to be a “Windows Administrator” and all the bits and pieces required to get some systems back online. We were more than capable of completing the work, but not necessarily in the most efficient fashion. We called in “a few favours” from friends with the required skills to sort this out, but  sometimes it goes against our gut feeling to seek advice. In times like this we need to accept that there are in fact people with strengths in these areas, and to ask for help is not about admitting defeat.

As Full Stack Engineer

And this leads into my last incomplete thought. Some time last week I engaged in a rant-fest with Paul Gear regarding the topic of breaking down silos within IT and what it means to be a “full stack engineer” (for lack of a better word). Admittedly the rant was primarily around how its not easy to break down silos when we still all refer to each other as “Server Guys”, “Storage Guys” and “Networking Guys” (nobody talks to/about “Security Guys” because they “just say no“), but over the course of the conversation I developed a few opinions (as I am known to do). And these are my thoughts on this…

Recently:

• I have started learning to program in Ruby, as well as learning about OpenStack. I have spent a fair amount of my “down time” learning about coding in Ruby and learning how to automate many of my regular tasks. I have picked up a few libraries developed by the guys at Juniper (mad props to Jeremy Schulman) and started learning more.
• I have started getting to grips with service automation using Puppet, and investigating how end-to-end provisioning of both server and network resources as a single process can improve delivery and satisfaction.
• I have built an OpenStack lab (as part of my beta testing of the Juniper Contrail product), and started to learn how to spin up and down workloads on demand and again how to integrate that with the various network requirements.

Do I want to be a programmer or a guy running servers all day? No, but I do want to be able to “talk the language” of the guys who do. There is great benefit in this ability not just as a consultant but as a member of a “Silo Free IT Team”. My view is that in the future, we will have blended teams of engineers, each with an understanding of the various aspects of the stack (Server, Storage, Security and Network), but we will still specialise in one or two of those areas. Working in such a team will be all about admitting you are not the smartest guy in the room and simply admitting “Yes, but I do know this part pretty well. Let’s come up with a solution”.

I dont see specialisation going anytime soon. Even within each of those four areas of IT there is a world of sub-specialisation that can be entire careers for people. The key is to have an understanding of all of the areas involved and know where best to apply your learned skills and experiences. *This* is what makes the future of IT teams… not a group of “jack of all trades” engineers, as some people interpret the idea of “breaking down silos”.

Mop and Bucket

I’ve learnt over time to know what I know, and to ask for help when I dont. Sometimes it takes me a while to get there, but almost every time that I do things get better 🙂 There is too much happening in IT and too much change in our industry to remain an expert at anything and everything for long. Widen your perspective for high level understanding, and learn where you can best apply and extend your skills.

If nothing else, it will make for an interesting and progressive career!

I may not be the most “travelled” person in the world, but over the past couple of years I have managed to find myself in several places across Asia, the Pacific Islands and also the US. One thing has always stood out – Speaking the same language is the hardest part of travelling! Now when I travel to parts of Asia and Im dealing either in hand gestures or with somebody trying their very best speak English (Their English is 1000x better than my Cantonese or my Khmer), and we both make allowances for the difficulty of not speaking the same language.

Sadly, when I travel to the United States and we both attempt to speak “English” nobody can ever seem to understand me. Sometimes its my accent, and other times its the colloquialisms I am using that do not translate effectively, and I am treated by blank stares on the other person trying their hardest not to say “Huh?”.

I’ve learned to deal with this by talking slower and thinking carefully about the words I use to ensure that they dont have some local significance. Anybody who has met me in person knows that I talk loudly, quickly and for a long time, but put me on a call (or a podcast) with guys from the US and I have to slow it right down (and trust me, its painful listening back to it!).

So what does this have to do with networking? Well I was reading this post by Mrs. Y over on the Packet Pushers blog, and it started to resonate with some thoughts I had been having problems putting into words. In the article she mentions how the various wireless vendors implemented the CAPWAP “standard”, but the actual implementation of the standard with accompanying “extensions” means that solutions did not allow for cross vendor integration. I read this post about 30 seconds after reading this post from Ivan Pepelnjak where he mentions about NEC needed to implement certain features to their ProgrammableFlow controller to provide better stability, but at the potential cost of not being compatible with other vendors – the very thing OpenFlow was designed to support.

It may also be of note that I spent most of my day today working on some integration testing between Juniper MX and Cisco ASR9k routers to facilitate Multicast across Point-to-Multipoint MPLS LSPs. Part of todays work involved finding out exactly which parts of the “standards” were best supported across both vendors while not taking advantage of either vendors “nerd knobs” and “enhancements”. Sometimes each vendor had implemented different solutions to get around an “on the ground problem”, and it was partly my job to which ones played nicely across both – and with the least downtime.

So my thoughts (and I dont have any answers… just thoughts), come down to “What is better? Implement a standard, then put fixes and extensions on top of it to do what I need, or to build a new protocol?”. To a degree having “some form” of integration between vendors is very important, but if the 95% use case is making use of proprietary extensions then have we really achieved that? People ask “Why did Cisco come out with onePK instead of OpenFlow?”, but then I question “Well if they wanted to provide more functionality than OpenFlow provides, why extend the model with non-interoperable parts and pretend to still be OpenFlow?”.

Just like speaking the same language but still not understanding each other, how should we ensure better communication while not causing unneeded long term pain? Maybe the answer is to implement extensions, then work with the community / standards bodies to allow cross vendor in the future (and still provide first mover advantage for the innovator). Obviously commercials around this make this a less attractive option for the big players, but this is how standars get retroactively created.

I would be really curious to know your thoughts.

Earlier today Ethan Banks wrote a really good blog posts about “Thoughts on Working as a Consultant for a VAR“. I found his point of view quite interesting and I will say I can understand his points. I can also say that I would rather be a consultant than a full time engineer at a customer site. As a little bit of background I have spent most of my career working as a consultant. I did do a two year stint as network operations manager for a wireless ISP which itself was quite fast paced, but other than that Ive work as a consultant in one form or another.

Maybe I have ADD, maybe I just need to focus, but I have found that constantly having different projects going allows me to satisfy these tendencies. I feel I work better with more than one thing to occupy my time. I see friends who work for enterprise customers who spend their days submitting change requests that third party support companies fulfil, or spend months writing detailed design guides for projects that inevitably get canceled and all that time is spent without getting to touch the things they got into this industry for. Which brings me to my first point…

Toys, Toys Toys!

Im 32 years old, and I still love toys. Sometimes I think the only reason why Im in this industry is the ability to play with expensive and complex toys. And I get paid to do it! I enjoy the challenge of learning a new piece of equipment and deploying it into a customers network. I enjoy learning all the new opportunities that a new software update or a router line card can offer to my upcoming network designs (my sales guys also enjoy being able to sell those things to existing customers… bloody sales drones!).

I have worked at my current employer for 10 months now and in that time I have worked on many projects, not the least of which has included rolling out the second Juniper QFabric deployment in Australia (and one of the first in the world), delivering security solutions based on high end data centre SRX devices including IDP, SSL Inspection and Proxying, a relatively large Juniper MAG deployment across multiple sites with failover, and numerous network audits across many verticals including finance, utilities, enterprise and service provider networks.

The other aspect to toys is the ability to get your hands on all the right equipment to learn and play with *before* you need to deploy it. This really comes into its own when you have…

The right employer

Ethan’s posts hinted at being burned by his previous employer. I know I can attest to that over the past 15 years Ive been in the industry. Sometimes its the customers, sometimes its the employer. The right employer will understand your career aspirations and will endeavour to provide you with all the resources you need to be successful during both your time with them as well as your career beyond them. The right employer understands how often engineers change jobs in this industry and that what goes around comes around.

Your employer needs to understand that certification is beneficial to him, but more importantly that you as the engineer need to understand the content of the exams you are passing. Perpetuating the degradation of industry certifications by promoting brain dumps or becoming a “puppy factory” for certifications churning out sub-par engineers devalues both the certification as well as your ability to sell your services to customers.

The right employer will cover your expenses for gaining certification, and provide you with the resources, both equipment and time, to actually complete these career goals.

If you’re lucky your VAR will give you access to both pre-sales and post-sales implementation work giving you a wider view point of the network life cycle as well as building up skills that end customer engineers will never get to practice. And you will do that many times a week!

Honest about the downsides

• You’re supposed to know everything: I agree with Ethan that this can be stressful, but it doesn’t take long to pick up the habit of either being one google search ahead of the others, or as I often say “Not knowing the answer, but knowing where to find them”. If you work for a consulting firm that doesn’t require your customers to buy some form of vendor TAC support then you are further against the wall and you/they are not operating in your customers best interest. *Always* have the ability to escalate.
• You’re a commodity: Yes you are a commodity, but you also are (should be) “in demand”. If you are doing your job well, and your employer knows how best to leverage your skill set appropriately (not always in your control) then this works in everyones favour. If you are not doing your job well then you shouldn’t be a consultant, and if your employer doesn’t know how to effectively sell your skills you should be looking elsewhere. You will get abused, ignored and eventually get bored and quit or made redundant. Scary truth, but its better to move on and find something you are happier doing.
• You’ll sacrifice your body if you’re not careful: Well I’m not even going to pretend to deny this one. I am living proof that ignoring the health aspects of your life will creep up on you, but Im not convinced this is specifically related to VAR lifestyle. Any high stress, fast paced career can do this to you. You just need to keep these things in check. On a side note to this one, I’ve taken up a Paleo/Primal lifestyle for the past couple of months and Ive found wonderful improvements to my health without major sacrifices (in both time and diet), but thats a whole other blog post!

Mop and Bucket

Yes working for a VAR can be hard work. It can be fast paced and involve a lot jumping between contexts, but if you are like me you just may find that the very draw card to this career option.

Sometimes I dream about working on a single network and finishing all those “little things” that I wish I could implement for all of the networks I build. Then I spend two weeks onsite at a customer and all I want to do is get out of there.

If you are willing to invest 12 months of your life to “give it a go”, you will learn far more in that time, and touch a wider array of equipment than you could hope to touch in 3 technology refresh cycles at an end customer.

And this is what keeps me coming back to work tomorrow!

The Setup

So there I was, setting up my “command centre” for the 15 hour flight from Sydney to San Jose via San Francisco. I had my Macbook and my iPad fully charged, as well as ye olde print book ready just in case. I idly flipped through the inflight entertainment guide to see what movie would be playing on the main cabin display.

“Midnight in Paris” ? What’s this rubbish? Oh… Woody Allen… right… yeah… that’s just what I need 🙁

Thankfully I usually fall asleep the minute the plane pulls away from the terminal and wake up just as we’re coming into land. Should be able to sleep most of the way, and maybe read a few pages of my book between naps. This usually gets me in trouble with my wife, but I was flying solo on this trip so I was free to do as I pleased.

As fate would have it, my Macbook battery was at about 50% just as the movie started so I decided to “save some for a rainy day”.

Let me put on my headphones and see what this movie is like. It’s a Woody Allen flick (in Paris no less) so Im bound to be asleep in 20 minutes.

Oh it’s about “Art” is it? Lets make that 15 minutes!

Why is Rachel McAdam’s character such a bitch? That’s not like her.

Hrrmm… I never realised how much Owen Wilson acting the way he always does was so much like Woody Allen before.

And just like that, I was sucked into watching this movie I had no desire to see!

The Wonder

So the premise of the movie is that Owen Wilson’s character, Gil, is a writer on vacation in Paris. His romantic notions of life in Paris during the 1920’s, during the time of Picasso, Hemingway and F. Scott Fitzgerald, are at odd’s with his wife and her family. While he wants to revel in the beauty and atmosphere, they are more interested in keeping up appearances.

During a stroll on his own looking for inspiration, and after a few drinks, he finds himself lost as the clock strikes midnight. At this time an old car pulls up along the road and the occupants invite him to join them. During this trip he is transported back to the 1920’s where he meets his heroes Hemingway, the Fitzgeralds, Gertrude Stein, and a whole host of other literary and artistic greats. Over the course of the movie he spends his days with his family and his nights with his heroes. During his time with his heroes he is able to approach each of them for advice about his book, his personal situation and life itself.

This same opportunity was offered to me back in 2011, and that was why I was on this plane. I had been invited to attend Tech Field Day’s Network Field Day 2 at the end of October of that year. I had actually received the invitation on night of my birthday in early September, and here I was with the opportunity to rub shoulders with my own personal heroes in the networking industry. The likes of Ivan Pepelnjak from a blog I read daily (blog.ioshints.info), Greg Ferro and Ethan Banks from the Packet Pushers Podcast, not to mention Stephen Foskett the man behind the whole event, as well as several regular contributors to the Packet Pushers Community and friends from Twitter. Just like Gil, I had the opportunity to get feedback and advice from those whose advice I sincerely valued. The ability to sit down with both the other delegates as well as the representatives of each of the sponsors was truly insightful.

Coming from Sydney, the “scene” in the whole Bay Area is entirely different to what I was used to. People dressed extremely casually here and there was a vibrant community buzzing with all sorts creative and cutting edge technologies. This was uniquely demonstrated during the Open Flow Symposium that was held on the day before the Network Field Day event began. Both vendors, implementors and attendees were all excited about the possibilities that were coming. Some of these ideas I was aware of before arriving in San Jose, but the depth and pace of these changes took me by surprise. Everybody was open about ideas – both what would, wouldn’t and couldn’t possibly work! Many names and faces I had been following online for the past 18 months attended this event. I was in awe of everything. I dont think I spoke to too many people that day as I was too stunned.

Over the next (densely packed) two days, we were taken from vendor site to vendor site and presented with their latest and greatest – and usually by their brightest. No questions seemed off limits and each of the vendors truly seemed to take on board the questions we raised and the “advice” we offered. These two days were completely information overload mode, and I dont think I was prepared to capture all of the useful information that was being presented to us. I have had to go back and watch the various videos several times to see what parts I had previously missed.

One thing that was pointed out to me by Stephen Foskett as we were driving through all these massive campus buildings was that “Over there is this massive building, costing millions of dollars. They do something in our industry, and I have no idea who they are. And this area is filled with these places. It’s exciting to see so many new companies and wonder ‘whats going on in there’ “.

I had an opportunity to spend intense nerdy times with the other delegates and sponsors during the jam-packed days, then follow on into the night just geeking out because the atmosphere was so charged. Everywhere I turned there were people doing “great things” that I wanted to be a part of.

Just like Gil, I too was caught in the position of “being where I wanted to be, and never wanting to leave”

The Reality

After 5 days away, I returned home to my normal life. Well, sort of normal. My wife was in the middle of a three month stint working in an outback Aboriginal Community School, and I wasn’t going to see her for another week or so. Then there’s all the things at work that needed tending to since I had been away.

I found my self longing to be back in San Jose with the group I had just spent so much time with. It didn’t matter that all the other delegates had returned to their own homes, that is where I wanted to be. And my reality was far removed.

“Nostalgia is denial – denial of the painful present… The name for this fallacy is ‘Golden Age Thinking’, the erroneous notion that a different time period is better than the time one is living in. It’s a flaw in the Romantic imagination of those people who find it difficult to live with the present.”

This quote above comes from a character in the movie who was a know-it-all A-hole, but he had a point. His witty remark to Gil’s character was both spiteful yet accurate, though his didn’t stop Gil from still pursuing his dream.

I knew that if I wanted to progress it into the areas in the industry that I wanted to be a part of, that I would have to make certain changes to my career path. I mapped out what I felt were a series of career goals and achievements that I would need to accomplish in order to make headway. Since this time I have changed jobs to focus on the particular projects and technologies I felt I needed, buckled down, put in a lot of study and research.

Also during this time I went into “Social Media Radio Silence”, as I was busy focusing on some of my end goals and needs. I was so busy focusing on the future, I was completely ignoring the present. I had lost my inspiration to write because I felt so overwhelmed by all the things I didn’t know. Instead of documenting my discoveries, I was actively avoiding the things I couldn’t answer.

Life got in the way and I was feeling discouraged.

The Re-Awakening

During Gil’s trips back and forth between time periods, he meets a young French girl from the 1920’s named Adriana. Even though she lived in the time period that Gil romanticised, she longed to live in an earlier period of Paris’ history – a time she felt was truly inspired. And fate (and of course the script) would have it, both Gil and Adriana found themselves in Adriana’s romanticised time – La Belle Epoque!

While Adriana is lost in her dream, they meet Paul Gauguin and Edgar Degas who themselves were discussing how “this generation is uninspired”. This is when Gil discovered the truth, and had to explain it to Adriana who didn’t want to leave:

“Adriana, if you stay here though, and this becomes your present then pretty soon you’ll start imagining another time was really your… You know, was really the golden time. Yeah, that’s what the present its. It’s a little unsatisfying because thats what life is – just a little unsatisfying”

And in with these words both Gil, and myself of re-watching the film, knew what needed to be done.

If I ever want to write something worthwhile, I have to get rid of my illusions that I’d be happier in the past”

Mop and Bucket

In early February, Stephen Foskett was out in Australia to keynote two VMUG gatherings in Sydney and Melbourne. During this time we spent the better part of two days catching up, discussing life, the industry, and careers. Whether he knows it or not, those conversations really helped solidify some of my goals and plans.

I’ve learnt that Australia is no farther away than I make it, and I can still be quite active and involved even if I am not “in the heart of things”. Positioning oneself to take advantage of opportunities that arise, and to connect with various people both within our industry, as well as the clients we have, is the best way to stay in front of the game and stay involved. This is evident in the fact that I was invited to be a Juniper Ambassador in October 2012.

I would like to think that I have more technical posts coming in the near future, because it certainly feels like a long time since my last one. I’m feeling inspired again, and getting ready to re-engage. And for that, I am truly sorry 😉

PS. Please refer to my Disclosure Statement in reference to my participation in the Tech Field Day events, as well as the Juniper Ambassador program.

I have a bee in my bonnet. After my last post full of love and bromance, this one is full of hate and vitriol – and I don’t apologise! We have all seen many presentations on each vendors latest and greatest “fabric” technologies over the past 18 months. It doesn’t matter which vendor, whether the presenter is sales or tech, or even enterprise or service provider focused – at some point almost every one declares that their solution is “the end of spanning tree”. It gets worse when they actively advise that you do not run spanning tree in your environment.

And I don’t buy it 🙁

The Premature Death of Spanning Tree

Spanning Tree: noun – A pox on the house of networking

Everybody loves to hate on Spanning Tree. Haters gonna hate. While we’ve all been bitten by something horrible happening related to spanning tree, I have seen many more things go wrong because people *didnt* configure Spanning Tree properly.

Vendors knew how painful it was and went to great lengths to ensure that we didn’t need to do anything so that it would “just work”. Which is great… right up until the point that you run into a limitation on the STP PixieDust Mode. Often this comes in VLAN dense environments when you max out the total number of spanning tree instances that your devices will handle. Oh thats easy – lets run out Multiple Spanning Tree!

I can hear the gasps from many people now. If people hated Spanning Tree, then they have a full lynch mob ready with pitch forks and stakes at the mere mention of poor MSTP. And they hate it for a reason. MSTP makes you think about how spanning tree works again and all the PixieDust goes away. And networks become hard again.

I will let you in on a little secret:

I actually really like MSTP and I implement it every chance I can get.

Yes its a little harder and requires a little more forethought, but I would rather do that at the design time then have to overhaul my design later to meet some new need well after my network has reached “critical mass”. I have spent many hours rebuilding spanning tree designs because I needed more than 128 instances. Sadly in more than one case I needed to work out the best way to deal with a group of Catalyst 3750 in PVST+ with 1000 VLANs configured (and 900 VLANs with STP disabled).

And things get messy, and things get hard. So lets find a new solution.

The Magical Healing Powers of Woven Unicorn Hair Fabrics

Somewhere over the rainbow, far beyond the Dark Forest of Broccoli Despair, many magical elves have worked hard to deliver us a the perfect solution to the problems I listed above. Vendors have taken this creation and moulded into their own “Fabric Solutions”. Some created skinny jeans, others an uncomfortable sweater vest. Sadly most of the time they have just presented us with a sensible pair of slacks that the sales people try to sell as a three piece suit.

A sensible pair of slacks (Unicorn Hair or otherwise), is perfectly apt when used as intended, but if you drape them over your shoulders and call them a shirt then your wrong (or a hipster. In which case your cardigan is probably over the top of your shirt-slacks).

And so it is with data centre fabrics. I agree that most of these solutions will allow us to disable spanning tree on our core/fabric facing interfaces. We will get many of the benefits of multi-path layer 2 and some times efficiencies gained by avoiding the flooding of L2 addressing information. Turning off spanning tree into the fabric core makes sense. Im happy with that.

So what about all those edge interfaces?

Do we live in a world where end users never plug two ports together?

A client PC never bridges interfaces?

How about “Oh my VoIP phone has two network ports let me just…. BOOM!”

Maybe you have no requirement to integrate with other networking infrastructure, but end stations can still do bad things and thats usually when you don’t want them to.

The Indiscriminate Killing of Canaries

So how do we go about detecting these loops? Well over the past couple of decades we’ve presented ourselves with a whole cage full of canaries that can alert us to loops or other similar problems in the network. These are our early warning signals that “Something bad just happened…” and better yet “… so let me just fix that for you!”. And sadly, many of these have been built around the functionality that Spanning Tree provides.

Let’s take the BPDU Guard feature as an example. BPDU Guard is set on an access port or another port that you do not expect to see Spanning Tree Packets (BPDUs). If a BPDU is detected, the switch will usually log a message and send the port into a blocking mode. In the scenarios listed previously the offending port is now taken out of action and the loop is removed. If we have disabled Spanning Tree on all ports then the BPDU will never be sent or received and our little bridging loop will happily continue. Well at least until your switch is a bubbling blob on the bottom of your rack.

Another feature available on most switches is the BPDU Filter. With BPDU Filter enabled on a port the switch will pass all traffic on the port but silently drop the BPDU messages. Now I agree that their are certain times when this feature is useful, such as when interconnecting with a 3rd Party that you “know” can never form a loop with you and you do not want to either learn a STP root from them or go into block due to election issues.

Sadly, our good friends at VMWare love to advocate that we implement BPDU Filter on the ports facing our VMWare Hosts. Unfortunately I have been bitten by loops coming from inside a VMWare environment due to a Microsoft Guest Bridging two vNICs in separate VLANs. A BPDU from the came in from the Physical NIC on VLAN A out to the vNIC in that VLAN and back out through vNIC and VLAN B. Thankfully when this happened, my canary (BPDU Guard) signalled that there was a problem then promptly died in its cage and disabled the port to the VMWare Host. Yes this would have some undesirable effects on all the other guests on that host, but we were alerted to the problem and needed to fix it. In the scenario with BPDU Filter these alerts would have been filtered out and the loop would continue unnoticed.

So what other methods do we have to detect possible bridging loops that do not involve Spanning Tree to be operational? I have the following list as a start to some ideas, and I am looking for others that you might know of too:

• Broadcast Storms
  • Possible Mitigation: Storm Control
• Multiple Mac Addresses on a  Port
  • Possible Mitigation: Max MAC Address restrictions
• MAC Address Flapping
  • Possible Mitigation: MAC Flap Dampening
• High CPU Usage (in some cases)

Mop and Bucket

Yes, I’ve written his post at 2am, but its been something that I have been thinking about for the past 8 or 9 months.

I can see that Spanning Tree doesn’t have an indefinite future, but calling it dead today is premature. If you are looking at fabric technologies or worse still you dont have a new fangled fabric but hate spanning tree so bad that you have just turned it off, then ask yourself how you will detect loops in the edge networks and how you will mitigate them.

Take your canaries with you and let them do their job and don’t strangle them at the top of the mine shaft.

If you do you might just find that the Emperors new cloths are just a sensible pair of slacks!

I don’t know about you, but I am regularly told “You worry too much” or “You don’t need to worry about that”. Sometimes its “What are the chances of that ever happening?”. These are things that Ive heard from many people over the years and the best I can come up with is:

“That’s what you pay me for! Im here to think of the worst case scenario and then mitigate against that.”.

This is usually followed by confused looks from those around me who do not seem to grasp what I am getting at here.

The way I see it, it is my job to constantly be thinking about worst case scenarios.

• “What happens if we lose this device/site/cable?”
• “What happens if we all backups are lost?”
• “What if…”

These sorts of questions are exactly the reason why I fell in love with Networking as a discipline within IT. The very fact that I have the ability to build redundant systems that take serious effort to bring down draws me deeper in (assuming Im given appropriate budget 😉 ),

Why do I build many of my networks like a Service Provider network? Because I have found that these basic design principles are usually the most robust. Configurations utilising OSPF to carry core routing information and BGP to provide end user routes stands up to some serious beating – and it is extensible too!

Why do I cry when I hear vendors pronouncing “With our new Wonder Fabric Technology you can now turn of Spanning Tree”? I cry because I feel that this is sending the wrong message. I have a whole other post coming on that topic, but please people don’t just turn off spanning tree. Are all your edge ports protected? Can you ensure that nobody will ever mis-cable? (And don’t even get me started on VMWare’s view about filtering BPDUs!)

Why do I prefer two stand alone systems providing redundant network services over a single HA unit? Devices redundant power, RE’s and line cards, but with a shared management plane are still susceptible to risk of incorrect configuration causing a service interruption. Switch Stacks, Virtual-Chassis, VSS and what ever other similar technology all suffer from this problem. I would rather a technology such as Multi-Chassis Link Aggregation, Virtual Port Channels, or even utilising VRRP/HSRP or anycasted Services to provide the desired network redundancy. Sometimes this is “harder”, but again – This is what you pay me for 🙂

Mop and bucket

While I know that I am pre-disposed to the negative and pessimistic tendencies and views, but am I the only one who feels that “Worst Case Scenario Thinking” is one of the prime reasons people pay us? My wife could easily plug in a couple of cables and “make intarwebz happen”, as is proven by the millions of home users CPE, but true network design and redundancy comes from thinking about the worst that can happen and how to mitigate against these risks.

Im curious as to the thoughts of those of you out there.

Hey All,

I know I have kind of been MIA for the past 3 months. Ivan reminded me recently that I haven’t actually published a single post since September, so I wanted to give a brief “status update”.

The last couple of months have included the following:

• Network Field Day
• Meeting some of my “Industry Heros” (Read ‘Rockstars’)
• My wife spent 3 months living 12 hours away, teaching at a remote Aboriginal Community.
• I passed two new certifications (JNCIS-SP and JNCIS-SEC)
• Some changes at work including staff restructuring and and office move
• Our Lead Engineer from our Cambodian office was out here for a couple of weeks so I was showing him around Sydney as well as preparing him for his JNCIS-SP
• I had an accident that involved writing off my motorbike (Sad Panda)
• Spending some time training junior engineers, both inside my company and some outside (You really should follow Anthony as he is really starting to prove himself as an up and comer! Also his wedding is in two weeks from today!)
• Taking on 3 new big projects at work
• Christmas and New Years festivities

There have been some other ups and downs in this time and plans and opportunities that have come and gone that provided their own excitement, but the above would probably have to be the key points.

I have a whole pile of half-written blog posts from Network Field Day, and a few blog replies to various “Blog Discussions” that have been going on recently, but I didn’t feel I had the right focus to dedicate to giving them their proper attention to detail. Having a sudden dose of “The Real World” after returning from San Jose was a bit of a bummer 😉 Im going to work on getting out a few of these over the next few weeks, using some of my “down time” to get them squared away.

I hope you all had an enjoyable “Q4” and Holiday Season, now its back to work, blogging and studying!

Ok all, Im going to let out a secret. Long ago when I was a small child (long before I dreamed of being a janitor), when people would ask me what I wanted to be when I grew up I would answer:

“I want to be a swimming pool”.

Cute, no? I guess not, but that never stopped my folks from telling it to everyone of my friends. In fact my Dad put that in his speech he made at my wedding. Usually I would go all red in the face, but denying it was pointless.

This may seem like a weird introduction to this post, but self-deprecation is not a problem to me and on top of that, I am now owning my former aspirations – I’m “taking it back”!

I was sitting in a Juniper training course for the last two days, and during one of the breaks the topic came up about certifications and about people collecting a wide range of certifications and spreading themselves thin. At this point I made the following statement:

“I generally think about our skills and abilities as being a volume of water. We can either have a very deep understanding like a diving pool, or a wider but less deep understanding like an olympic swimming pool.”

At the time I made this claim I was trying to explain a concept in terms I could explain to people. It wasn’t until my drive home and later thinking about writing this post that I remembered my childhood dreams.

Thinking through this line of thought, I started thinking about my own career progression and in particular changes that have come about in the last 12 months. I have worked as a network engineer for the past 11 years, and I can see some stages of growth.

The Kiddy Pool

When I started my professional career in 1999 I had already been using computers for most of my life, and had experience with Linux as well as programming experience. What I soon learnt was that I really didn’t have a lot of experience, but I had a few skills that I could build upon.

My first boss took me under his wing, and taught me a lot about Client-Server computing, hardware repair, customer interaction and regression testing. During this time he was preparing the foundation for where the rest of my career would go.

I learnt quite a few skills in this job, but I knew that to grow I would need to move to another company where I would not be viewed as “The Kid”.

The Lap Pool

About this same time I had a friend who had been working for a consulting company who were also an ISP for their customers, but he was leaving for a new career in the Computer Security industry (Just like everyone was in 2000!). Given my experience with both Linux systems as well as my skills gained from my previous job in Microsoft networks I was able to gain exposure to a varied collection of customers and requirements.

This was the job where I first learn about Cisco equipment. I was handed a Cisco 800 and a print out and told to go install an ISDN service for a customer because nobody else in the company were “Cisco Guys”. This was followed a few weeks later when our upstream transit provider had a network failure and I was forced to troubleshoot our core router using only a console session and a copy of the DocCD I found on a bookshelf. Fun times 🙂

I was working here when Windows 2000 was first released, and everyone had a steep learning curve ahead of them. We had a customer with a new Windows network being rolled out, and our main Microsoft consultant was preparing to do the rollout. Due to unforeseen delays he ended up being away for 4 weeks when the project finally got the go-ahead, so it landed in my lap to implement. Unfortunately I had no notes or documentation from the previous guy, so I had to learn it all on the spot.

This particular company had a very strong emphasis on certification, Microsoft in particular to maintain their partner status. I was able to learn and study here due to the exposure I was given and was able to achieve my Microsoft Certified Professional certification. I also convinced them to buy me the study materials required to gain my CCNA (it is 2001 by this stage).

By now I certainly was gaining a broader set of skills, and they were starting to get deeper.

The Olympic Swimming Pool – Round 1

Not long after I passed my CCNA I changed jobs (for various reasons), and was offered a senior position at the first company I was working for. My first day at this job was September 11 2001 – so this is probably not the most newsworthy thing to happen at the time.

Over the next two and a half years I was able to utilise my skills with Microsoft Networks, coupled with my networking theory and my Linux skills to develop several multi-site networks incorporating all manner of “Directory Services”, “Collaboration” and other buzz word compliant systems. I hired a few friends into this company (one of whom I still work with quite closely).

During this time in “The Olympic Swimming Pool” I was still dealing with a broad range of skills and technology owing mainly to my job role as a consultant. There was systems administration, desktop support, hardware builds and troubleshooting, programming and customer support. The depth of my skills was also starting to get deeper.

Sunbathing by the side of the pool

I knew at this stage that my ideal job was working specifically in computer networking. I mean REAL networking. Routers, switches, blue cables. Not PCs, and very few servers. I decided to take leave my job in early 2004 and I worked for he next 18 months doing various non-IT related jobs. This is also around the time I moved out of Sydney and up to the NSW Central Coast.

During my time away from the industry I really discovered how much I enjoyed working in IT. As with many geeks I couldn’t keep myself away for too long. Thankfully a few days after deciding I should return to IT, I received a call from a friend who had an ISP customer looking to hire a Network Operations Manager – and they were based on the Central Coast.

The Empty Diving Pool

I like to think that by the time I made it to this stage in my career I was standing at the bottom of a diving pool in about waste deep water. I was focusing on Service Provider networking. In particular this was a Wireless ISP, so I was dealing with a whole range of new technologies. Some of these technologies only had a handful of implementations around the world, so the user and support communities were very small.

I was finally away from desktop support, and all of the servers I was looking after were specifically related to the functioning of the network itself. I still had to deal with customer support while we built out our Helpdesk and Support staff. I gained experience with project management as well as working on large network deployments that spanned hundreds of kilometres.

When I started there we had had about 150 customers. Over the time I worked there we grew from that base to over 20 networks across Australia and bought and integrated several other ISPs on the way. Each new acquisition was another technology and “unique” user base. By the time I left there were about 10,000 users across the different networks.

Filling the Diving Pool

After 2 years at this company I was ready to move on, and my friend who introduced me to the Wireless ISP offered me a job working at his consulting company. I was employee #2. Since then we now have a team of network engineers, systems administrators and programmers.

This is the position I currently hold, and during my three and a half years in this job we have been able to land some pretty impressive and interesting projects and contracts. I have designed and managed many ISP networks and evolved my designs of optimal network design in relation to Wholesale providing of end user services as well as scalable Co-Location facilities. I have designed and implemented large networks that only lasted for 14 days during an international event including manning a 24 hour by 8 day media centre for all international media outlets. I have worked on designing networks for Digital Cinema delivery, as well as large Enterprise WAN deployments.

The opportunities presented in this role have enabled me to also take on a new path in my career, one that I never imagined I would be able to do – I have now presented Technical Presentations and Training seminars in several different conferences across the Asia Pacific region. The skills I have learnt during this process are very different from those in my technical background. Each new speaking engagement has taught me something new and I am taking all advice and criticism on board and trying to improve with each new opportunity.

The Future

From my current standpoint, the future of my career looks to be heading back towards the Olympic Swimming pool phase – not as deep but covering a wider range of skills. Maybe not the same skills from the last time I did a few laps in this pool, but certainly broad none the less. I expect to be focusing more on design and team management, and leaning towards supporting my existing engineers in developing and implementing the solutions we come up with.

This very blog, as well as other social media such as Twitter, has also opened up a whole new world of opportunities, and I am looking forward to spending more time focused in this aspect over the next phase of my career. The people I have “met” and the opportunities to engage and interact with people from all aspects of the IT and in particular the networking field has been amazing.

Final Thoughts?

So that has been a somewhat narcissistic look at my career progression so far. In short I feel that we each start our careers with a set of abilities and improve and expand upon those through out our career, but at a point your skills can either go deeper into a specific subset of topics, or broader across a wider range of topics. Your particular career path and goals will determine how and when you will spend time in each of these swimming pools.

For now, I am owning the fact that I am indeed a swimming pool. If I ever become a hot tub I promise to invite you all around for BBQ and a few drinks!

Feel free to comment (or to ridicule my childhood ambitions!).