A couple of weeks back, when we recorded Episode 33 of Packet Pushers Podcast, one of the items we had on the list of topics to discuss was that of multi-vendor networks and the recent Gartner report on the topic.

Due to various reasons this topic was taken off the list, but I still had a few thoughts on the topic so I decided to write this blog post to discuss some of them.

The Gartner Report

In late 2010 I attended a HP Australia executive briefing and breakfast in Sydney. The keynote speaker at this event was Mark Fabbi, and I had the honour (luck?) of sitting at the same table as him during the event. Mr Fabbi was the author of a paper called “Debunking the Myth of the Single Vendor Network”. The presentation was aimed at C-level executives along with the various Magic Quadrants that we Network Engineers love to laminate and stick to the walls of our cubicles!

Whilst telling us (well me at least) something we already knew, he laid out in a fashion suitable for presentation to upper management for discussion and approval to add addition vendors to your network.

Suggestions have been made that vendors with deeper pockets have been pushing the agenda for these reports, but I think that at the end of the day there are compelling reasons not to have vendor lock-in.

The right tool for the right job

At eintellego we specialise in building multi-vendor networks. I like to call this the “Chasing Amy School of Network Design”. Anyone who has seen that movie should know what I am referring to, but basically why should I limit my choice of devices to one particular vendor. If there is a better tool to complete the task I am trying to do then I will not put blinders on because it does not come from my vendor of choice.

By way of an example, for quite a few years we had been selling and supporting the Cisco PIX and ASA platforms for our customers because we felt they offered the best overall hardware firewall solution for our customers. Other vendors had products that were competitive but none of them were compelling enough for us to move away from the ASA. Then in mid-2009 in a space of maybe 3 weeks I had two separate customers ask me on my opinion of the Juniper SRX platform. I had previously looked at Juniper equipment and was quite impressed with the Junos Operating System, but when it came to firewalls I was not impressed with the ScreenOS options. I could have easily just dismissed the new product because it wasn’t from Vendor X who I was used to buying from, but after several recommendations from colleagues as well as requests from customers I decided to take a look. And I am very glad I did!

What I found was a product that met several of the short comings I had with the ASA platform (dynamic routing with redundancy etc), was based on the Junos Operating System, and had quite an impressive throughput and capabilities for the price. Today at eintellego we will recommend and sell SRX solutions to our customers before an ASA solution because we feel that in 2011 it is truly a better option.

Multi-Vendor for the sake of Multi-Vendor

One point I should make clear is that I do not advocate multi-vendor when it is not required. For a long time the Security Industry Best Practice promoted the idea of separate firewalls from separate vendors. While there are sensible reasons behind this approach (vulnerability in one platform will be safe on another), often during the implementation one of the two devices was treated worse than the other. Maybe installing the second box was to achieve a “tick in the box” for some industry security certification, or maybe a choice by a previous admin who no longer is there to champion the cause for the device, but sometimes one of these boxes was modified and updated far less than the other.

Why should we make our lives complicated for ourselves? Why would I force myself to use, say, an ASR and an MX series router as my border routers purely to meet the requirement of multivendor. Valid reasons may exist for this solution, buy “multi-vendor” is not that reason.

I need to learn another System?

I often hear from engineers that they do not want to learn another operating system. I usually laugh at this point at tell them their future doesn’t look that bright. Even within a single vendor you have several new operating systems now, an depending on the direction your network takes you, you may find yourself working with IOS, NX-OS, IOS-XR and/or possibly IOS-XE.

When I train my staff in networking concepts, I try to re-enforce the theory behind the solution or protocol more than the commands required to complete the solution. If you understand why Spanning Tree or OSPF behave a certain way or respond to an event in the network, it does not matter what operating system you are on, all you need to do is work out where to find those options. Knowing the right commands is only 10% of the job!

As an example, we recently rolled out 500+ HP E-series switches for a customer. Prior to this project my experience with the (former) ProCurve range was limited mostly to simple Layer 2 designs with a little bit of AAA and SNMP thrown in. After about an hour of playing on the device, I was already comfortable with the CLI and ready to start “translating” the config I was after into the correct dialect.

ProCurve CLI is very similar to IOS you say? Well how about Junos then? I think we can all agree that Junos is significantly different to many of the other competitors, but it only took me a day or two in the lab before I was comfortable enough to deploy some of these in a semi-production network to start getting some real world experience.

If you’re working in the network field DO NOT be afraid of new tech. That doesn’t matter if it from the same vendor or a new vendor.

My Vendor is the only one who does X?

Personally I think this one falls under the same heading of “The right tool for the right job”, but sometimes people have engineered themselves into a corner and decide to continue implementing the same solutions because to change or re-design is “too hard”.

Maybe you deployed a proprietary protocol and now you are locked into that vendor. In a previous post I discussed how I was not opposed to proprietary solutions, particularly when they are the only option for a solution. Does this mean that you should stick with this solution when new alternatives exist? The same driving force behind the implementation of the proprietary solution should drive the review of new alternatives – The right tool for the right job!

I understand that certain environments have strict design/innovation schedules, and that once a network is built it is extremely hard to get changes made, but we should always be looking for the best way to do our jobs and to design our networks. Don’t be designing networks based on the text-book from 10 years ago!

Wrap up

Multi-Vendor is not scary or hard, as long as you do it for the right reasons.

Feel free to share your comments, flames or multi-vendor nightmares 🙂

Driving home the other night I was listening to the latest episode of the podcast “Coffee with Thomas”. This episode had our host, Thomas Jones, interviewing Steve Chambers of ViewYonder (and also a history of great vendors!). During the interview, Steve made the following comment:

It amazes me that people criticise Cisco for not be standardised on things that are brand new, as if the standards bodies are innovators. That is not their job. They follow up after things have been invented.

This statement took me back at first, and I was about to write it off as protecting your own team, but the further I drove (I have a very long commute – about 100km each way) the more I thought about this statement and the evidence – both historical and current.

As network engineers we often love to get into religious arguments about which technology is better, and almost invariably a comparison will be made between a vendor proprietary solution and an open standards solution. Whether that is EIGRP vs OSPF, HSRP vs VRRP, or FabricPath vs TRILL they all seem to come down to a core need:

I have a need right now, and the standards bodies will take years to agree. Do I innovate and press forward with what I can do now, or do I wait?

When HSRP was introduced the market had a need for redundancy of Layer 3 gateway devices. A vendor saw a problem, thought of a solution, and implemented the solution without waiting for the rest of the market to catch up. And I firmly believe that this is in the best interest of the market. Once HSRP was recognised as a viable solution other vendors and standards bodies worked on bringing to market an industry standard protocol. Admittedly Cisco could have done many things to improve HSRP (proper load balancing is just one idea, though they brought similar options into GLBP), but they brought a solution to market when it was needed.

Zoom forward to today. The market today is seeing a push towards large virtualised domains (dare I say “Cloud”?), and as a side effect of this we (currently) need to implement these solutions using large Layer 2 domains. As has been the history of Layer 2 this comes with the “nightmares of spanning tree”, that seems to wake up so many admins at 2am in the morning in a cold sweat.

A problem exists now, and a solution is needed now. As is usual with standards bodies there is much debate as to how to move forward as everyone has their own opinion. In fact, which standards body do we look to? IETF with TRILL or IEEE with SPB? Life is slow in the land of standards, and while the boffins are battling it out there is a market place looking to roll out a solution to meet a need in there own environment.

So who is tackling these problems today? Well, like in the past, some vendors are claiming “There is no standard to move forward with”, or “We will support it when there is a standard”. This is not innovation – this is doing the same as everyone else. Im honestly not sure how I feel about a tech company who will only do exactly what everyone else is doing. Maybe in the sub-$50 SMB switch and router market sure – but not somebody I have to justify budgets and ROI and staking part of my career on!

Cisco’s FabricPath and OTV solutions may not be standards compliant. They do appear to have taken several of the nice features of the proposed standards and added some other features that will improve the Nexus platform overall, but they are not backwards (forwards?) compatible with either of the proposed standards.

Is this a bad thing? They have gone to market with a solution to a problem we have now. From the accounts I have heard the technology appears to do what it promised (even if it was behind schedule). With the standards “almost ratified” should they have waited so that their competitors could go to market with a solution at the same time?

The main thing I would like to hear from Cisco is that they will also support the standards compliant version when ever it finally gets ratified.

Im sitting in a hotel room in Hong Kong right now, with a whole city to explore so I will leave it here for now.

As always, thoughts, flames and comments welcome.

As some of my readers may know, I have spent the last week and a bit in Phnom Penh (Cambodia), setting up the new office for my company (eintellego). I had never been to Asia before this trip – outside of a couple of lay-overs in Japan in 2002 and 2003, or the 45 mins sitting on the plane in Bangkok Airport on the way to Abu Dhabi last November. I had no idea what to expect when stepping out of the airport.

I had no idea of the adventures that awaited me. I could tell the story of the motorbike rider attached to a medical drip riding down the road on the way from the airport. I could tell the story of the three legged dog standing next to a guy eating what appeared to be a “drumstick”. I could tell the story of being surprised by a Gibbon (Hack the Gibbon!) while walking to the office. I could tell the story of shooting AK-47s and M-16s at a Military base (Good prices!). Maybe somewhat more interestingly I could tell the story about the border war that started on the day I arrived here when the Thai Army “invaded” Cambodia over a disputed bit of land and a 1000 year old temple on the border of these two countries. And none of these even come close to the horrible abuses in translation on almost every menu I have seen since we got here, but alas this is a technical blog so I should focus on some of the things more interesting to my readers.

The thing I found immediately obvious to me as a westerner in Phnom Penh, was the contrasts in network design. Each street was lined with power poles with what at first sight looked like messes of spaghetti cabling. Upon closer inspection, and a week of reflection, there started to be some order to the chaos I was seeing.

At the same time back in Australia that we were debating VDSL implementation over existing copper telephone lines through to how/when/where/who of the national FTTH project (called the “National Broadband Network” (NBN) in Australia), the Cambodian government made a simple rule:

“$2 USD per block, per cable payable to the government for access”

An important point to make from the above – That is per cable! Whether 12-Core or 100-Core “$2 per block”. This simple rule made it affordable for the various Carriers and Telcos to deploy a FTTH project across the city of Phnom Penh and to provide end user services for $10/month (possibly cheaper but I didn’t look too hard).

Existing copper telephone networks were either non-existent or so poorly maintained as to be unusable for anything other than voice in many areas of the city, but at the price of deploying Fibre and Coax based solutions, this hardly seemed like too much of a problem.

Another thing I noticed – Almost every cafe and restaurant had some form of Free Wifi to customers. This was usually an Open Access point, or an encrypted SSID and the wait-staff would tell you the key (Note: Ask them to write it down for you! Saves a lot of time in mis-typing the key when you could be more productively uploading pictures of your food to Facebook and Twitter!). In Australia, those places that offer WiFi usually either had “50mb total in a 24 hour period” or charged a not-so-nominal price for access. I have to say, this aspect certainly made my me happy.

iPhone + Skype + Free_Wifi = @MrsJanitor + Happy

I would also like to take the time in the post to answer the most common question I received from both friends and collegues regarding this trip:

“Why are you opening an office in Cambodia?”

I guess there are several correct answers to this, but the truth is the amalgamation of several facts:

• We knew of several businesses working over in Cambodia working to improve the quality of life for the people there
• Through some of these contacts we have met many Cambodians who have amazing potential and just need the opportunities
• Cambodia gave us a presence within Asia that was no more than 2 hours flight from over 10,000 potential customers.
• There is an organisation called CIST who, in partnership with several large Corporations (including Cisco), go out to the provinces of Cambodia to find people with potential and train them in many aspects of IT including: Microsoft Technician Training, Linux Administration Skills, and Cisco Academy training to the CCNA level.

This last point alone has proven to be a really eye opening experience. When we advertised for our current positions (mostly HelpDesk and 1st level Network Ops), we received nearly 100 applications from people who had been through this training. We did two rounds of interviews – 1st round in December, and the 2nd round last week.

I wasn’t present for the first round of interviews, but the candidates we interviewed last week were great. I have them the usual network engineering and Linux admin questions I would ask any candidate back in Australia – no softball questions! They managed to answer each question beyond what I had expected and were able to discuss various protocol level concepts as well as configuration and management ideas to show to me that they actually understood what I was asking. All of this, while explaining it to me in something other than their native tongue! I had planed to bring on an additional staff member, but hired two more engineers after that round because I was so impressed.

All of this, and the fact that the Khmer people have such a great personality and sense of humour, so I felt like I was at my second home for much of my trip.

Yes, I understand that the above advantages could be said about many developing countries, but we had the opportunity and the ability to get on the ground in Cambodia. For more information regarding the how and why of Cambodia, Skeeve Stevens is working on a post on his new blog. Hopefully it will be published soon.

Im not sure if there will be many comments on this post, but feel free to send me any questions etc.

I recently gave a presentation to the New Zealand Network Operators Group (NZNOG) 2011 conference on “Data Centre 3.0”. During my research over the last 8 months coupled with the fact checking I had been following up during the creation of the slides, I kept asking myself:

“Would we need all these protocols if we, as an industry, had made better technology implementation decisions?”

I understand the background and requirements for some of the different technology proposals, particularly Layer 2 Multi-path and the various Data Centre Bridging (DCB) QoS standards, but I cant help but feel that we are trying to bring features of the higher layer protocols down into Layer 2.

Back when I started studying networking (probably around mid 2001 when I first obtained my CCNA), the CCNA curriculum was quite clear on the OSI layer and how each layer had a very particular purpose, with clear functional definitions: