One of my recent forays into Increasing the Awesome has involved learning about NETCONF and the Python programming language. I was lucky enough to spend some time with Jeremy Schulman during my trip to Sunnyvale for the Juniper Ambassadors Summit, and he introduced me to the new py-junos-eznc Python library he has been working on. I had spent a little bit of time earlier in the year looking at the original Ruby library, and I was amazed at how much thought had been put into this new library – obviously Jeremy’s learned a lot on the way!

 An Impatient Start

Let me make a couple of things clear right from the outset:

1. I am not a programmer! Yes I have written the odd script here and there in the deep dark past, but I am by no means a programmer. All of my scripts have been about automating some task I had to do. As long as it worked, I didn’t care how efficient or pretty it was – it did what I needed.
2. I have no intention of becoming a full time programmer! I like being a network architect and I like building and playing with network toys. All I want is the ability to make my job easier, which leads me to my last point…
3. I am lazy! I don’t like repetitive work. I would rather do something once or twice and move on. Computers are here to do the mundane stuff for us, so we can create more awesome. I would rather write scripts for other people to do it next time instead of bugging me about it.

So with the above three stipulations in hand, I started to learn Python. Now, when I say “started”, I literally mean a week ago. I already knew the basics of loops and conditionals etc, but I couldn’t read a lick of Python this time last week. I tried the various online tutorials such as over at Code Academy and Learn Python the Hardway, but I knew the only way I was going to get my head around Python was to jump in and just start coding the working I had on my plate.

A Looming Project

I have a project starting in the next couple of weeks that is going to entail configuring about 50 switches with mostly the same configuration, followed by customising port layouts etc. Now just time involved in unboxing all those switches, cabling them up in the lab, firmware upgrading them, prepping the config and doing burn in tests can take a couple of days.

I knew this would be a prime candidate for “automation”. I had better got off my butt and work out how to do this!

So I contacted Jeremy and we hopped onto a Google Hangout where he was able to unleash 2.5 months of “Python knowledge” on me. It was 11pm for me and 5am for him, and it was so much fun 🙂 Over the course of our conversation we discussed a couple of the core concepts and data structures structures in the Junos-EZ libraries, and Jeremy discussed an idea he had to make extending the existing code base even easier. When I woke up the next morning, he had already committed it to the Github repository.

The Basic Structures

There is a whole heck of a lot in the Junos-EZ libraries, but by only understanding the very basics of the following components, I was able to create some pretty awesome scripts – with only 6 days of Python experience and about 24 hours with the Junos-EZ libraries 🙂

Device

The Device object represents a particular network device. When you call the .open() method, the script establishes an NETCONF over SSH connection to the device. From here you can make any of the NETCONF RPC calls you want against the device.

Tables

Now one of the great things about this library is that Jeremy has gone to great lengths to ensure that the average user of these libraries don’t need to understand the deep internals of the code to be able to add functionality. Tables are a prime example of this.

A table essentially represents all the data collected for a certain RPC request, sorted and keyed on a particular set of values and presented to the user as a collection of native Python data structures. The end user only needs to describe the data sets in YAML format without knowing the Python objects below.

The Table I use in this example is called PhyPortTable, shown here:

[snippet id=”38″]

Views

A view is applied to a table to create a custom combination of the data in the table. As such, a combination of table and view definitions can be created with just a handful of lines of code yet scale infinitely.

The above PhyPortTable example calls the PhyPortView filter as shown here:

[snippet id=”39″]

The Opportunistic Script

So there I was onsite working with a customer yesterday afternoon, and I was asked –

“Can you please get me a list of all of the ports on the following switches, and tell me the current operating status and when they last flapped?”.

This request required looking at 5 switches and about 300 ports. Now what was I to do? Within the last 24hrs I had learned enough to know I could probably whip up a piece of code that would generate the required output. And it turned out to be relatively simple.

The code I wrote is shown here:

[snippet id=”36″]

This is by no means the prettiest code on the planet, but remember our goal was to just remove the mundane so we could increase the awesome, not become a programmer.

• Lines 1 and 2 tell Python to load the relevant libraries.
• Lines 4 and 5 make the NETCONF connection to my lab switch.
• Line 7 is where “the magic” lies. This line requests that the collection of data be returned as defined by the “PhyPortTable” and assign it to a variable called ports.
• Line 9 prints a simple CSV style header to the console
• Lines 12 – 14 loop through the collected data set and print the values for “oper” and “flapped” (names as defined in the view) in CSV format.

The output of this script is shown here:

[snippet id=”37″]

I was able to run this script across each of the switches I needed to query and provide an “Excel Version” for the customer who asked for the information.

Mop and Bucket

This post only covers an extremely simple example, but most of what we really want to stop doing manually every day is not much more complicated that the above example.

I am going write a follow up post in the next couple of days to discuss how within 24hrs of starting with the Junos-EZ library I was able to bring LLDP discovery functionality to the library with no real skill other than the ability to cut, paste and pray my way through it.

Anybody can work on this stuff, so get out there and start coding!

There is one thing that anybody who has been in a room with me longer than 5 minutes can tell you – I am not a smart guy! I have lots of smart friends. I am not one of them. Sometimes I feel like I’ve done more stupid things, more often than I would like to admit, and its only that I have been stupid enough often enough that I have eventually learned “dont do that!“.

A couple of things have happened over the past few weeks that made me think about “The Smartest Guy in the Room”, and I thought I would share a few incomplete thoughts on the matter.

As a Consultant

Back on June 18, Matthew Norwood (who I would nominate for the award of “Nicest man in Network Blogging and Puppeteering”) wrote this blog post about consulting. In this post he talks about how as a consultant sometimes you have to accept that you are not the smartest guy in the room. At first this may seem odd, especially because usually our customers are paying us lots of money because we are experts. Funnily enough, on consulting engagements my job is more “I know where to find how to do that thing you are expert in”.

The most recent example of this was a few weeks back I was working on a Proof of Concept lab with a customer to see how we could integrate a series Juniper MX routers with their existing ASR9000 MPLS topology. Thankfully these sorts of things are fairly standard offerings on both of these platforms but this particular customer had their entire network built on Multicast over MPLS Point-to-Multipoint LSPs. Now when you consider that most people consider Multicast a “corner case” in networking, then these additional requirements made it a corner case inside of a corner case! On top of the fact that the customer engineers I was working with did this day-in and day-out, there was no pretending I was the expert. Instead my job became “I know where and how to make the Juniper MX do what you want”, and that was how I proceeded. And I can tell you this was one of the most enjoyable engagements I have had in quite some time.

As an IT Guy

It’s been stated many times that when you look at a group of IT guys, that most likely many of them were the “nerd” or “geek” growing up and were used to being the smartest guy in the room. As such, we often get interested in all sorts of technologies and ideas and run off on different tangents. I work with a couple of amazing guys who seem to tackle any odd-ball concept that comes to mind – like rigging a relay in circuit with the door control system so they can control the door with their pebble watch or simply building 3D printers to allow them to make all the toys they dream of. And often times they love to do this towards the end of the day/week when we are wrapping up units of work, and I know many late nights have been spent back in the office hacking away on something crazy “just because it can be done”.

Sadly, occasionally we take on tasks that are not our core competencies. This isn’t to say that we are unable to do the work, but simply it is not our strength and it can take much longer to resolve than it would a suitably trained / skilled engineer. A recent example of this was an incident that recently caused our team of network engineers to remember what it means to be a “Windows Administrator” and all the bits and pieces required to get some systems back online. We were more than capable of completing the work, but not necessarily in the most efficient fashion. We called in “a few favours” from friends with the required skills to sort this out, but  sometimes it goes against our gut feeling to seek advice. In times like this we need to accept that there are in fact people with strengths in these areas, and to ask for help is not about admitting defeat.

As Full Stack Engineer

And this leads into my last incomplete thought. Some time last week I engaged in a rant-fest with Paul Gear regarding the topic of breaking down silos within IT and what it means to be a “full stack engineer” (for lack of a better word). Admittedly the rant was primarily around how its not easy to break down silos when we still all refer to each other as “Server Guys”, “Storage Guys” and “Networking Guys” (nobody talks to/about “Security Guys” because they “just say no“), but over the course of the conversation I developed a few opinions (as I am known to do). And these are my thoughts on this…

Recently:

• I have started learning to program in Ruby, as well as learning about OpenStack. I have spent a fair amount of my “down time” learning about coding in Ruby and learning how to automate many of my regular tasks. I have picked up a few libraries developed by the guys at Juniper (mad props to Jeremy Schulman) and started learning more.
• I have started getting to grips with service automation using Puppet, and investigating how end-to-end provisioning of both server and network resources as a single process can improve delivery and satisfaction.
• I have built an OpenStack lab (as part of my beta testing of the Juniper Contrail product), and started to learn how to spin up and down workloads on demand and again how to integrate that with the various network requirements.

Do I want to be a programmer or a guy running servers all day? No, but I do want to be able to “talk the language” of the guys who do. There is great benefit in this ability not just as a consultant but as a member of a “Silo Free IT Team”. My view is that in the future, we will have blended teams of engineers, each with an understanding of the various aspects of the stack (Server, Storage, Security and Network), but we will still specialise in one or two of those areas. Working in such a team will be all about admitting you are not the smartest guy in the room and simply admitting “Yes, but I do know this part pretty well. Let’s come up with a solution”.

I dont see specialisation going anytime soon. Even within each of those four areas of IT there is a world of sub-specialisation that can be entire careers for people. The key is to have an understanding of all of the areas involved and know where best to apply your learned skills and experiences. *This* is what makes the future of IT teams… not a group of “jack of all trades” engineers, as some people interpret the idea of “breaking down silos”.

Mop and Bucket

I’ve learnt over time to know what I know, and to ask for help when I dont. Sometimes it takes me a while to get there, but almost every time that I do things get better 🙂 There is too much happening in IT and too much change in our industry to remain an expert at anything and everything for long. Widen your perspective for high level understanding, and learn where you can best apply and extend your skills.

If nothing else, it will make for an interesting and progressive career!

Earlier today Ethan Banks wrote a really good blog posts about “Thoughts on Working as a Consultant for a VAR“. I found his point of view quite interesting and I will say I can understand his points. I can also say that I would rather be a consultant than a full time engineer at a customer site. As a little bit of background I have spent most of my career working as a consultant. I did do a two year stint as network operations manager for a wireless ISP which itself was quite fast paced, but other than that Ive work as a consultant in one form or another.

Maybe I have ADD, maybe I just need to focus, but I have found that constantly having different projects going allows me to satisfy these tendencies. I feel I work better with more than one thing to occupy my time. I see friends who work for enterprise customers who spend their days submitting change requests that third party support companies fulfil, or spend months writing detailed design guides for projects that inevitably get canceled and all that time is spent without getting to touch the things they got into this industry for. Which brings me to my first point…

Toys, Toys Toys!

Im 32 years old, and I still love toys. Sometimes I think the only reason why Im in this industry is the ability to play with expensive and complex toys. And I get paid to do it! I enjoy the challenge of learning a new piece of equipment and deploying it into a customers network. I enjoy learning all the new opportunities that a new software update or a router line card can offer to my upcoming network designs (my sales guys also enjoy being able to sell those things to existing customers… bloody sales drones!).

I have worked at my current employer for 10 months now and in that time I have worked on many projects, not the least of which has included rolling out the second Juniper QFabric deployment in Australia (and one of the first in the world), delivering security solutions based on high end data centre SRX devices including IDP, SSL Inspection and Proxying, a relatively large Juniper MAG deployment across multiple sites with failover, and numerous network audits across many verticals including finance, utilities, enterprise and service provider networks.

The other aspect to toys is the ability to get your hands on all the right equipment to learn and play with *before* you need to deploy it. This really comes into its own when you have…

The right employer

Ethan’s posts hinted at being burned by his previous employer. I know I can attest to that over the past 15 years Ive been in the industry. Sometimes its the customers, sometimes its the employer. The right employer will understand your career aspirations and will endeavour to provide you with all the resources you need to be successful during both your time with them as well as your career beyond them. The right employer understands how often engineers change jobs in this industry and that what goes around comes around.

Your employer needs to understand that certification is beneficial to him, but more importantly that you as the engineer need to understand the content of the exams you are passing. Perpetuating the degradation of industry certifications by promoting brain dumps or becoming a “puppy factory” for certifications churning out sub-par engineers devalues both the certification as well as your ability to sell your services to customers.

The right employer will cover your expenses for gaining certification, and provide you with the resources, both equipment and time, to actually complete these career goals.

If you’re lucky your VAR will give you access to both pre-sales and post-sales implementation work giving you a wider view point of the network life cycle as well as building up skills that end customer engineers will never get to practice. And you will do that many times a week!

Honest about the downsides

• You’re supposed to know everything: I agree with Ethan that this can be stressful, but it doesn’t take long to pick up the habit of either being one google search ahead of the others, or as I often say “Not knowing the answer, but knowing where to find them”. If you work for a consulting firm that doesn’t require your customers to buy some form of vendor TAC support then you are further against the wall and you/they are not operating in your customers best interest. *Always* have the ability to escalate.
• You’re a commodity: Yes you are a commodity, but you also are (should be) “in demand”. If you are doing your job well, and your employer knows how best to leverage your skill set appropriately (not always in your control) then this works in everyones favour. If you are not doing your job well then you shouldn’t be a consultant, and if your employer doesn’t know how to effectively sell your skills you should be looking elsewhere. You will get abused, ignored and eventually get bored and quit or made redundant. Scary truth, but its better to move on and find something you are happier doing.
• You’ll sacrifice your body if you’re not careful: Well I’m not even going to pretend to deny this one. I am living proof that ignoring the health aspects of your life will creep up on you, but Im not convinced this is specifically related to VAR lifestyle. Any high stress, fast paced career can do this to you. You just need to keep these things in check. On a side note to this one, I’ve taken up a Paleo/Primal lifestyle for the past couple of months and Ive found wonderful improvements to my health without major sacrifices (in both time and diet), but thats a whole other blog post!

Mop and Bucket

Yes working for a VAR can be hard work. It can be fast paced and involve a lot jumping between contexts, but if you are like me you just may find that the very draw card to this career option.

Sometimes I dream about working on a single network and finishing all those “little things” that I wish I could implement for all of the networks I build. Then I spend two weeks onsite at a customer and all I want to do is get out of there.

If you are willing to invest 12 months of your life to “give it a go”, you will learn far more in that time, and touch a wider array of equipment than you could hope to touch in 3 technology refresh cycles at an end customer.

And this is what keeps me coming back to work tomorrow!

Yesterday was possibly one of the proudest days in my professional life. My very good friend Anthony Burke sent me a text message just after 9am to tell me that he had passed his JNCIA-Junos exam. This is a close tie with when he earned his place as a delegate to Network Field Day 4 back in October.

So why am I so proud about this? Well Anthony and I have become great friends over the past 18 months all through the magic of social media. One day for no reason (as is the way of things) we just started talking. Tweets became skypes and in January, even though we had only ever met face to face once, my wife and I drove the 10 hour trip to Melbourne to attend his wedding. (Sitting at table answering the question of “So how do you know Anthony?” with “Oh. From the Internet” was a humorous experience)

So how is this different to any other random person on twitter? Well very early one I realised how eager Anthony was to learn more about network and to soak up knowledge from those willing to share it. I’m not entirely sure if he just caught me on the right day (Im pretty sure I commenced our first Skype with a rant about something he never saw coming because I needed to vent at somebody). Over the course of the coming months we maintained a Skype IM dialog open most of the day discussing various networking, technology and career issues.

I spent time helping him learn new topics and sharing any possible pointers and tidbits I had that might help him with what he was working on. In return he acted as a sounding board to bounce ideas off and offering up suggestions. Over time I noticed his advice back to me started to surprise me with how much he had already learned.

In my time as a senior engineer and various management roles I have been required to assist junior staff with both their technical and their career development. I always found this to be a difficult task because I was trying to balance what I wanted out of them as co-workers with what it is they wanted to do with their own careers. I had financial incentive to push them down the path that benefitted me more than it did them. Each time they moved away from “my chosen path” there was a feeling of disappointment or failure (on my part). Even when I worked with the staff to clarify where they wanted their careers to go, I felt they were trying give “the right answer” or at least the one they thought I wanted to hear.

I have since learnt that the most personally rewarding mentoring system for me is one where I get no personal benefit from the success or failure of the other participant. Helping them make the decisions that are right for them without those choices directly impacting my day to day work life has been quite a different experience. I am better able to frame my advice to give both the benefit of my own experiences coupled with knowing where they want to go professionally.

Mop and Bucket

In a recent email conversation, reference was made to the the gutting of the middle ranks of IT. We have a lot of junior staff and a lot of highly skilled engineers, but we are starting to see a thin mid-section (something my mirror cannot attest to). Whether this is due to the focus on specialisation vs generalist IT or not is uncertain to me, but I have seen evidence to suggest this accurate (or becoming so).

So how do we move junior engineers onto the path of becoming journeymen and future experts? I suggest that each of you should keep your eyes open to “up and comers”. The ones who “have it”. You will be surprised when and where you find them, but they are everywhere. Take the time to help them out when possible. Answer the questions that they ask. Take time to get to know their strengths, their weaknesses and their career aspirations. Put them in contact with your contacts and help build their network of resources. Everyone “knows a guy who knows that thing”. Pass it on. Our contacts are not just secret tools to make us look good.

Once they become a journeyman engineer, encourage them to also look for people to mentor and repeat the process. I have found this process to be very rewarding for me personally, and I plan to continue doing it. Having a mentor (or several) above you and below you can do a lot to encourage and maintain your own growth.

When I first started talking with Anthony, he had passed his CCNA and started studying for his CCNP. Since then he has continued on with increasing his Route, Switch and Security skills. He fell in love with the ASA (urgh) and despite much encouragement I couldnt talk him around to playing with Junos. In late October he finally got his hands on his own SRX110 and started using Junos for the first time. I just over a month he has learnt more than I learnt in my first 6 months playing with Juniper kit. He passed his JNCIA-Junos yesterday, and apparently preparing for his JNCIS-SEC.

Taking on new technologies and really understanding them so quickly? This is why I am proud.

If this sounds like a bit of a love letter to a friend then so be it 😀

And Burkey…. #FHP 😉

Well it appears that I am well on my way to holding onto the title for two years running, and once again, I haven’t posted anything for a couple of months. There have been a lot of changes going on around here in the past few months which have really taken me away from my “janitorial duties”.

There are three key items that I would like to take the time to update everyone on:

1. All Good things…

The first piece of news, and possibly the biggest, is that in March I handed in my notice as the CTO of eintellego. I have spent the last four years working with an excellent team of engineers and support staff, helping to build up a really impressive list of projects and achievements. I must point out that I myself was only one player in this effort amongst a team of driven and talented engineers.

During my time here I had the chance to work on building Service Provider wholesale networks, Enterprise WANs with 100’s of sites, leading edge Security implementations, Video Content Delivery networks and not to mention a range of emerging “Cloud Provider” solutions. I also had the opportunity to work on my skills in management, sales, and human resources (Glad I’m an engineer!). These opportunities have been a large influence on shaping my career to date as well as helping me plan path into the future.

I would like to give a special thanks to the entire eintellego team, especially Skeeve Stevens who took a chance on me and gave me to opportunity and freedom to grow my skills and develop my own “style”. Many closed door meetings were held between the two of us to “thrash out” the best solution to a problem, and this was certainly an exercise in seeing things from every angle possible (even if it sounded like a battle royal to those on the outside 😉 )

I wish the best of luck to everybody working there, and the best for the future.

2. Of New Beginnings

In late April, after taking my wife on a long over-due vacation, I took on my new position as Senior Network Engineer at ICT Networks. ICT is a Juniper Elite Partner based out of Sydney, focusing on Data Centre and Security deployments for some of the largest enterprise customers in the Australian market.

Even in just the few weeks I have been working here at ICT so far I have already gained exposure to a range of Juniper products that I hadn’t used before, and am learning a lot from engineers who have been working with these products for a long time. Some of the projects I have on my roadmap for the next couple of months are going to provide quite a bit of interesting blog content based around Data Centre and Security implementation and best practices. STAY TUNED!

I’m still finding a bit unsettling to come from a situation where I knew pretty much everything about all customers and all networks to being “the new guy”. Going from a list of a million things to complete and not enough time to complete them to all of a sudden trying to get my hands on as many new projects and I can trying to learn as much as I can 🙂 It’s a good feeling but Im anxious to jump in and play 🙂

3. Of Getting Back On The Horse

Some of my readers may remember that I was involved in the beta program for the JNCIE-ENT lab in August of last year. Needless to say I didn’t go as well as I had hoped, and with the range of things occurring in my personal and professional life over the past 8 months, I didn’t feel up to the challenge of trying again.

Well the good news is that after my vacation and the change in lifestyle, I have gotten back on the horse. My JNCIE-ENT lab exam is booked for 9am this coming Friday (May 25th). This is another reason why I haven’t been visible much lately because I have been trying to study as much as I can to hopefully “bring home the digits”. I also have to get right back into my CCIE R&S studies as the end of September marks 18 months since taking the written!

Now that I have put it out there publicly in this post, Im going to have to work my hardest to pass 😉 Hopefully I will have a happy post in the next few weeks.

Mop and Bucket

Well I hope that explains some of the reasons why things have been quiet around here, though possibly not a great excuse! As I hinted at above, there are some very good blog posts coming out of these upcoming projects that I hope you will all enjoy.

Until then… may all your packets be forwarded expeditiously!

Recently I have been working on a solution to provide Wholesale Access to hosted VMs. Several of my customers have “Cloud Environments” – call it IaaS, virtualisation, a fad or whatever, this is something that I have been asked to come up with a solution for more than once.

To explain the requirements outlined in this article, I should give a little background on the design requirements and constraints. For some of my customers the standard build is to include 2x VLANs for each Customer – a Live VLAN, and an Internal/Backend network. If the customer has more than one VM in a cluster then all servers will share these VLANs. Unfortunately this quickly runs through the available VLANs (And we love that the Nexus 5k only supports 512 vlans). This limited the number of customers in a single VM cluster, due to VLAN limitations inherent in Data Centre switches.

To scale out from these limits, and dealing with network growth and evolution additional clusters have been built and added. Each VM Cluster is self contained, including its own Compute, Network and Storage infrastructure. Within a single cluster we have allowed the use of the full range of VLANs from 1 through to 4094.

As happens with many successful ventures, customer numbers have grown considerably and with it came the requirement for third parties to be able to host their own customers within these VM clusters. I call these third parties “Wholesale Customers”, because they are buying services in bulk from the Service Provider and then splitting them up between their own distinct customers.

As a “value add” we have allowed these wholesale customers to interconnect directly with the switching fabric to we trunk the VLANs across to them as needed. This allows the Wholesale Customer to add their own services to the VLANs, or to integrate these VMs inside customer WAN VRFs etc.

The problem comes when the Wholesale Customer has VMs in more than one cluster. Do they need to order a cross-connect to each cluster or can we aggregate VMs from all clusters across a single link? Due to the VLAN reuse policy in action within the clusters, traditional switching and trunking would cause an issue as it is possible for a wholesale customers separate VMs to exist in the same vlan-id in different clusters.

Config Snippet

interface GigabitEthernet0/1
 description Trunk to Cluster 1
 port-type nni
 switchport trunk allowed vlan none
 switchport mode trunk
 service instance 1 ethernet
  encapsulation dot1q 200
  bridge-domain 3600
 !
 service instance 2 ethernet
  encapsulation dot1q 201
  bridge-domain 3601
 !
interface GigabitEthernet0/2
 description Trunk to Cluster 2
 port-type nni
 switchport trunk allowed vlan none
 switchport mode trunk
 service instance 1 ethernet
  encapsulation dot1q 200
  bridge-domain 3700
 !
 service instance 2 ethernet
  encapsulation dot1q 301
  bridge-domain 3701
 !

interface GigabitEthernet0/3
 description Trunk to Customer A
 port-type nni
 switchport trunk allowed vlan none
 switchport mode trunk
 service instance 2 ethernet
  encapsulation dot1q 300 second-dot1q 1-4094
  rewrite ingress tag pop 1 symmetric
  bridge-domain 3600
 !
 service instance 3 ethernet
  encapsulation dot1q 400 second-dot1q 1-4094
  rewrite ingress tag pop 1 symmetric
  bridge-domain 3700
 !
interface GigabitEthernet0/4
 description Trunk to Customer B
 port-type nni
 switchport trunk allowed vlan 3601,3701
 switchport mode trunk

A Solution is needed

As Im sure many of my readers know, we could easily use dot1q-tunelling (otherwise called QinQ) to encapsulate traffic from each cluster inside an outer vlan (Referred to as a Service VLAN or S-VLAN). The downside to this solution is that each wholesale customer would need a separate QinQ port per cluster, and this could get very unwieldy very quickly. I needed to come up with a solution that would scale better than this, without requiring needless wasted ports within the switching infrastructure.

We outlined the following requirements:

• Allow full VLAN range use within a cluster
• Allow each the same vlan-id used in different clusters be sent to the same wholesale customer
• Allow for future integration with the proposed MPLS routing between clusters and DCs
• Be economic on scale and reduce wastage
• Allow the same wholesale handoff infrastructure to be used by multiple customers

I knew that I could accomplish these requirements using a Service Provider technology called “Selective QinQ”. Essentially this allows a single incoming port to determine the S-VLAN to according to certain attributes of the incoming packet. With this in mind I went through the various vendor offerings on this front, and in the end we settled on using the new Cisco ME3600-X Metro Switching platform.

Enter the Cisco ME3600

The Cisco ME3600 is a new offering in Cisco Metro Ethernet series of switches. The Metro switches, as their name implies, are aimed at Service providers building large cross-city ethernet networks. These type of networks have very similar requirements to those I listed above – in particular facilitating the carriage of distinct customer traffic across a common backbone while maintaining scalability and economic hardware investment.

The ME3400 has long been the bastion of this family of switches, being used all around the world in the basement of many buildings providing access into Service Provider networks, so naturally this was the first place I looked for a solution. We quickly determined that the new switch on the block, the ME3600, could meet both our current and future requirements so we ordered a pair of these for our tests.

This new switch introduces some new features over and above the ME3400, in keeping with the definitions of the Metro Ethernet Forum (MEF). In particular it utilizes three particular features that might be new to engineers used to working with traditional Cisco Catalyst switches.

Ethernet Virtual Connection (EVC): An EVC is a logical collection of interfaces within a service provider network that is linked to a particular end customers network. These can be either point-to-point or multipoint-to-multipoint.

Bridge Domain: A Bridge Domain is broadcast domain that is local to the switch, but is not limited to a certain VLAN. This allows a set of Service Instances on ports to be treated the according to a defined pattern.

Ethernet Flow Point: An Ethernet flow point is a logical flow or set of traffic within an EVC and Bridge Domain. On the ME3600 an EFP is represented as a Service Instance on an individual interface. Service Instance numbers are unique to the interface and do not relate to the same instance number on another interface.

The Design

Now essentially my design was to take the ME3600 switch and turn it on it head. In my design WE were going to be the end customer, and the wholesale customer was going to be the “Service Provider” or upstream side. As is shown in the diagrams above the Layer 2 network from each cluster is configured with a trunk link into the ME3600. On ingress the switch would match based on incoming VLAN and assign traffic to a bridge domain associated with the wholesale customer. One bridge domain, per Wholesale Customer, per Cluster. On egress the switch will assign a set S-VLAN as determined by the bridge domain of the traffic.

If you refer to the Config Snippet above, and look specifically at the config for Interface GigabitEthernet0/1 (The link to Cluster 1) you will see two seperate Service Instances defined:

• Instance 1 is configured to match incoming traffic with the 802.1Q vlan-id of 200, and assign it to bridge domain 3600.
• Instance 2 is configured to match incoming traffic with the 802.1Q vlan-id of 201, and assign it to bridge domain 3601

Most of this is straight forward and as you can see, Interface GigabitEthernet0/2 is configured in a similar fashion. You should note from the diagram that Customer A has two VMs each of which is located in “VLAN 200”.

The interesting work happens on the ports heading to the Wholesale Customers. There are two possible options, and I have shown both of them in the Config Snippet above.

1. As is shown on Interface GigabitEthernet0/4, you can simply configure the outbound port as a trunk. In this case all traffic in the bridge domain will be encapsulated with the S-VLAN tag matching the Bridge Domain id. In the case of Customer B this would be vlans 3601 and 3701.
2. The other more flexible option (and the one I have chosen to go into production with), requires a little extra configuration but is a lot more flexible. As shown on Interface GigabitEthernet0/3, I have configured two Service Instances – one for each Bridge Domain. Service Instance 2 is linked to Bridge Domain 3600 and is configured to take an packets inbound on the port with an S-VLAN of 300 and dump it into the Bridge Domain. The rewrite rule essentially says to reverse the procedure for any packets egressing GigabitEthernet0/3. Service Instance 3 takes traffic in Bridge Domain 3700 and associates it with S-VLAN 400.

In the example included here, The Customer A VM in Cluster 1 would have an S-VLAN of 300 and a C-VLAN of 200. The VM in Cluster 2 would have an S-VLAN of 400 and a C-VLAN of 200. When they traverse through the Wholesale Customer network they will remain distinct and separated. The biggest advantage to using the second method is that you are able to set an S-VLAN that is suitable for the Wholesale Customer without worrying about having that VLAN clash with those inside your own network.

There are many other features possible on the ME3600 utilising Service Instances and Bridge Domains that I have not covered here. These include Layer2 Protocol tunneling, Split Horizon groups to ensure certain ports in a Bridge Domain do not share traffic which can be utilised in to control loops in the network.

I hope this has been a useful introduction to the ME3600 and Selecting QinQ, and in particular using it in a location that it was not originally intended.

Feel free to add your own comments and view points, as this is still a developing design so I am happy for all your input.

NOTE: It should be noted that this switch is essentially transparent to your network, and that you are also bridging the Layer2 in your network with that of a third party, and all measures should be taken to reduce the impact of third party network problems taking out your own network.

I was recently called into a new customer’s network to help recover some passwords on some Cisco switches and to map out the network structure. Unfortunately nobody had any idea of the last time the switches had had their configs saved or even when the last time the switches had been power cycled. From what I can gather the previous IT guy didnt leave any information for those who followed.

TIP: Dont do this. It just makes people curse your name!

The problem here was that this network was carrying somewhat sensitive information and any “unplanned outages” would need to be minimised to an appropriate outage window. We did some physical tracing of the equipment connectivity and determined an eligible switch that only had a couple of nodes plugged into it, and no other switches hanging off it (as best we could tell). We scheduled an outage (thankfully it turns out that the middle of the day was actually the best time to do this), and just as we were about to start the first recovery one of the onsite guys found a USB key that just happened to have a very old backup copy of one of the configs. And just my luck the “line vty” password was in clear text! It cant hurt to try this password on the console can it?

WHAM! Unprivileged access. That was almost too easy 😉 A quick check around some of the other switches confirmed the same password on all of them. So I now at least had some form of access to the switches. I still needed to reset the enable secret entry which would require a reboot of the switch to perform, but maybe I would be able to gather some more information about the running state of these switches before a reboot. Maybe I could save myself from all sorts of hell if the configs weren’t saved after the last set changes,.

Now I, like Im sure many of you, spend most of my time on network devices barking my commands from the Ivory Towers of Priv15 land. I’m used to typing commands and having the router or switch go out of its way to provide me with any decadent output I requested! How much could I learn about these switches from the land of the plebs (What is this “>” prompt I see before me?). So I investigate…

“show tech” – OK, so I expected this one to fail, but I had to be sure 🙂

“show run” – This was another I expected to be slammed shut in my face!

“show cdp neighbor” – Not likely, but maybe? – NO CDP FOR YOU!

“show ip int brief” – Yes! First success, and now I know about any active primary IP addresses on this switch (TIP: “show ip int | inc Vlan|Int” provides a really handy output)

“show vlan” – Yes!

“show interface” – Yes

“show interface summary” – A given after the previous entry worked. A good overview though.

“show interface trunk” – Yes, and know I can re-create switch trunks with a little more confidence.

“show interface switchport” – Oh, now this is a good one. Lots of information about each and every port on the switch should I need to rebuild the configs

“show mac address-table” – Handy to know how many devices were coming in over a given interface before the reboot to cross check

“show ip arp” – Much like the previous MAC table

“show ip route” – What sort of routing table layout does this device have? What sources of routes? Static and Connected – any dynamic routes?

“show version” – Now I know how long the device has been online, what version it is running, and any changes to the config-register.

“dir flash: “ – No… but

“show flash:” – This gave me the exact same output that I would have gotten from “dir flash:”, so Im not sure why its not allowed by default.

From the “show flash” and the uptime value from the “show version” output, I was able to reasonably estimate when the config was last saved. Mind you when your switches dont have the clocks set and they think it is 1994 and the file modified date was 1996, you can rest assured the config most likely hasnt been saved since the last reboot!

So I felt pretty happy with myself about being able to get as much information as I reasonably could about these switches. When speaking to Ivan over at IOS Hints, he said that the fact I was able to get that much information from the devices might be worth noting for other people, and possibly used as part of your security measures for increasing the privilege levels required to run these commands on your production equipment if you are so concerned.

My tests here are just a few of the many commands available to non-privileged users, and I would be happy to hear from anyone else with some useful additions to the lists I put together here.

As always comments, flames, fanmail welcome!

It’s seems to be the craze this week to write all about our tool bags. Stretch wrote an article on it, followed by Jeff Fry‘s blog post and Tony Mattke over at Router Jockey. Not to be out done (and ever the trend setter) Jennifer Huber wrote her post 18 months ago!

So I guess to be just like the cool kids, I should write a post about my tool bag. Well I was going to, then I realised that despite how nerdy we all really are, there a limit to how many pictures of screwdrivers, cable testers and multimeters that we can actually all look at. Yes I carry the usual sorts of cables, screwdrivers, multimeters and crimping kit. I used to take my Leatherman everywhere until a run in with Airport Security on the way to Cisco Live in Melbourne (a moments silence please!).

Instead I have decided to write about my second most useful bit of kit I carry around. Up until yesterday I might have told you my favourite tool was the cage nut tool, but seeing as Jeff has already extolled the virtues of this tool I decided that I should talk about my “Second Favourite Tool”.

Now wait for it… My blog post is about the roll of purple masking tape I keep in my bag!

I first bought this roll of tape for a data centre move I was doing that involved unracking 5 racks worth of equipment and moving it across the other side of town in a single night. Since then I have always kept a roll in my bag ever since.

Now I don’t want to take any of the mystical powers away from Duct Tape (which still remains “The Force” for all things DIY), but there are certain advantages to having a paper based tape in your tool kit:

• You can tear it with your tiny girl hands
• You can write on it with pen, pencil, sharpie or even crayon (dont ask)
• You can remove it very easy (So the exact opposite reason to why Duct Tape is great)

The advantage of the purple colour is that it is a less used colour in the data centre so you can usually see it in the rack (Except the last time I used it when the rack was full or purple cat-5 cables… or if you use Extreme Networks switches).

I will often use this purple tape to tag cables as I unplug them from active equipment during moves or upgrades. Now Im sure everyone labels there cables in a sensible and efficient manner, but just in case its easy enough to remind yourself “Router-A:Gig0/0” at this stage. I then work through what ever changes I was making and as I plug equipment back in I remove the tape. In theory if I follow this procedure I should be able to look into the rack and not see any purple tape, and thus I should have every thing plugged back in as before.

NOTE: Purple Tape cannot stop you from plugging a cable into the wrong port!

Now please by all means, go and spend lots of money of good screwdrivers, cable testers and other tools, but think about throwing in a roll of paper based masking tape as well.
Postscript: Please don’t be dazzled by this blog’s first every use of images! Welcome to the future people – this is what 2011 is like 🙂

As some of my readers may know, I have spent the last week and a bit in Phnom Penh (Cambodia), setting up the new office for my company (eintellego). I had never been to Asia before this trip – outside of a couple of lay-overs in Japan in 2002 and 2003, or the 45 mins sitting on the plane in Bangkok Airport on the way to Abu Dhabi last November. I had no idea what to expect when stepping out of the airport.

I had no idea of the adventures that awaited me. I could tell the story of the motorbike rider attached to a medical drip riding down the road on the way from the airport. I could tell the story of the three legged dog standing next to a guy eating what appeared to be a “drumstick”. I could tell the story of being surprised by a Gibbon (Hack the Gibbon!) while walking to the office. I could tell the story of shooting AK-47s and M-16s at a Military base (Good prices!). Maybe somewhat more interestingly I could tell the story about the border war that started on the day I arrived here when the Thai Army “invaded” Cambodia over a disputed bit of land and a 1000 year old temple on the border of these two countries. And none of these even come close to the horrible abuses in translation on almost every menu I have seen since we got here, but alas this is a technical blog so I should focus on some of the things more interesting to my readers.

The thing I found immediately obvious to me as a westerner in Phnom Penh, was the contrasts in network design. Each street was lined with power poles with what at first sight looked like messes of spaghetti cabling. Upon closer inspection, and a week of reflection, there started to be some order to the chaos I was seeing.

At the same time back in Australia that we were debating VDSL implementation over existing copper telephone lines through to how/when/where/who of the national FTTH project (called the “National Broadband Network” (NBN) in Australia), the Cambodian government made a simple rule:

“$2 USD per block, per cable payable to the government for access”

An important point to make from the above – That is per cable! Whether 12-Core or 100-Core “$2 per block”. This simple rule made it affordable for the various Carriers and Telcos to deploy a FTTH project across the city of Phnom Penh and to provide end user services for $10/month (possibly cheaper but I didn’t look too hard).

Existing copper telephone networks were either non-existent or so poorly maintained as to be unusable for anything other than voice in many areas of the city, but at the price of deploying Fibre and Coax based solutions, this hardly seemed like too much of a problem.

Another thing I noticed – Almost every cafe and restaurant had some form of Free Wifi to customers. This was usually an Open Access point, or an encrypted SSID and the wait-staff would tell you the key (Note: Ask them to write it down for you! Saves a lot of time in mis-typing the key when you could be more productively uploading pictures of your food to Facebook and Twitter!). In Australia, those places that offer WiFi usually either had “50mb total in a 24 hour period” or charged a not-so-nominal price for access. I have to say, this aspect certainly made my me happy.

iPhone + Skype + Free_Wifi = @MrsJanitor + Happy

I would also like to take the time in the post to answer the most common question I received from both friends and collegues regarding this trip:

“Why are you opening an office in Cambodia?”

I guess there are several correct answers to this, but the truth is the amalgamation of several facts:

• We knew of several businesses working over in Cambodia working to improve the quality of life for the people there
• Through some of these contacts we have met many Cambodians who have amazing potential and just need the opportunities
• Cambodia gave us a presence within Asia that was no more than 2 hours flight from over 10,000 potential customers.
• There is an organisation called CIST who, in partnership with several large Corporations (including Cisco), go out to the provinces of Cambodia to find people with potential and train them in many aspects of IT including: Microsoft Technician Training, Linux Administration Skills, and Cisco Academy training to the CCNA level.

This last point alone has proven to be a really eye opening experience. When we advertised for our current positions (mostly HelpDesk and 1st level Network Ops), we received nearly 100 applications from people who had been through this training. We did two rounds of interviews – 1st round in December, and the 2nd round last week.

I wasn’t present for the first round of interviews, but the candidates we interviewed last week were great. I have them the usual network engineering and Linux admin questions I would ask any candidate back in Australia – no softball questions! They managed to answer each question beyond what I had expected and were able to discuss various protocol level concepts as well as configuration and management ideas to show to me that they actually understood what I was asking. All of this, while explaining it to me in something other than their native tongue! I had planed to bring on an additional staff member, but hired two more engineers after that round because I was so impressed.

All of this, and the fact that the Khmer people have such a great personality and sense of humour, so I felt like I was at my second home for much of my trip.

Yes, I understand that the above advantages could be said about many developing countries, but we had the opportunity and the ability to get on the ground in Cambodia. For more information regarding the how and why of Cambodia, Skeeve Stevens is working on a post on his new blog. Hopefully it will be published soon.

Im not sure if there will be many comments on this post, but feel free to send me any questions etc.

The Intro

One of the things I do regularly for my clients is to build LNS Infrastructure to take wholesale L2TP handoff. It is not uncommon to take these handoffs from several suppliers covering technologies such as DSL, Dialup ports, Wireless, 3G etc.

Wholesale pricing in Australia means that it is not unreasonable to take a wholesale L2TP handoff from an Aggregator company and to build your corporate WAN using wholesale DSL services. Often these networks are built for customers who’s primary business is not running networks, but rather delivering a service. I have an upcoming series of blog posts that will cover creating these services in more detail.

Once building this infrastructure, barring a new wholesale type, there is little more required than to order the service from the wholesaler and add a service to your RADIUS server. Other than the regular maintenance and patching of any network device, things generally go smoothly day-to-day.

That is until the “provisioning team” from said customer sends out a poorly configured router, and nobody discovers till it gets onsite. Usually this manifests in an incorrect PPP password.

I wont say that this happens often, but it happens enough that I need a “fix” ready to apply when the customer calls.

Isn’t it amazing how a poorly configured CPE is my fault?.

The Fix

The RADIUS standard provides us with an attribute that can be utilised in this situation – The Auth-Type Attribute. This attribute can be used to forcibly “Accept” or “Reject” a RADIUS account. All you need to do is to add “Auth-Type := Accept” to your RADIUS Check entry and RADIUS will ignore the password stored.

A simple RADIUS entry looks something similar to:

ppp-user        User-Password:= "password"
		Framed-IP-Address = "10.10.10.10"

With the Auth-Type override it would look like this:

ppp-user        User-Password:= "password", Auth-Type := "Accept"
		Framed-IP-Address = "10.10.10.10"

Next time the client tries to re-authenticate, RADIUS will choose to authorise this user even if the password is wrong.

The Catch

So after having to once again implement this attribute earlier this week to fix a router that was only sending a CHAP (and therefore encrypted) password, I decided to make a tweet about it. Not long after had I posted this, Ivan Pepelnjak of IOS Hints replied with:

It's also a terrific security hole. How do U make sure the acct is not used to log into your router?

My first response was that you should use a separate AAA server for your Console/Admin Authentication to Customer/PPP Authentication. Ivan promptly replied with:

Still, is there a way to prevent that acct from being used as a router login?

Thinking this was some kind of test, I started thinking…

The Answer

Before I was able to tweet a reply, Petr Lapukhov replied with:

@ioshints @networkjanitor hmmm, "autocommand=ppp" for PPP accts? remember doing that back in 98's :)

Now while I believe Petr’s response will have the desired overall effect, I suggest that it is less than ideal. Essentially the “autocommand” feature is used to run a set command on login of the user and then log them out. Ivan has written some posts on this on his blog. The two short-comings I see to this method are:

1. If the you try to use this account for console access to the router you are given a session with either a stream of rubbish on the terminal, or “This line may not run PPP.” followed by a disconnection.
2. The user is actually authenticated “OK”, by the RADIUS server, and no denied attempt will be logged for auditing purposes.

My view is the correct fix to this problem lies in RADIUS Best Practice implementation of the “Service-Type” Attribute.

Yes, I purposely left this out of the RADIUS configurations above!)

When used in the RADIUS Check section, the Service-Type attribute will only allow a match if the Correct Service type is sent. The two general ones that relate to this post are:

1. Service-Type = Login-User – Used for console/vty access to a RADIUS client
2. Service-Type = Framed-User – This is generally the service type that is used for PPPoE and L2TP termination.

I built a test lab this morning, and tested the above assumption with the following results:

When configured with the original settings listed above. the “ppp-user” account was able to successfully authenticate and log into the router.

The modification below was able to successfully stop the account for being used for vty login to the router, while still successfully being able to login via PPP.

ppp-user        User-Password:= "password", Service-Type = Framed-User
		Framed-IP-Address = "10.10.10.10"

And just because I wanted to ensure that “Auth-Type” would not override the “Service-Type” attribute, I tested the below config as well, and confirmed that this also would not allow vty console access to the router.

ppp-user        User-Password:= "password", Service-Type = Framed User, Auth-Type := "Accept"
		Framed-IP-Address = "10.10.10.10"

The Wrap Up

What was meant to just be an off handed tweet about a useful feature ended up sending me to the lab to test a bunch of assumptions I had been making in the past. I ended up learning a lot about a task I spend a lot of my week doing.

If you have any other suggestions (or corrections!), then please comment below.

Stay tuned for the upcoming series on creating your own LNS infrastructure.

Update: The Forgotten Credits

I would also like to thank Marko Milivojevic from IPExpert who acted as a sounding board while working through this post, and confirmed some of my suspicions before I went on to lab them. As always Marko, thankyou for your assistance!