Wireless Drinking Stories
I had a “peaceful” weekend pretending to study for my upcoming CCDA exam in a couple weeks. Trying to convince myself that I cared, I decided the best course of action was to watch a movie! Sounds reasonable, right? I’ve been working my way back through Kevin Smith’s movies over the last couple of nights, and decided “Chasing Amy” was the perfect way to concentrate on what needed to be done.
There is a scene in this movie where two of the main characters are trading stories about their scars, paying homage to the classic scene in Jaws. At approximately the same time @amyengineer and @NetDonkey were on Twitter discussing geek talking at parties, and this got me thinking about some of the usual stories I tell after a few beers at the different conferences I attend.
So here it is. Think of this as my “Greatest Hits” of stories so far, these ones in particular relating to my experience as the Network Operations Manager of a wireless ISP in Australia. Alternative title – “When its not just an Invisible Blue Cable”.
The one about the tide
This particular story occurred within the first few weeks of my job at this company. At this stage we only had wireless coverage in one region of Australia, and were rolling out a test network in a region about 6 hours away using a new technology. In our local network we had a wireless back haul that used Motorola Canopy in a PtP arrangement across Brisbane Waters bay that backed onto another 900MHz Canopy AP.
The engineers who had been troubleshooting this problem on and off since before I started explained the situation as such:
“At random times our signal strength goes out the window, and we get to the point where we lose signal to the point that the PtP link drops out”.
I started troubleshooting this myself with another engineer who gave me my first lesson in real world wireless deployments:
“Sometimes its not just an invisible blue cable”
We had the device in MRTG and were reviewing historical signal strength readings when it started to occur to us that the signal strength issues followed a 6 hour cycle. Upon further investigation we realised that this link that went across a reasonable expanse of water was being affected by the change in sea level as the tides came in and out.
As I soon learnt there is a piece of critical wireless theory called “The Fresnel Zone“. Basically this is the area of space between two radios in a 3D elliptical shape and the size is based on both distance as well as the radio properties of the radio. Generally you don’t want to have more than 20% of the Fresnel zone obstructed, though it is often acceptable for up to 40%. There is plenty of stuff relating to this piece of wireless theory, but as long as you remember 60% clearance and the fact it is pronounced “Fray-nel” not “Frez-nal” you should be able to fool you way passed many RF guys and other lesser beings 😉
In our story the original link must have had marginal clearance within the Fresnel zone and the mere impact of the tides was enough to throw our signal strength out the window! Simple things like this can make your day disappear real quick.
The one about the GPS
In our second network we rolled out a (for the time) new and fancy wireless mesh product from SkyPilot Networks. On paper this product seamed to tick all the boxes for me – Single Channel for every repeater in the mesh, 360 degree coverage, Auto-provisioning, 802.1q and QoS capable and the ability to perform signal testing from the repeater.
In a SkyPilot network you generally have three types of devices:
1. The SkyPilot Gateway – Looks much like a up-turned garbage can, but is the brains of the mesh network. This device has a built-in GPS for coordinating communication within the mesh. This device is what you would plug into your traditional networking equipment in the datacenter / Point of Presence.
2. The SkyPilot Repeater – exactly the same look as the gateway, but a bit less smarts. This box needs to talk either directly to a Gateway device, or another repeater that is somehow connected to the gateway.
3. The SkyPilot Connector – This is the end customer CPE. Connects to either a Repeater or a Gateway on the wireless side, and Cat5 down to the customer’s computer via a PoE adapter on the LAN side. You can set the 802.1q VLAN id associated with the LAN interface per radio.
The basic concept is that each Gateway and Repeater device has 8 directional antennas arranged internally to provide 360 degree coverage. All sorts of wonderful maths comes in to play, but suffice it to say that the Gateway device uses timing sequences from the built-in GPS to (try to) ensure that no to panels that are facing each other are transmitting at the same time in an effort to reduce back chatter and radio interference. In a perfect world! All of this means that without correct GPS sync the device goes down the toilet taking your mesh with it. If the device detects a problem with the GPS of a continued period of time, the gateway will reboot… once again taking your mesh with it.
I came across a problem where at a “random” interval one of our gateway devices would reboot, and when it finally came back online (sometimes 15 – 45 minutes later), the reboot cause was listed as “Lost GPS Sync”. Much troubleshooting went on over quite a period of time.
The vendor suggested ensuring correct grounding of the device, because if there is significant difference between the electrical ground provided to the grounding lug on the base of the gateway unit can be enough to distort the signal and cause a reboot. This did reduce the frequency of the reboots, but they were still occurring.
Eventually on our 3RD RF check using expensive equipment we discovered that there was a Pager service on the water tower at the end of the block that was used as part of council surveying procedures. This device broadcasts on (iirc) 450MHz and had its antenna pointed almost directly into our Gateway device. It was merely coincidence that we were there at exactly the same time as the signal was being sent, but we caught it.
The lesson we took away from this:
“When troubleshooting problems with wireless, be aware of not just your broadcast frequency, but that of your GPS, and the various half, quarter and other offset frequencies for your application”.
Forgive me if my science is a little off on this, but while I spent 2 years working there, I was an “IP Guy” and not an “RF Guy”. Once again proves there is more than just an invisible blue cable!
The one about the barn
This is one of my favourites. It has become much like one of those fishing stories where the size and height of the barn get bigger with each retelling. I love to fish, but I rarely catch anything so this is the closest I get. I like to tell this one to younger staff who complain when I ask them to get on a ladder or crimp a cable from the comfort of their desks.
The setting is our third region we rolled out into. I had never been to this region let alone know the location of the sites in relation with each other – but I had a map, and the guy who chose the sites. While standing on the top of a mountain with beautiful scenery all around me I was given the following instructions:
“OK, you see those mountains on the horizon? They are about 8km away. You see that dip where my finger is pointing? OK, well another 8km on the other side of that dip is where the other side of this link is going”.
At this stage we had a discussion about paralax error and the fact that “where my finger is pointing” is not an exact instruction. Eventually I put together which “dip” in the mountains we were talking about and proceeded to mark out the point where the mast would be erected. We then drove to the proposed location of the other side of the link.
When I get there, I was informed that we were going to erect a 10m mast on the top of a barn that just happened to be in the right spot to see “through the dip in the mountains”, as well as line of sight back into town where a usual fibre backhaul provider had a POP at the local hospital.
After a day and a half of troubleshooting bringing up this link and taking both masts up and down several times, we discovered that our two antennas were assembled incorrectly. Our first few lessons from this trip:
“When installing any wireless link, ensure that you have matching polarity in your antenna.”
“Just because someone more knowledgeable than you assembled your antenna back in the office and shipped them to site, don’t just blindly trust that they did it correctly”
“One antenna set to horizontal polarity and one antenna set to vertical polarity can cause enough loss to make sure you never see signal from the other side (especially when your aiming at a non-descript dip)”.
After another day of troubleshooting the long haul link we finally got a good signal and locked off the mast where it was. Now to focus on the 1500m short hop into the hospital. This is about the same time that you learn that the cable leading up to the AP was crimped incorrectly and the device is not powering up. Not long after this you learn that the riggers who have been helping you erect the mast refuse to take it down again so you can fix up a cabling problem.
How does one solve this problem? That’s easy! You borrow an extension ladder straddle it either side of the peak of the roof while somebody who pretends not to look holds it steady. Climb said ladder and unclip the cover off the AP at almost full stretch of your arms. Remove the cable and cut the RJ45 end off. Recrimp cable above you head while trying not to think about the fact you are standing atop a barn that is about 8m high to the guttering, another 3m to the peak, atop a 2m ladder leaning on a flimsy pole. So thats White-Orange/Orange/White-Green/Blue/White-Blue/Green/White-Brown/Brown, yeah? This lesson:
“Always crimp your cable correctly. Then check it, and check it again”
“Test you cabling while everything is still on the ground”
“If you do something like this, don’t post it publicly where WorkCover can read about it… oops!”
Eventually I got the whole thing sorted out, but not without a lot of wasted time because of silly assumptions. This leads to my final lesson from the trip, which I call “Kurt’s Law of Away Trips”:
“When somebody else spec’s a job and insists its a 2 day job, pack for a whole week just in case and take a good book!”
The one about the Boat
This next story begins with me driving down the freeway with a family member when I got a series of alerts from our NMS system indicating that we just lost the primary hop between one of our POPS and the first AP up on the hill about 2km away.
Being the always on call network ninja that I am, I started SSH’ing from one smartphone, while RDP’ing into the NMS server from another. Things didn’t look good – there was lots of red. Red is never good! I called the local support engineer who informed me that there was a storm in the area, but he would make the trip up to the tower to check on everything. He braved the rain on the way up the mountain, and I braved the local Chinese take away.
I got a call about 45 minutes later informing me that everything was powered up correctly, but there was still no link. At about this time our conversation went a little like this:
Engineer: Oh! Could it be the boat?
Me: What boat?
Engineer: The boat that pulled into port today?
Engineer: A big container ship pulled into port today and its about 2 blocks from the POP and smack bang between the POP and the tower!
Me: Umm… what?
As it turns out, that is exactly what the problem was! Much like the first story, when the tide was at its highest (coupled with the storm) the boat was blocking enough of the signal to cause an outage. And three days later when the boat left signal was restored. 5 9’s anyone? I took a few lessons from this:
“Network documentation for remote wireless hops needs more than some Visio diagrams. Get photos in both directions as well as the general area. It will come in very handy when trouble shooting”
“Don’t let people without the technical ability determine where sites will be located”
“When the name of the region is something like “Port X”, there is a pretty good chance a big damn boat is gonna end up nearby so make sure that doesn’t ruin your weekend!”
The one where my car fell of a mountain
Funnily enough, this is the one my old work mates like to remind me of! One of our main tower sites for the wireless network on the Central Coast was only about 10 minutes drive from the office. I decided on the way home one night (spur of the moment decision) to drive up the tower and try and replace a management card in a UPS that had been giving us grief.
In theory this should have been a simple 5 minute stop off – and I guess it was. The real problem occurred as I was walking back to my car. While I had been inside it had started to rain ever so lightly. This was the first rain we had had in about 2-3 months at that stage so the (very well sealed) road was covered in a fine layer of dirt and once dried leaves.
The road was kind of winding, but with a max speed of 45km/hr we arent really talking about rally driving here. coming around the last bend before hitting the actual public road, I hit a patch of leaves and my car decided to move sideways off the road towards the down hill side of the mountain. Nothing here happened fast. I had a good 15 to 20 seconds of sideways drift as my car decided to continue with no traction off the road and down the side. The add insult to injury, when my car finally came to rest it was stopped by a tree no thicker than my arm!
Upon getting out of the car and surveying the situation, I realised there was no way I was getting myself out of this one alone. I had to admit the defeat. I called our Operations Manager who lived a couple of blocks away and who owned a 4WD. The phone call went something like this:
Me: Ralph? Do you love me?
Ralph: What? Why?
Me: I drove of a cliff!
Ralph: BAHAHAHA! I’ll be there in 5!
When he finally arrived, recovered from the site of me and my car sitting on the side of a mountain, and took enough photos to make fun of me the next day, we hooked up a tow strap to the back of the fourby and tow my car out. I busted up the front grille of my car and put a small ding in the bonnet, but other than that the Lancer came out OK. Me on the other hand am still held accountable for this story when enough alcohol is applied!
The one where you got bored and stopped reading
Well I guess I will leave it there for now. I’m sure you all have much better things to spend your time doing, and as it is I have just given most of my best stories away. I guess this means that if we ever do meet, we will have to talk about the weather or some such.
So this wasnt the most technical blog post on the face of the earth, but lessons learned in the field are the ones that seem to stick! Feel free to share any interesting stories you have that may or may not be related. In the mean time, I am going to go back to studying for this stupid exam!