Flexible Power Distribution on a Budget (Part 2)

Now that our switchboard is in place we need to get power to where it's needed the most and for the bulk of you, that's going to be server racks.

If you're reading this, I'm going to assume that Datacentre style overhead power distribution is out of your budget. This how I accomplish the same task.

Firstly, start with a good rack. One that's  designed for the purpose. Using a server rack designed to house servers and manage cabling at the rear is probably not the best idea for a network patching rack  which will generally have cable management at the front. After having learnt this lesson the hard way, I can't help but try to impart this knowledge onto all of you.

APC AR3100 Server Rack	Panduit S6212BF Server Rack
APC AR3340 Network Rack	Panduit CQSMDC79 Network Rack

While i love both APC and Panduit racks and accessories, i do tend to favour APC. Mostly because i always seem to be able to get my hands on their server racks with relatively short notice. As always, it's up to you what rack fits your needs but just remember, there's a little more to choosing a server rack that simply looking at its dimensions.

Overhead Power Distribution On A Budget

Now that our rack is sorted, I would normally turn my attention to overhead cable management'. The APC AR8560 cable trough connects securely to the top of the APC rack though you can get away with using this setup on other racks. These cable troughs should be Connected together allowing a single rack to be removed independently of your electrical circuits running overhead.

Have the electrician mount the power outlet boxes on the rearward facing side of the cable trough and you'll no longer need to pay an electricians call out fee whenever you need to move things around.

Just remember to mark your outlets so you can tell at a glance which electrical board they're connected to. I suggest using paint pens to colour in part of the outlet box and colour part of the electrical board cover to match. I like to use red for UPS power and blue for mains power.

Earthing

The most important part of your install should be your earthing system. It's a safety system which should never be overlooked and your plan for this system should be discussed with your electrician.

Without your direction, the electrician will run a dedicated earth from the electrical board to your server rack and use a self tapping screw and appropriate washer to earth your server rack and yes, it does work but anything beyond that will generally be your problem to deal with not theirs.

If your thinking "well, I don't know anything about this stuff" I have some suggestions below.

Piggyback Style Earthing

The piggyback style is generally going to be the cheapest. Mostly because it can be wrapped into other jobs (like running a new circuit) for relatively little labour. It also requires the least amount of additional consumables.

Despite being easy on the hip pocket, you potentially introduce additional resistance into your earthing system and by piggybacking your earth, you are creating a number of potential points of failure. Severe a link in the chain and you compromise everything that was piggybacked to it.

Not good. Let's try and float another idea shall we?

Earth Your Overhead Power Distribution System

Consider having your electrician earth your overhead power distribution system. Why you ask? Your overhead power distribution system is unlikely to be moved. If your power is also terminated here then it absolutely cannot be moved without the assistance of an electrician.

Accidents happen and by being higher up means it's less likely to be damaged by an ignorant mug like me. I don't know about you, but it seems like a safe bet.

The astute among you are probably thinking that the overhead cable management still needs to be piggybacked together so there's still a flaw isn't there? You'd be correct, however it is mitigated to a certain degree.

Finally, if you disconnect a rack (for removal) there is no risk of compromising the rest of the system.

Copper Bus Bar System

Essentially this is a long square/flat bar of copper dipped in tin to inhibit corrosion. Suppliers normally sell these in different lengths but you can also ask them to cut them down to the size you need with a variety of different hole spacing (NEMA, BICSI etc). I personally try to avoid cutting them if I can as it saves time and money.

When it comes to mounting, there are a few ways to mount them. For me, the most important thing to remember is to secure it to your overhead power distribution so the racks can still be removed independently of your bus bar.

How many you choose to install is up to you, but at the very least you'll need one per row. This will ensure your devices have a clean and solid connection to mother earth.

While your at it, you should pick up some ring terminals, a crimping tool, ground wire (min 2.5mm or 13 gauge but no more than 4mm or 11 gauge) and probably a small wire brush or sandpaper  to ensure the surface is clean. You'll need to earth PDUs, UPSs, Network gear, Blade chassis etc so it's better to spend a few bucks on tools and do this part yourself.

Flexible Power Distribution on a Budget (Part 1)

One of the most common questions that's asked by literally no one is what's the best way to power my server room?

Over the years I've probably powered server rooms every which way you can think of and there definitely isn't a one size fits all solution to the problem.

Today, I'm going to walk you through a couple of different solutions which will hopefully make this decision just that little bit easier.

The Pay As You Grow Model

In most small scale installations an electrician is brought in, installs a circuit breaker at one end, an outlet at the other and runs some cable (generally single phase) between the two. Once he's finished, he hands you a very large bill and calls it a day.

... 250 million metres of cable? 1 bazillion hours of labour? You can scarcely believe your eyes! Surely this can't be real... But you pay it despite lamenting over the fact that you'll have to pay it again when you add another rack.

When you eventually start adding extra servers, racks, electrical circuits,  switching, SAN and a blade center, you end up in one or more of the following situations:

• You cannot effectively manage your energy usage efficiency ratio's
• You run out of space in your electrical board for new circuits
• You run out of capacity (amperage) in your electrical board
• You destroy your budget due to additional, unaccounted for electrical works.

Planning Starts From The Beginning

Let's rewind the clock to when your office/server room fit out is first taking place. Actually, let's rewind the clock even further to when we're planning the electrical for the floor. It's at this time where you should be considering how the server room should be powered because in many cases, you'll only get one shot.

Ensure you share your plans with the electrician or electrical engineer at every step of the way. If not, you could end up with a bunch of standard 10 amp single phase circuits.

Going too small will not only limit your options in the future, but you also risk overloading the circuit. Conversely, going too large may limit your equipment choices down the road and put a larger dent into your pocket than you expected.

Using A Dedicated Electrical Board

If you're installing an electrical board for the entire floor, chances are they need to run a new circuit up to your floor from the meter room. If they're going to all that trouble, now might be a good time to run an additional line straight into your server room and connect it to a new electrical board.

If Joe bloe decides to switch the power off to a circuit and gets the wrong one, at least you know he's not going to be messing with your server room. Yes, I've seen this happen, by an electrician funnily enough. We can laugh about it now though.

Also, by obtaining our own dedicated feed, we can cater for much larger amount of growth than we would have been able to otherwise.

Secondary Electrical Board

The above option may not always be available to you. If that is indeed the case, I normally opt for running ONE massive three phase circuit (the bigger the better) from your existing electrical board to a shiny new electrical panel inside your server room.

You can of course mount the panel outside if you want but subsequent new runs will require more labour, more materials and will probably look ugly. Mind you, it'll probably still look ugly inside the server room but fewer people will notice.

Don't Forget The UPS!

If you have a bare wire UPS output, don't forget that a separate electrical board  can also be used to distribute UPS power to your racks! Connect your UPS to the electrical board (to where mains power would normally be connected).

Just remember to use an appropriately sized electrical board. Using the same size as your mains electrical board may be overkill although I will admit, it would look perfectly symmetrical.

*ahem* Just remember, your UPS will always power less than your mains electrical board. Don't believe me? In part 2, we'll discuss overhead power distribution from the switchboard to the equipment.

Rack 2 Rebuild

If you haven't already read it, i suggest reading about the BW Rack 1 Rebuild because all of the nitty gritty is covered in that blog post.

The only difference with this rebuild is the fact that we pre-cabled every single port and put cable labels on both ends of each network cable.

Before The Rack 2 Rebuild

And the moment you've all been waiting for...

After The Rack 2 Rebuild

The middle of the server rack.

The lower half of the server rack.

The top of the server rack before we re-routed the pesky electrical conduit.

And the front of the rack 2 server rack. Doesn't that look nice?

Took two people a day to get that done but i'm sure you can agree it looks great.

IITG 

Rack 1 Rebuild

So how do we do it? and why do we do it?

That's easy. How we do it is listed below. Why we do it, is pretty simple. It makes maintenance easier, troubleshooting easier, and it just looks like it's something that you can really be proud to show off.

Planning

This all starts as most things start, with an Excel spreadsheet. I find that a spreadsheet allows me to visualise exactly which RU the equipment will be loaded into. After all, it's better to shuffle machines around in a spreadsheet than it is once the servers are in place.

When finding a home for a piece of equipment, there's a couple of things i try to keep in mind. I try to load all of the equipment as close to the middle of the rack as i can with the most frequently accessed pieces (such as an LCD KVM) mounted closer to the middle so your not having to bend down or look up to use them. I will leave a space or two here and there for additional pieces of equipment but those with a lot of potential for expansion (such as a SAN) i will load at the edge. This of course means that I'll generally fill a rack with a maximum of two pieces of equipment which have a large potential for expansion. It's at this stage that a capacity plan could really come in handy. Unfortunately, i never seem to have one so i do my best with the information i have at the time and add an extra RU or two where i can.

Rack UPSs

When it comes to a rack UPS, i mount them at the bottom of the rack. Pretty simple really, they're usually quite heavy and by mounting them lower down they're easier to slide in and out. At least, that's what i think. Usually i mount these UPS systems by myself and i find it easy. With two people, it should be no problem.

Patch Panels / KVMs etc

I keep my patch panels, KVMs etc for smack bang right in the middle of the rack. Why? because it allows to me to purchase standard length cables which are readily available. Sometimes you forget to order enough, or new equipment is added after you've finished planning and it's more likely you'll have a few of these shorter cables just lying around.

Once all of the equipment locations are finalised, it's time to move onto the cabling. My spreadsheet tracks the numbers, lengths and colours of each cable that's connected to each piece of equipment. Once i'm done, i can tally up the totals ready for the purchasing. Don't think the spreadsheet just tracks network cables either. I also keep track of the quantity, length and colour of each power cable as well.

Purchasing

Server Rack

We replaced the existing server rack which was an old Dell PowerEdge rack with an APC AR3100 42U server rack. This also allows us to pre-cable and pre-load as much equipment as possible into the new server rack and simply unload the old rack, remove it from the area, 

The other benefit was the fact that the APC had Zero U mounting points allowing us to mount PDUs on one side and cable management on the other.

Cable Management

We use APC Vertical Cable Organizers for vertical cable management. There's not much to be said but i highly recommend them.

At the top of the rack, we have a 600mm APC Cable Trough. The electricians use the cable trough mainly. Also, the the outlet boxes are mounted to the cable trough.

APC cable troughs, when connected together allow the server rack to be removed independently of the electrical. When the server rack was replaced, we needed electricians onsite to disconnect and relocate the outlets. With the outlets mounted on the cable trough, the electricians are no longer required when replacing the server rack. Below is what the top of the server rack used to look like and as you can see, there's no way around getting the electricians onsite without that cable trough.

We used a ton of velcro cable wraps for data cables and cable ties for power cables, SAS cables etc. Buy this stuff in bulk because you'll rip through the stuff quite quickly if you want your installation to look neat.

Also, replacing cable ties is annoying. Use them only where necessary.

PDUs

We used a pair of APC AP8959 Switched IP PDUs. In a 240v single phase environment with 20amp circuits these are probably the best for the job. In this environment, they'll handle up to 16 amps each which is enough to meet the CTO's needs.

One of the PDUs (the inner most one) is connected to a UPS while the other is simply connected directly to mains power.

KVM

We purchased a 16 port Dell 2162DS IP KVM. It's cheap and nasty but does the job. I have to say, it was my first time dealing with it and i probably don't want to re-use them ever again. Check out the marketing video. It played no part in selecting the KVM but here you go:

You might be wondering, what happens when you have more than 16 servers in a rack? Well, either you upgrade the KVM or you designate one SIP to be a 'floating' SIP to quickly connect to the new servers.

Before you ask why not just buy a bigger KVM? It wasn't an option and the CTO is quite happy to upgrade when necessary.

Rack UPS

Again we turn to Dell for a UPS. From memory it was a 1920w line interactive UPS. We also added an Extended battery module (EBM) and an network management card so it could be remotely managed and monitored. We also purchased a Dell temperature probe to keep track of the temperature within the rack.

While you might seem to think that this would be quite small, you need to remember that this UPS is connected to just one PDU and therefore only needs to power HALF the total power demand of the rack.

During post installation testing, we were at 1 hour of battery backup time for the entire server rack. This far exceeded the 20 minute requirement set by the CTO. Without the EBM though, we were at 15 minutes of battery backup time. The system can't be expanded beyond this capacity. You get what you pay for, nuff said.

Pictures Before The Rebuild

After The Rebuild

Although the picture doesn't show it, we wrapped the IEC leads connected to the outlets in the same colour as their feed cable (Red or Blue). This ensures you always know your dealing with the right power outlet.

You can also clearly see the status of those two outlets without opening the doors. This was something that wasn't possible with the old setup.

And this is the front of the server rack. In the end, we ditched the Dell covers on the front of the servers. I'm sure there's plenty of good reasons to do it such as airflow, better visibility of status lights etc. The real reason though was that they look ugly.

And then before and after. You'll also notice we scrapped the Dell cable management arms. I'm sure there's plenty of good reasons to do it such as airflow, better visibility of status lights etc. The real reason though, well, you know why.

And that's how you rebuild a server rack, costs a little bit but so much easier to work on than what was there before.

IITG