Rack 2 Rebuild

If you haven't already read it, i suggest reading about the BW Rack 1 Rebuild because all of the nitty gritty is covered in that blog post.

The only difference with this rebuild is the fact that we pre-cabled every single port and put cable labels on both ends of each network cable.

Before The Rack 2 Rebuild

And the moment you've all been waiting for...

After The Rack 2 Rebuild

The middle of the server rack.

The lower half of the server rack.

The top of the server rack before we re-routed the pesky electrical conduit.

And the front of the rack 2 server rack. Doesn't that look nice?

Took two people a day to get that done but i'm sure you can agree it looks great.

IITG 

Rack 1 Rebuild

So how do we do it? and why do we do it?

That's easy. How we do it is listed below. Why we do it, is pretty simple. It makes maintenance easier, troubleshooting easier, and it just looks like it's something that you can really be proud to show off.

Planning

This all starts as most things start, with an Excel spreadsheet. I find that a spreadsheet allows me to visualise exactly which RU the equipment will be loaded into. After all, it's better to shuffle machines around in a spreadsheet than it is once the servers are in place.

When finding a home for a piece of equipment, there's a couple of things i try to keep in mind. I try to load all of the equipment as close to the middle of the rack as i can with the most frequently accessed pieces (such as an LCD KVM) mounted closer to the middle so your not having to bend down or look up to use them. I will leave a space or two here and there for additional pieces of equipment but those with a lot of potential for expansion (such as a SAN) i will load at the edge. This of course means that I'll generally fill a rack with a maximum of two pieces of equipment which have a large potential for expansion. It's at this stage that a capacity plan could really come in handy. Unfortunately, i never seem to have one so i do my best with the information i have at the time and add an extra RU or two where i can.

Rack UPSs

When it comes to a rack UPS, i mount them at the bottom of the rack. Pretty simple really, they're usually quite heavy and by mounting them lower down they're easier to slide in and out. At least, that's what i think. Usually i mount these UPS systems by myself and i find it easy. With two people, it should be no problem.

Patch Panels / KVMs etc

I keep my patch panels, KVMs etc for smack bang right in the middle of the rack. Why? because it allows to me to purchase standard length cables which are readily available. Sometimes you forget to order enough, or new equipment is added after you've finished planning and it's more likely you'll have a few of these shorter cables just lying around.

Once all of the equipment locations are finalised, it's time to move onto the cabling. My spreadsheet tracks the numbers, lengths and colours of each cable that's connected to each piece of equipment. Once i'm done, i can tally up the totals ready for the purchasing. Don't think the spreadsheet just tracks network cables either. I also keep track of the quantity, length and colour of each power cable as well.

Purchasing

Server Rack

We replaced the existing server rack which was an old Dell PowerEdge rack with an APC AR3100 42U server rack. This also allows us to pre-cable and pre-load as much equipment as possible into the new server rack and simply unload the old rack, remove it from the area, 

The other benefit was the fact that the APC had Zero U mounting points allowing us to mount PDUs on one side and cable management on the other.

Cable Management

We use APC Vertical Cable Organizers for vertical cable management. There's not much to be said but i highly recommend them.

At the top of the rack, we have a 600mm APC Cable Trough. The electricians use the cable trough mainly. Also, the the outlet boxes are mounted to the cable trough.

APC cable troughs, when connected together allow the server rack to be removed independently of the electrical. When the server rack was replaced, we needed electricians onsite to disconnect and relocate the outlets. With the outlets mounted on the cable trough, the electricians are no longer required when replacing the server rack. Below is what the top of the server rack used to look like and as you can see, there's no way around getting the electricians onsite without that cable trough.

We used a ton of velcro cable wraps for data cables and cable ties for power cables, SAS cables etc. Buy this stuff in bulk because you'll rip through the stuff quite quickly if you want your installation to look neat.

Also, replacing cable ties is annoying. Use them only where necessary.

PDUs

We used a pair of APC AP8959 Switched IP PDUs. In a 240v single phase environment with 20amp circuits these are probably the best for the job. In this environment, they'll handle up to 16 amps each which is enough to meet the CTO's needs.

One of the PDUs (the inner most one) is connected to a UPS while the other is simply connected directly to mains power.

KVM

We purchased a 16 port Dell 2162DS IP KVM. It's cheap and nasty but does the job. I have to say, it was my first time dealing with it and i probably don't want to re-use them ever again. Check out the marketing video. It played no part in selecting the KVM but here you go:

You might be wondering, what happens when you have more than 16 servers in a rack? Well, either you upgrade the KVM or you designate one SIP to be a 'floating' SIP to quickly connect to the new servers.

Before you ask why not just buy a bigger KVM? It wasn't an option and the CTO is quite happy to upgrade when necessary.

Rack UPS

Again we turn to Dell for a UPS. From memory it was a 1920w line interactive UPS. We also added an Extended battery module (EBM) and an network management card so it could be remotely managed and monitored. We also purchased a Dell temperature probe to keep track of the temperature within the rack.

While you might seem to think that this would be quite small, you need to remember that this UPS is connected to just one PDU and therefore only needs to power HALF the total power demand of the rack.

During post installation testing, we were at 1 hour of battery backup time for the entire server rack. This far exceeded the 20 minute requirement set by the CTO. Without the EBM though, we were at 15 minutes of battery backup time. The system can't be expanded beyond this capacity. You get what you pay for, nuff said.

Pictures Before The Rebuild

After The Rebuild

Although the picture doesn't show it, we wrapped the IEC leads connected to the outlets in the same colour as their feed cable (Red or Blue). This ensures you always know your dealing with the right power outlet.

You can also clearly see the status of those two outlets without opening the doors. This was something that wasn't possible with the old setup.

And this is the front of the server rack. In the end, we ditched the Dell covers on the front of the servers. I'm sure there's plenty of good reasons to do it such as airflow, better visibility of status lights etc. The real reason though was that they look ugly.

And then before and after. You'll also notice we scrapped the Dell cable management arms. I'm sure there's plenty of good reasons to do it such as airflow, better visibility of status lights etc. The real reason though, well, you know why.

And that's how you rebuild a server rack, costs a little bit but so much easier to work on than what was there before.

IITG

Soundproof Server Room In A Box

This was part of an office fit out. The fit out consisted of a total of 140 odd network ports. My job was to figure out what needed to be done and get all of it up and running. 

Obviously, running 140 odd cat 6 cables back to the other side of the office would take quite a few people or a considerably long time and become quite costly.

Also, the area around the centre of the building is a common area so we can't very well run 140 odd network cables as the crow flies.

Instead, we decided to run a multicore fibre cable encased in corrugated PVC conduit connected to two, 1RU patch panels at each end. Cisco 10GBASE SE-SR SFP Modules are used at each end connecting the two locations together. Along with the fibre cable, there's four cat 5E cables which are used to connect the console port on each switch and UPS to an Out Of Band (OOB) management device allowing network maintenance to be performed remotely.

The SFP modules are connected to a stack of three, Cisco Catalyst 2960S 48 port switches (non POE) providing a small measure of redundancy.

We tried to use different coloured power cabling to try to make it quick and easy to identify what device your dealing with. The image above was taken before the project was finished and as such doesn't show the labeling that was added to each power cable. When your in the field and you've got an issue to deal with, it's nice having the information as fast as possible centred around where the problem usually is.

The switches are connected to a Zero U basic PDU which came with the rack although i tried really hard to find a switched IP PDU but couldn't find a single one which would fit. If someone out there makes switched IP PDUs, please make me a half height zero U PDU please.

There are three Cisco SPA 514G IP Phones, a 2n Helios IP Uni Single Button intercom and two Cisco Aironet 2600i Access Points. Being POE devices, we installed POE injectors which are hidden inside the rack. The cabling for POE devices is standard cat 6 cabling but with yellow cabling it sticks out. Obviously, i would have preferred to use POE switches but the limited number of POE devices at the time of switch selection coupled with the additional cost put POE switches out of reach.

The PDU is powered by a an APC 750VA UPS giving the entire system a good few hours of battery backup. An APC UPS Network Management Card with Environmental Monitoring ensures UPS status as well as temperature and humidity can be monitored within the server rack.

All of the data across the floor is connected to a number of Krone 24 port 2RU patch panels. These patch panels are pretty standard fare but they don't work well with the Neatpatch NP2K648 2RU Cable Management Kit. What's that you say? To give you a brief overview, check out this video from Cable Supply.

The Neatpatch NP2K648 2RU Cable Management Kit came with 48, 2 foot cat 6 patch leads. Each of these patch leads were labelled at both ends using cable labels oriented so the label was easy to read. Have a look at the before and after pics to see what it can do.

All of this goodness is contained within an APC NetShelter CX 18U Secure Soundproofed Server Room in a Box Enclosure. It remains locked at all times and when closed, no one really knows there's IT equipment whirring around inside the thing. Check out the video from APC for more information.

Thanks for reading everyone!

IITG

Who Is The Ignorant IT Guy

An IT guy who's been in the industry for over 10 years. I left school after year 10 and started working packing computers into boxes for Acer before being moved to the production line building and imaging the computers. Earning a little over $200 AUD and being 17 years old at the time, obviously i felt like a millionaire being able to purchase random crap for myself, my girlfriend and my friends.

As my contract with Acer finished, i found work at a local sports centre. I was able to use the facilities for free, and learn about linux by implementing an LDAP domain, internal DNS/DHCP, VPN and a generic fileserver.

I managed to someone secure an interview with a medical company and began work as a helpdesk engineer. In under two years, i had received four promotions making it to a linux systems administrator by hard work and hands on self education.

I've had no formal training prior to entering the industry and still continue in this fashion. I've pretty much learnt everything i know from just getting stuck in and finding out what works and what doesn't.

Generally when something doesn't work, i need to get it fixed as fast as possible. Preferably before anyone notices.

That's who i am. That's why i do what i do. I don't claim to be an expert on anything so don't assume i am. I don't claim to be anything really.

I set this blog up to share my experiences and hopefully expand my knowledge by sharing my thoughts with you and hopefully being corrected at the same time.

IITG