Our team has recently been implementing some design modifications to our “Top of Rack / Middle of Row” Cisco Nexus deployment. Our previous design was basically Nexus 2K’s ToR, with twinax back to a pair of Nexus 5K’s mounted in the middle of the row. At first this was great, removing bulky patch panels and their associated trunks of cabling as well as freeing up the racks our retired Catalyst 6500’s occupied.
Over the last year, we’ve noticed some issues with this deployment design. The first and easiest to notice was cable management. A rack with 1 or 2 Nexus 2K’s only had its up-link twinax cables and power cables to contend with. A rack with a 5K and (2) 2K’s has all the Nexus 2K down-links and the 2K’s cabling. This can get pretty crowded fast. We also had to compete for cable management within the rack with the blade/server and SAN cabling.
The other issue we faced was finding a safe, fairly permanent home for the 5K’s inside the racks. There are multiple server groups with varying rack type and sizes in our data center. The optimal location for our 5K’s was not always a feasible location for the server rack owners. The switch installed in the rack is vulnerable to whatever cabling mishaps anyone working on the rack might cause. Some racks, like our SAN/NAS racks, come and go as the storage products are refreshed. This does not make for a great home for your 5K’s.
So what was our solution? Go above the rack and create our own home for the 5K’s.
We can still have great placement to take advantage of twinax down-links to our 2K’s and our 5K’s are in the optimal position for all nearby racks/rows. The 5K stays out of the potential cable management chaos inside the rack. If the racks are replaced, its only 1 or 2 Nexus 2K’s that have to be taken down.
Overall, we have found this new above the rack deployment scheme to be a success. Our cabling is more tidy, and outages because of cable mishaps is down. Using handheld infrared thermometer measurements, and Nexus’ own environment sensors we did see some slight temperature increases, but it was still within acceptable ranges and did not effect operation. This is something to consider given your potential heating/cooling situation.
What equipment did we use?
There are more than a few vendors with universal mounts available, and I believe there could be a few different viable solutions. We utilized Chatworth’s anyServer brackets and rack extension sets, mounted to the ceiling above the racks. For twinax cabling, we installed small L shaped brackets, just below the existing ladder rack for twisted-pair/cat6 cabling. In addition, we used some dedicated cable management guides above the 5K’s. For fiber up-links to the Nexus 7K’s, we installed 1u MultiMode fiber patch panels just above the 5K’s.
We have a killer facilities team in our DC and I want to thank them for working with us and creating this solution!