JSR (JCBC MER) - Customer Information Document

Owned by Paul Sumption
Last updated: Nov 08, 2023 by Dean Sweeney

Introduction 

Purpose

This document assists in providing information from CSCS to the JCBC Main Equipment Room (MER) tenants relating to aspects of the MER fit-out. The purpose is to collate answers to common questions and provide information that tenants may need to consider when installing equipment into the MER.

This document does not replace any separate service agreement documents or other agreements in place.

This document is a general information document and other documents, directives and agreements may override what is contained in this document.  In addition this document may later be superseded by other documentation.

Owner 

This document is owned by the Clinical School Computing Service. All requests for updates are to be addressed to the owner.

Glossary of terms

TermDefinition 
ATSAutomatic Transfer Switch
CSCSClinical School Computing Service
GBN Granta Backbone Network (University provided fibre circuits around Cambridge)
JANETJoint Academic Network
JCBCJeffrey Cheah Biomedical Centre
LVLow Voltage (50 to 1000 VAC or 120 to 1500 V DC)
MERMain Equipment Room, located: level 6 JCB
PDUPower Distribution Unit
SERSecondary Equipment Room.  To be found on the following floors Lv0,1,2,3,5 – one per floor. They serve one side of the building as opposed to serving the whole floor.
UISUniversity of Cambridge Information Services
UPSUninterruptable Power Supply
WCDCWest Cambridge Data Centre


MER Fit-out Concepts

Concepts 

A common theme for the development of the MER fit-out is that it should be “standards based”. 

Primarily, this means that, where appropriate, CSCS follow the lead from West Cambridge Data Centre and the standards they employ.

Additionally, any new networking or server infrastructure equipment installed by CSCS in the MER is also to be “standards based”.  CSCS have collaborated with UIS to ensure similar levels of equipment and equipment suppliers are used, or, where required, CSCS have developed a standard if none are available where it may be deemed practical and useful.

The primary objective from, the above considerations is that CSCS aim to provide consistency in both reliability and function.

WCDC Information

WCDC have their own service level agreement and documentation which can be obtained from WCDC.

WCDC publish further information available via the following link: https://help.uis.cam.ac.uk/service/network-services/data-centre

WCDC have a “fact finding” document, which provides useful points to consider for any move to a new equipment room: https://help.uis.cam.ac.uk/service/network-services/data-centre/services

Selected WCDC Points

If following the WCDC standards, the following selected WCDC standards are likely to become discussion points between CSCS and prospective MER tenants.

Devices having only one power supply

Equipment owners, having essential equipment capable of only supporting a single Power Supply Unit (PSU), should notify CSCS of this requirement at the earliest opportunity.

The use of an ATS device may be mandated, for example Z-ATS - Micro Automatic Transfer Switch (further details at end of document).

An ATS device may be mandated, for example Z-ATS - Micro Automatic Transfer Switch (further details at end of document).

Equipment with 3-pin plugs

Equipment owners needing to rack mount devices requiring three-pin power plugs, must make a prior request to CSCS.

Environmental Requirements

All equipment must operate within the ASHRAE (American Society of Heating, Refrigerating and Air-Conditioning Engineers) A2 envelope of inlet temperatures. (10oC to 35oC, WCDC will operate an inlet temperature of 27oC +/- 2oC)

Devices narrower than 19"wide

For equipment owners having essential devices less than 19" wide, it is essential that suitable rack mounting equipment be sourced prior to installation. For example: https://www.rackmount.it/products/fortirack/rm-fr-t3-industrial.html

Cabinet air flow

Equipment must be installed so that airflow is from the cold aisle to the hot aisle

Cable management arms

The use of server cable management arms is prohibited.

Airflow sleeves

Cabinet integrity is further enhanced by utilising airflow sleeves in conjunction with switches. These help funnel exhaust air out of the cabinet. Without these, exhaust air circulates within the cabinet, reducing airflow efficiency and, because the heated air isn't vented, the temperature within the cabinet increases.

Blanking Panels

Blanking panels must be installed in any unused rack space to minimize cold/hot air mixing.

CSCS have standard products for this as listed at the end of the document. CSCS may be able to provide these depending on the quantity required.

Cabling standards

Further information can be found here: https://help.uis.cam.ac.uk/service/network-services/data-centre/wcdc-standards/wcdc-cabling-standards

Other linked to information from other WCDC pages:

Structured cabling is to be used for inter-rack connectivity

Labelling standards

Further information can be found here: https://help.uis.cam.ac.uk/about-us/data-centre/wcdc-standards/wcdc-labelling-standards

Other linked to information from other WCDC pages:

Labelcore

  • PANDUIT ® LABELCORE™ Fiber Optic Cable Identification System

CSCS Comments & Variations on WCDC

Power cables

  • Tenants are to provide their own power cables – see details later in this document for recommendations and colour codes. Refer to 6.3.2

Cable Ties

  • Use of plastic tie wraps or cable ties is prohibited unless by specific exemption from CSCS. The use of hook and loop (Velcro) ties is a permitted alternative.
  • Recommended products can be found at the end of this document.

Cage Nuts

  • Along with racking equipment to mount on the cabinet 19” vertical rails, tenants should provide cages nuts and their own mounting hardware as relevant, for example some equipment will be “tool less” and not require cage nuts.
  • Recommend products can be found at the end of this document.

Inter-cabinet Connectivity

  • Cabling and inter-cabinet connectivity between separate banks of cabinets must be addressed by the use of structured cabling within the MER and requirements for this must be made with prior consultation with CSCS.
  • For interconnected open banks of cabinets, CSCS will allow internal to the cabinet cabling between cabinets.
  • There should be no cabling between cabinets that runs on top of the cabinets.
  • All external inter-cabinet cabling will use appropriate high-level cable trays and structured cabling as discussed with CSCS.

Equipment Age

  • To help run the server room in an energy efficient manner, CSCS will operate under the same model as the UIS WCDC hosting agreement, which states equipment must be no more than 5 years old (unless otherwise agreed in advance).
  • We may allow the installation of old equipment on a fixed term arrangement, to ensure Power Usage Effectiveness (PUE) of the room is not impacted by hosting legacy equipment, which over time consumes proportionally more and more power compared to contemporary equipment.

Equipment Type

  • Original intended use of the MER is low density server and storage, with HPC located in the West Cambridge Data Centre
  • It has been agreed there will be no high density/high power draw systems (e.g. HPC clusters) based on the original MER design requirements and that UIS would be the primary provider of HPC and large (PB scale) storage.

Alternate Locations for Unsuitable Equipment

To be discussed with CSCS, but may include:

  • UIS HPC service (e.g. HPHI, RCS IaaS etc.)
  • CSCS Managed Server Service.


MER Information for Customers

Power

  • The MER is fed with two power circuits.
    • Circuit A - Supported by a UPS
    • Circuit B - Generator backed essential services from the main switchboard
  • Each cabinet is fed a circuit A and circuit B supply via respective IEC 309 32A connectors positioned above each cabinet.
  • Two PDUs, 0U installed at the rear of each cabinet are: APC AP8653
    • Each PDU provides:
      • (21) IEC 320 C13 – some of these outlets may be used CSCS equipment
      • (3) IEC 320 C19 – Presume that the bottom C19 connector will be unavailable
    • One PDU provides circuit A UPS power, the other circuit B Generator backed.
  • The PDUs will be installed as follows as viewed looking into cabinet from opening the rear door:
    • Circuit A UPS on right hand side
      • Deemed majority of installed equipment has primary PSU on right hand side.
    • Circuit B Generator backed on left hand side


Power Leads

 Locking

  • CSCS strongly recommend locking power leads to avoid power leads being dislodged and equipment served losing power or experiencing brownouts.

JCB MER Power Leads Colour Coding

To aid with identify power leads and power circuits, similar to WCDC we have provided a colour coding scheme.

Based on reviewing product ranges and those that are available to the university at the time of the JCB MER build, the following is the recommend standard for the MER.

  • Supply A – Black , Supply B – Red or Orange


  • Supply A = UPS backed
  • Supply B = Generator backed

Product Supply

Further details on recommend products can be found at the end of this document.

Lengths

  • In general, all cabling should use the shortest suitable length to aid cabinet management, cable traceability, tidiness and reduce negative effects on other cabling (cables tangled together).
  • Based on PDU and cabinet arrangement it is likely that the following cable lengths will be most appropriate. This information provided as a guide only – installers should double check their requirements:
    • 1m PDU to equipment out on same side (1.5m if extra length/routing required)
    • 1m PDU to equipment outlet on opposite side (1.5m if extra length/routing required
  • Measurements are dependent on depth of equipment installed.
  • Before ordering cables, installers should check all measurements on site.

Cabinets

  • Excel cabinets SR 42U 800W 1200D-Black (544-42812-WDBR-BK)
  • 19” adjustable vertical rails mounted nominally as follows:
    • Clearance to front door: 15cm
    • Clearance to rear door: 18cm
    • Distance between front and rear rails: 85cm
    • All measurements approximate.
  • Two rear 10cm wide vertical cable trays, one either side 0U mounted.
  • Front vertical cable management, either side of 19” mounting rails.

Networking

  • Networking arrangements will be setup as discussed between CSCS and MER tenants.
  • Fibre panel termination at patch panels will be OS2 Single Mode LC/APC unless otherwise stated.
  • The JCB building is specified to CAT6A.
    • Screened cables will be used in the MER, SERs, structured cabling and in high density, higher noise and higher throughput conditions



CSCS Recommend Consumables & Equipment

The following recommend products are selected based on their function and availability from university suppliers and commonality with UIS.

Example relevant suppliers include UFS Market Place, Comms Express, Comtecdirect, CMSplc.

They are listed here for quick selection and ordering.

Power leads

  • CSCS strongly recommend locking power leads.
  • CSCS have evaluated a number of products which has led to the following selection and the colour coding standards. Refer to: 4.1.1
  • Recommended lengths 1 or 1.5m, refer to 4.1.1.4

C14 to C13

Zonit zLock (Universal locking both ends) C14 – C13

C14 to C15

Zonit zLock (Universal locking both ends) C14 – C15

  • Recommend Product: Zonit zLock IEC 320 Dual Locking Cable C14 – C15
  • Supplier: comms-express.com
    • Product not listed on website, but a custom quotation can be requested.
  • Colours Available: Black, Red or Orange manufactured (however generally black usually available in UK stock unless ordered in large quantities).
    • Black cables can be identified if desired, using other suitable colour coded cable labelling or identification methods.
  • Lengths Available: 0.5,1,1.5,2,2.5m
  • Cable size: 2mm2
    • This is larger than typical C14-C13 power cables that use 1 to 1.5mm2
    • As this is locking at both ends, consideration to tight loop back cable securing methods will not be required.
    • CSCS may have example cables that can be tested for suitability before making further purchases.

TrippLite  (non locking)

This is non locking cable, however higher end switch/server hardware often has other methods of securing cable. Please note, at the PDU end there will be no securing options provided.

Note this is 2mm2cable specification, which may be too thick for some types of tight loop back cable securing methods.

CSCS may have example cables that can be tested for suitability before making further purchases.

  • Supplier: CUFS market place
  • Recommend Product: See list below
  • Black various lengths:
    • TrippLite C14 to C15, 61cm, Marketplace Supplier: Softcat Limited, Mfr. Part No: P018-002
    • TrippLite C14 to C15, 90cm, Marketplace Supplier: Softcat Limited, Mfr. Part No: P018-003
    • RS Pro C14 to C15, 1M, Marketplace Supplier: RS Components, Mfr. Part No. 1230951
    • TrippLite C14 to C15, 1.8m, Marketplace Supplier: Softcat Limited, Mfr. Part No: P018-006
  • Red various lengths:
    • TrippLite C14 to C15, 61cm, Marketplace Supplier: Softcat Limited, Mfr. Part No: P018-002-ARD
    • TrippLite C14 to C15, 90cm, Marketplace Supplier: Softcat Limited, Mfr. Part No: P018-003-ARD
    • TrippLite C14 to C15, 1.8m, Marketplace Supplier: Insight Direct, Mfr. Part No: P018-006-ARD

C20 to C19

Zonit zLock (Universal locking both ends) C14 – C15

  • Universal locking both ends:

CSCS have not yet purchased this product, but we would recommending approaching comms-express and asking for a custom quotation as is the case with: Zonit zLock IEC 320 Dual Locking Cable C14 – C15.

IEC-LOCK

APC


Cage Nuts

  • When scaled to working with 000’s of these, product choice make a difference to install time and reliability.

Product:           StarTech.com M5 or M6 Mounting Screws and Cage Nuts

Part Numbers:  CABSCREWM5 or CABSCREWM6

Supplier:           UFS MarketPlace via Softcat

Hook and loop

“Wide”

“Narrow”

ATS

Equipment owners, having essential equipment capable of only supporting a single power supply unit (PSU), should consider the use of an ATS.

  • Z-ATS - Micro Automatic Transfer Switch
  • APC Rack-mount Transfer Switches

Blanking Panels

CSCS can provide blanking panels.

APC 1U Modular Toolless Blanking Panel

Eziblank 19 Inch Universal 6RU Modular Blanking Panel - 10 Pack

Copper Cabling

  • JCB structured cabling is specified to CAT6A.

General Recommendations:

  • Within 10Gig Server Room, SER, cable bundles, noisy environments:
    • CAT6A F/FTP version.
  • At 10Gig desk/device:
    • CAT6A F/FTP preferred, U/UTP may not actually be smaller diameter.

Screened 6A cable may not necessarily be less flexible than un-screened CAT6A, as un-screened CAT6A may have larger overall diameter to compensate and ensure wire pair separation.

JCB fit-out standard cable:

Fibre Cabling

  • Latest UIS fibre patch panels and the JCB building will use LC/APC.
  • Matching patch lead cabling at time of JCB build can sometimes be on longer lead times.

LC/APC to LC/UPC

LC/UPC to LC/UPC

This cable type is more widely available. CSCS tend use Comms Express.

SFPs

If converting from using twinax or DAC cables to go between cabinets to using in cabinet fibre patch leads, structured fibre preterm between cabinets and appropriate network switch provision; one consideration will be the supply of SFPs.

CSCS tend to use the following supplier for OEM SFPs from the following supplier: fibre.co.uk

http://fibre.co.uk/category/solutions/components-and-consumables/pluggable-transceivers

Typically they are manufacture by Optomark, example information:

http://www.optomark.eu/index.php/products/transceivers/sfp-10ge-and-8g-fc

CSCS normally request a current quotation from fibre.co.uk.

Example part numbers:


SM/MM

Interface

Fibre.co.uk model number

Fibre.co.uk description

SFP 1G 1000BaseLX SM - 1310nm up to 10Km


Cisco coded optics

SM

LC (UPC)

OPM-SFP-LX-CA

SFP LX 1310nm GbE 10km SMF DDMI 0-70  CiscoASR Compatible

SFP 1G 1000BaseLX SM - 1310nm up to 10Km


Cisco coded optics

SM

LC (UPC)

OPM-SFP-LX-C

SMF 1G 1310nm 10km LC - Cisco coded

SFP 1G 1000BaseSX MM - 850nm up to 550Mtr


Cisco coded optics

MM

LC (UPC)



SFP+ 10G 10GBaseLR SM - 1310nm up to 10Km
Cisco coded optics

SM

LC (UPC)

OPM-SPP-LR-CRS

SMF 10G 1310nm 10km LC