Barreleye G2 User-guide: Setup OpenPOWER P100 GPU Server with Ubuntu 17.04 Zesty

HOST Console (Serial Console):

To get to the host console via serial / VGA use the following connections on front panel:

a) VGA + USB on the front panel.

B) Host serial console on the front panel (RJ-45 Serial)

 

Screen Shot 2017-05-11 at 3.27.44 PM

HOST Console (via BMC / OOB network):

You can get to host console via BMC / OOB IP Address. By default BMC will DHCP for IP Address. For getting the BMC IP , simply look up BMC IP corresponding to BMC dedicated Mac (for Example : 00:1f:e2:fd:f4:32)  or BMC shared Mac (for example:  00:1f:e2:fd:f4:33)

one you have the BMC IP, log into host console via following command

(Password: 0penBmc ):

ssh -p2200 root@<bmc-ip-address>

BMC / OOB Power-on : Power-off:

For powering the system on / off via BMC console, first ssh into BMC via following command (Password: 0penBmc  ):

ssh root@<bmc-ip-address>

Once logged into BMC , use the following commands to:

Check the state of the host:  obmcutil state

Power on the Host:                obmcutil poweron

Power off the Host:                obmcutil poweroff

SSH into host shell:

In order to ssh into the host , notice the host is going to default to DHCP for Broadcomm LOM Mac address(for example:  00:1f:e2:fd:f4:31) by default.

Look up your dhcp server for this mac to get Host IP

Log into host shell with below ssh command: (with password:  passw0rd)

ssh nvidia@<host-ip>

How to install P100 GPUs into server:

Screen Shot 2017-05-11 at 4.25.39 PM
Remove 4 screws around GPU metal base to release it from server tray. You will still have the PCIe cable connected
Screen Shot 2017-05-11 at 4.25.51 PM
With the released metal base (PCIe cable still connected), Start sliding the P100 GPU until it “clicks” with PCIe connector.
Screen Shot 2017-05-11 at 4.25.58 PM
Put 2 screws in the front of the metal base so that GPU sits on PCIe connector firmly
Screen Shot 2017-05-11 at 4.26.07 PM
Put the metal base with P100 GPU on Barreleye G2 Tray. Allign the metal base so that screw holes align with server tray 
Screen Shot 2017-05-11 at 4.26.15 PM
Put 4 screws back to secure the metal base (with GPU) to the server tray

 

Errata on Barreleye G2:

a) 17.04 daily builds are unstable, they fail to boot intermittently

b) 17.04 Serial console has a bug where it flashes the login screen continuously making it impossible to login. When this happens , the host is still up and functioning properly. Just SSH instead using Host IP to execute your commands.

c) Gen3 link training fails on boot up occasionally and link falls to Gen2. If you want GPU to train Gen3 , reboot until both GPUs train as Gen3 (Check via lspci) . Once link trains to Gen3 speed there are no issues running tests. The issue is only with the training algorithm in processor, once it trains the speed you want, things should be stable.

How to Install 17.04

Here’s how to install Ubuntu 17.04 on PPC64LE server. In my case Barreleye G2, our openPOWER server is running OPAL as BIOS firmware with Petitboot bootloader:

Here are steps through install:

1) Login to Out of band management and get into host console

  • Logging into host OOB console  is important since Ubuntu 17.04 PPC64EL installer may not have  ASPEED VGA / Graphics drivers. So you cannot see the install progress through the VGA console.
  • You can connect a host serial console via RJ-45 Serial on front panel OR
  • As indicated before you can login into host console via network
    • (Password: 0penBmc ):

      ssh -p2200 root@<bmc-ip-address>

2) Power on the server and get to Petitboot bootloader. Which should look like below:

Screen Shot 2016-05-06 at 1.37.12 PM

2) Now type “n” to have add a new boot option.

We are netboot installing 17.04. So let’s grab and add the kernel and initrd url’s for Zesty (17.04 code name) respectively:

Kernel:

http://ports.ubuntu.com/ubuntu-ports/dists/zesty/main/installer-ppc64el/current/images/netboot/ubuntu-installer/ppc64el/vmlinux

Initrd:

http://ports.ubuntu.com/ubuntu-ports/dists/zesty/main/installer-ppc64el/current/images/netboot/ubuntu-installer/ppc64el/initrd.gz

Your screen should look like this:

Screen Shot 2016-05-06 at 1.38.58 PM

3) Press OK and you should return to Petitboot main screen. The boot option you just added should come up as “User Item 1”:

Screen Shot 2016-05-06 at 1.39.25 PM

4) Select and enter “User Item 1” for starting the install. Following which you will go through standard Ubuntu Install process, that you should be familiar with.

Thanks I hope this post helps you Install not just Ubuntu 17.04 but any OS of your choice using Petitboot.

Barreleye G2 User-guide: Setup OpenPOWER P100 GPU Server with Ubuntu 17.04 Zesty

How to set OpenPOWER server to boot from any device

OpenPOWER servers have Petitboot bootloader to manage the bootup options / preferences.

Since petitboot is relatively new I am putting together this guide to help walk through on how to set up devices to boot from any device (network or disk).

Step 1. 

Power on the server and wait for Petitboot to come up via the VGA console (or OOB)

Step 2:

Select and click “system configuration” using the key-board arrows and enter key.

Screen Shot 2016-06-09 at 11.14.06 AM

Setup 3:

You will find below menu to customize the boot up options. In order to boot from any device, select and press enter on ” Clear and boot Any” option as below

Screen Shot 2016-06-09 at 11.29.28 AM

Setup 4:

Once you finish step 3, you will see that Boot Order in the first line of the screen short below changes to “(0) Any Device”

Screen Shot 2016-06-09 at 11.29.57 AM

Step 5:

Now let’s save this boot config. For saving the boot order we have to go all the way down to the boot menu with down arrow key and select and press enter on “OK”

Screen Shot 2016-06-09 at 11.30.27 AM

Step 6:

At this point you will be returned to petitboot menu and you have successfully set to boot from any device. From your next power on your server will automatically boot into an image that it finds first.

Power-off, Unrack and rack your server (or hard reboot) to test that “boot from any device” preference is working. When you un-rack and re-rack your server, you will have to wait few minutes (for BMC orange light to go off) before you press the host power on button.

Screen Shot 2016-06-09 at 11.31.40 AM

How to set OpenPOWER server to boot from any device

How to Install Ubuntu Xenial 16.04 on PPC64LE / OpenPOWER

Here’s how to install Ubuntu 16.04 on PPC64LE server. In my case Barreleye, our openPOWER server is running OPAL as BIOS firmware with Petitboot bootloader:

Here are steps through install:

1) Login to Out of band management and get into host console

  • Logging into host OOB console  is important since Ubuntu 16.04 PPC64EL installer doesn’t have  ASPEED VGA / Graphics drivers. So you cannot see the install progress through the VGA console.
  • In your case this might be a ipmi based command to get to host console
  • In my case since we are running a openBMC firmware for our out of band needs :
    • a) SSH into BMC
    • b) Typed the command:       obmc-console-client

2) Power on the server and get to Petitboot bootloader. Which should look like below:

Screen Shot 2016-05-06 at 1.37.12 PM

2) Now type “n” to have add a new boot option.

We are netboot installing 16.04. So let’s grab and add the kernel and initrd url’s for Xenial (16.04 code name) respectively:

Kernel:

http://ports.ubuntu.com/ubuntu-ports/dists/xenial/main/installer-ppc64el/current/images/netboot/ubuntu-installer/ppc64el/vmlinux

Initrd:

http://ports.ubuntu.com/ubuntu-ports/dists/xenial/main/installer-ppc64el/current/images/netboot/ubuntu-installer/ppc64el/initrd.gz

Your screen should look like this:

Screen Shot 2016-05-06 at 1.38.58 PM

3) Press OK and you should return to Petitboot main screen. The boot option you just added should come up as “User Item 1”:

Screen Shot 2016-05-06 at 1.39.25 PM

4) Select and enter “User Item 1” for starting the install. Following which you will go through standard Ubuntu Install process, that you should be familiar with.

 

Thanks I hope this post helps you Install not just Ubuntu 16.04 but any OS of your choice using Petitboot.

 

How to Install Ubuntu Xenial 16.04 on PPC64LE / OpenPOWER

OS support for Linux on Power (Little Endian)

OpenPOWER movement is gaining pace. Linux community has been overwhelmingly supportive of Little Indian port of Power (PPC64LE).  As a result, most of the leading Linux server distributions have built server images in the past couple of years.

Through this post I want to aggregate the installer links to server install images for PPC64LE. Where available I also listed cloud images used to boot up VM’s / (OpenStack guests).

OS Hypervisor Install image Guest Installer Image
Fedora 23 Refer to Link Link
Fedora 22 Refer to Link Link
Fedora 24 Refer to Link Link
Ubuntu 15.10 Refer to Link Link
Ubuntu 15.04 Refer to Link Link
Ubuntu  14.04 LTS Refer to Link Link
Centos 7.2 Refer to Link NA (But you can use init from RHEL)
OpenSUSE 13.2 Refer to Link Link
Debian 8 Refer to Link NA
PowerKVM 3.1 Refer to Link NA
Ubuntu 16.04 Refer to Link Link
RHEL 7.2 Refer to Link (Needs Login) Link
SLES 12 Refer to Link (Needs Login) Link

Hopefully this table helps the audience who are just getting started on power servers, to finish a quick install.

OS support for Linux on Power (Little Endian)

Update Model /Serial number of OpenPOWER server

I’ve found the necessity of updating the Model / Serial numbers of our openPOWER servers at Rackspace. And here’s how I did it:

Using LC system VPD update tool
The VPD update tool is used to make VPD modifications to POWER Systems from the host OS of the machine.

For Ubuntu distributions:
ophtk-lht_20160128-1.ub14.04_ppc64el.deb

For RHEL distributions:
ophtk-lht-20160128-1.el7.ppc64le.rpm

Ubuntu
  • To install, run the following:
  • apt-get install libxml2-utils
  • sudo dpkg -i <deb downloaded above>
  • If you encounter a dependency error when installing the tool, run the following to install dependencies and complete the install:
  • sudo apt-get -f install
RHEL
  • To install, run the following as root:
  • yum install ./<rpm downloaded above>
Execution
  • To update the machine serial number:
  • sudo /opt/openpower/lht/bin/updateserial <0000000000000000>
  • Note: <0000000000000000> to be replaced with your Serial Number
  • To update the machine model:
  • sudo /opt/openpower/lht/bin/updatemodel <8335-GTA>
  • Note: <8335-GTA> to be replaced with your model

 

Image