How to Netboot install RHEL 7.4 on POWER9 / Barreleye G2

RHEL 7.4 as I’m aware is the only GA’ed enterprise OS on POWER9 and hence Barreleye G2. I’ve seen couple of folks have issues picking the right image / boot arguments while attempting this install , and fail, so wanted to clearly put the steps down here:

If you instead want to install Ubuntu 16.04 LTS on POWER9 I’ve already written instructions for that in previous post: 

  1. First grab the DVD (2.9 GB) from RHEL website. For the purposes of this post we’ll use evaluation version:

> Go to RHEL downloads page:

https://access.redhat.com/downloads/content/420/ver=/rhel—7/7.4/ppc64le/product-software

> Login and Get Download Link “Red Hat Enterprise Linux for Power 9”

 

NOTE1: DO-NOT get link for generic “RHEL big-endian for ppc64” and “RHEL little-endian ppc64”) rather get specific version of download provided for POWER9

NOTE:2 : Download the expansive DVD ( 2.9GB binary as RHEL calls it ) , just the basic ISO file is NOT enough for netboot, since there is no public mirror for repo / package data, that I’m aware of.

See picture below, get the second file. In my case file name was “rhel-alt-server-7.4-ppc64le-dvd.iso”. If you are missing the alt, you are getting the wrong file:

Screen Shot 2018-01-19 at 4.28.54 PM

2.  Once you get the download link for this image, wget it to local (apache) http server (obviously, this is different than the POWER machine you are doing the install) and mount that iso to a folder on the http server path

Log into

ssh root@<http-server>

cd /var/www/html/

mkdir rhel9

cd rhel9

wget <hyperlink to rhel-alt-server-7.4-ppc64le-boot.iso>

mount -o loop rhel-alt-server-7.4-ppc64le-dvd.iso\?_auth_\=1516572534_bc1326574be188a5e87bb59dfcd70b8e rhel9

See picture below for help:

Screenshot from 2018-01-21 20-45-36

3. Now get to main petitboot menu and add a new boot option by  “n” button. Fill in the new boot option menu with following mock links as example :

Kernel:  http://10.127.xx.xx/rhel/ppc/ppc64/vmlinuz

Inirdrd: http://10.127.xx.xx/rhel/ppc/ppc64/initrd.img

boot-arguments:    root=live:http://10.127.xx.xx/rhel/LiveOS/squashfs.img

4. Click save and execute the new boot option you just created “User item 1”:

Screen Shot 2016-05-06 at 1.39.25 PM

5. That will boot to RHEL 7.4 Install menu in about 2 minutes or so. Select “Text mode” as I ran into some issues getting VNC install working:

6. You will be greeted with the following text menu screen. The only tricky item here is number 3, “Installation Source”.

Select http mirror and enter the mount location of your iso as the selection:

Screenshot from 2018-01-21 20-28-26

7. After completing all the necessary options 1 through 9 , press “b” to begin the installation begin the installation.

POWER9 RHEL 7.4 Barreleye Installation Done

8.Installation should be done in under 7-8 minutes as we are choosing to install from local mirror:

Screenshot from 2018-01-21 20-42-42

Don’t hesitate to comment below for any questions.

Advertisements

Barreleye G2 User-guide: Setup OpenPOWER P100 GPU Server with Ubuntu 17.04 Zesty

HOST Console (Serial Console):

To get to the host console via serial / VGA use the following connections on front panel:

a) VGA + USB on the front panel.

B) Host serial console on the front panel (RJ-45 Serial)

 

Screen Shot 2017-05-11 at 3.27.44 PM

HOST Console (via BMC / OOB network):

You can get to host console via BMC / OOB IP Address. By default BMC will DHCP for IP Address. For getting the BMC IP , simply look up BMC IP corresponding to BMC dedicated Mac (for Example : 00:1f:e2:fd:f4:32)  or BMC shared Mac (for example:  00:1f:e2:fd:f4:33)

one you have the BMC IP, log into host console via following command

(Password: 0penBmc ):

ssh -p2200 root@<bmc-ip-address>

BMC / OOB Power-on : Power-off:

For powering the system on / off via BMC console, first ssh into BMC via following command (Password: 0penBmc  ):

ssh root@<bmc-ip-address>

Once logged into BMC , use the following commands to:

Check the state of the host:  obmcutil state

Power on the Host:                obmcutil poweron

Power off the Host:                obmcutil poweroff

SSH into host shell:

In order to ssh into the host , notice the host is going to default to DHCP for Broadcomm LOM Mac address(for example:  00:1f:e2:fd:f4:31) by default.

Look up your dhcp server for this mac to get Host IP

Log into host shell with below ssh command: (with password:  passw0rd)

ssh nvidia@<host-ip>

How to install P100 GPUs into server:

Screen Shot 2017-05-11 at 4.25.39 PM

Remove 4 screws around GPU metal base to release it from server tray. You will still have the PCIe cable connected

Screen Shot 2017-05-11 at 4.25.51 PM

With the released metal base (PCIe cable still connected), Start sliding the P100 GPU until it “clicks” with PCIe connector.

Screen Shot 2017-05-11 at 4.25.58 PM

Put 2 screws in the front of the metal base so that GPU sits on PCIe connector firmly

Screen Shot 2017-05-11 at 4.26.07 PM

Put the metal base with P100 GPU on Barreleye G2 Tray. Allign the metal base so that screw holes align with server tray 

Screen Shot 2017-05-11 at 4.26.15 PM

Put 4 screws back to secure the metal base (with GPU) to the server tray

 

Errata on Barreleye G2:

a) 17.04 daily builds are unstable, they fail to boot intermittently

b) 17.04 Serial console has a bug where it flashes the login screen continuously making it impossible to login. When this happens , the host is still up and functioning properly. Just SSH instead using Host IP to execute your commands.

c) Gen3 link training fails on boot up occasionally and link falls to Gen2. If you want GPU to train Gen3 , reboot until both GPUs train as Gen3 (Check via lspci) . Once link trains to Gen3 speed there are no issues running tests. The issue is only with the training algorithm in processor, once it trains the speed you want, things should be stable.

How to Install 17.04

Here’s how to install Ubuntu 17.04 on PPC64LE server. In my case Barreleye G2, our openPOWER server is running OPAL as BIOS firmware with Petitboot bootloader:

Here are steps through install:

1) Login to Out of band management and get into host console

  • Logging into host OOB console  is important since Ubuntu 17.04 PPC64EL installer may not have  ASPEED VGA / Graphics drivers. So you cannot see the install progress through the VGA console.
  • You can connect a host serial console via RJ-45 Serial on front panel OR
  • As indicated before you can login into host console via network
    • (Password: 0penBmc ):

      ssh -p2200 root@<bmc-ip-address>

2) Power on the server and get to Petitboot bootloader. Which should look like below:

Screen Shot 2016-05-06 at 1.37.12 PM

2) Now type “n” to have add a new boot option.

We are netboot installing 17.04. So let’s grab and add the kernel and initrd url’s for Zesty (17.04 code name) respectively:

Kernel:

http://ports.ubuntu.com/ubuntu-ports/dists/zesty/main/installer-ppc64el/current/images/netboot/ubuntu-installer/ppc64el/vmlinux

Initrd:

http://ports.ubuntu.com/ubuntu-ports/dists/zesty/main/installer-ppc64el/current/images/netboot/ubuntu-installer/ppc64el/initrd.gz

Your screen should look like this:

Screen Shot 2016-05-06 at 1.38.58 PM

3) Press OK and you should return to Petitboot main screen. The boot option you just added should come up as “User Item 1”:

Screen Shot 2016-05-06 at 1.39.25 PM

4) Select and enter “User Item 1” for starting the install. Following which you will go through standard Ubuntu Install process, that you should be familiar with.

Thanks I hope this post helps you Install not just Ubuntu 17.04 but any OS of your choice using Petitboot.