There are 2 ways you can update host / boot firmware in openPOWER machines
a) Via BMC console
b) Via BMC REST API
In this post we will see the steps for updating the host firmware via BMC console:
Step1: Get to the BMC console (via ssh)
ssh root@<openBMC IP>
Step2: Now we want to make sure host power is off
Step3: Get the host flash binary into /tmp location on BMC
wget <link to host firmware>
Step4: Start flashing the new firmware using ‘pflash’ utility. Once you know for sure host power is off ( you can check it via <obmcutil state>)
pflash -E -f -p <flash-binary-name>.pnor
Step5: Once you see the flash progress reach 100% and complete, try to get the power on
Step6: Check the console , if your host is booting successfully
Above steps (specifically the pflash command above) completely wipe out your flash chip (including boot preferences) and replace it with new host firmware. If you, instead wanted to save boot preferences then you need to back up the NVRAM partition using the flow below:
OpenPOWER CPUs and Nvidia GPUs represent the best in class computing Power per $
To test that, I thought I’ll give compiling Ethereum mining suite a try on our Barreleye G2 server. As background information Barreleye G2 is a server Rackspace is building in collaboration with Google, IBM and Ingrasys
To compile Ethereum on Power9 with Nvidia P100 GPUs you have 2 options:
A) Compile it for CUDA
B) Compile it for OpenCL
In this post we will go with Option A (Compile it with CUDA). To do this you need to first get Nvidia Drivers and CUDA working on your openPOWER server. The link to blog post on how to get these working is here:
Once you have CUDA working (run an example CUDA application to make sure), install the build dependencies necessary for compiling ethereum. We will use cmake to configure and build.
After looking around I realized that main github source for ethereum “ethereum/cpp-ethereum” no longer supports mining for GPUs. So instead I used: https://github.com/ethereum-mining/ethminer for my mining experiments with Nvidia P100 GPUs. I found the release “0.12.0.dev2” to be stable.
Download the latest official release zip and unzip
To get to the host console via serial / VGA use the following connections on front panel:
a) VGA + USB on the front panel.
B) Host serial console on the front panel (RJ-45 Serial)
HOST Console (via BMC / OOB network):
You can get to host console via BMC / OOB IP Address. By default BMC will DHCP for IP Address. For getting the BMC IP , simply look up BMC IP corresponding to BMC dedicated Mac (for Example : 00:1f:e2:fd:f4:32) or BMC shared Mac (for example: 00:1f:e2:fd:f4:33)
one you have the BMC IP, log into host console via following command
(Password: 0penBmc ):
ssh -p2200 root@<bmc-ip-address>
BMC / OOB Power-on : Power-off:
For powering the system on / off via BMC console, first ssh into BMC via following command (Password: 0penBmc ):
Once logged into BMC , use the following commands to:
Check the state of the host: obmcutil state
Power on the Host: obmcutil poweron
Power off the Host: obmcutil poweroff
SSH into host shell:
In order to ssh into the host , notice the host is going to default to DHCP for Broadcomm LOM Mac address(for example: 00:1f:e2:fd:f4:31) by default.
Look up your dhcp server for this mac to get Host IP
Log into host shell with below ssh command: (with password: passw0rd)
How to install P100 GPUs into server:
Errata on Barreleye G2:
a) 17.04 daily builds are unstable, they fail to boot intermittently
b) 17.04 Serial console has a bug where it flashes the login screen continuously making it impossible to login. When this happens , the host is still up and functioning properly. Just SSH instead using Host IP to execute your commands.
c) Gen3 link training fails on boot up occasionally and link falls to Gen2. If you want GPU to train Gen3 , reboot until both GPUs train as Gen3 (Check via lspci) . Once link trains to Gen3 speed there are no issues running tests. The issue is only with the training algorithm in processor, once it trains the speed you want, things should be stable.
How to Install 17.04
Here’s how to install Ubuntu 17.04 on PPC64LE server. In my case Barreleye G2, our openPOWER server is running OPAL as BIOS firmware with Petitboot bootloader:
Here are steps through install:
1) Login to Out of band management and get into host console
Logging into host OOB console is important since Ubuntu 17.04 PPC64EL installer may not have ASPEED VGA / Graphics drivers. So you cannot see the install progress through the VGA console.
You can connect a host serial console via RJ-45 Serial on front panel OR
As indicated before you can login into host console via network
(Password: 0penBmc ):
ssh -p2200 root@<bmc-ip-address>
2) Power on the server and get to Petitboot bootloader. Which should look like below:
2) Now type “n” to have add a new boot option.
We are netboot installing 17.04. So let’s grab and add the kernel and initrd url’s for Zesty (17.04 code name) respectively:
Before we go on to look at optimizing the CPU frequency on Power systems, it is important to know what frequency you are running at. For this make sure you have powerpc-utils package installed.
dnf install powerpc-utils
yum install powerpc-utils
apt-get install powerpc-utils
Once you install this package, you can run the following command to know the frequency your CPU can run at:
sudo ppc64_cpu --frequency
STEP 1 of Optimization:
To optimize CPU frequency for OpenPOWER distributions for ppc64le we first need install the kernel module specific to POWER-PC Architecture and then change the governor to “performance” enhancing mode.
On Fedora and some other openPOWER distributions this kernel module is not installed by default. For example: On Barreleye server, because of absence of this module on Fedora 23, Kernel doesn’t have enough info to scale the CPU very well. And hence puts CPU at a defensive “2.0 Ghz” instead of full “3.5 Ghz”.
Here is how you load the kernel module specific to PPC64LE to make sure CPU scales.
sudo modprobe powernv-cpufreq
Next step is to Change the governor to the one that gives you best performance . Here is the list of default governors for different OS and what it can to be changed to for best performance. Please note the performance governor may not necessarily be needed for your workload, but is good to to have for getting best performance out of the system.
OS / Distribution
DEFAULT CPU Frequency scaling Governor on PPC
BEST CPU governor for performance on PPC
Linux Package needed to set governer
Linux command for setting governor to performance
None (since builds are missing kernel module for frequency scaling)
cpufreq-set -g performance
cpupower frequency-set -g performance
cpupower frequency-set -g performance
In my specific example, once we load this module on Fedora, and set the governor to “performance” Barreleye (openPOWER server) starts to run at the full frequency.
Good number of openPOWER servers run openBMC. OpenBMC as of today does NOT support network IPMI but it does support host-BMC IPMI. We can use this to set static IP on BMC using standard IPMI tool commands
Set openBMC to static IP from Host / Petitboot:
To set the BMC running openBMC to static address, run following ipmitool commands from petitboot / host.
ipmitool lan set 1 ipsrc static
ipmitool lan set 1 ipaddr 10.127.xx.xx
ipmitool lan set 1 netmask 255.255.252.0
ipmitool lan set 1 defgw ipaddr 10.127.xx.xx
ipmitool raw 0x06 0x40
You can print lan credentials using:
ipmitool lan print
Remember default credentials of BMC is root and 0penBmc
Set openBMC to get static IP from openBMC linux shell:
You can use standard linux “ip” command to set the IP address to static
To check interfaces
To force eth0 down
ip link set eth0 down
To force eth0 up
ip link set eth0 up
To set static ip
ipaddr add 192.168.1.1 dev eth0
To set up routing:
ip route add 192.168.1.0/24 via 192.168.1.1 dev eth0
OpenPOWER servers have Petitboot bootloader to manage the bootup options / preferences.
Since petitboot is relatively new I am putting together this guide to help walk through on how to set up devices to boot from any device (network or disk).
Power on the server and wait for Petitboot to come up via the VGA console (or OOB)
Select and click “system configuration” using the key-board arrows and enter key.
You will find below menu to customize the boot up options. In order to boot from any device, select and press enter on ” Clear and boot Any” option as below
Once you finish step 3, you will see that Boot Order in the first line of the screen short below changes to “(0) Any Device”
Now let’s save this boot config. For saving the boot order we have to go all the way down to the boot menu with down arrow key and select and press enter on “OK”
At this point you will be returned to petitboot menu and you have successfully set to boot from any device. From your next power on your server will automatically boot into an image that it finds first.
Power-off, Unrack and rack your server (or hard reboot) to test that “boot from any device” preference is working. When you un-rack and re-rack your server, you will have to wait few minutes (for BMC orange light to go off) before you press the host power on button.