OpenPOWER at OpenCompute Summit 2018

I usually write about engineering topics surrounding openPOWER servers in this blog. This time, I thought it would be nice to extend that to list exciting things that are going to be shared at OCP Summit 2018

If you are attending Open Compute Summit 2018 (March 20-21) in San Jose, and want to see some cool things that OpenPOWER is bringing to the Open Compute eco-system, you are in the right place.



Here’s a list of things you could attend / visit to see what’s in store with Power9 in OCP form-factor .


Engineering Talks:

Accelerator Eco-System on Google / Rackspace – Zaius / Barreleye G2 Server

Tri-mode (SAS / SATA / NVMe) Storage Solution on Rackspace OCP Barreleye G2 Server

Accelerating Flash Memory with the High-Performance, Low-Latency OpenCAPI Interface

Hardware Showcase at OpenPOWER booth # C6 :

Witness OpenPOWER innovation in OCP form-factor servers

Zaius Tray , Google

Barreleye G2 Server, Rackspace

OCP Power9 Tray , Inventec

Molex Flash Storage Accelerator  .

OpenCAPI adapters – Alpha Data & Innova 2 (Mellanox)


Witness Gen4 Storage / Networks (2x faster) and OpenCAPI / NVLink 2.0 (3x faster) demos:

a) Gen4 based 2x 100 GbE networking

b) Gen4 based Storage Speeds

c) Tri-mode , Interchangeable SAS / SATA / NVMe storage

d) OpenCAPI buffer demo running at 25 GT/s (3x faster than PCIe Gen3)

OpenPOWER Member Booth at OCP:

a) Mellanox . Booth # C5

b) Broadcom: Booth #



How to Netboot install RHEL 7.4 on POWER9 / Barreleye G2

RHEL 7.4 as I’m aware is the only GA’ed enterprise OS on POWER9 and hence Barreleye G2. I’ve seen couple of folks have issues picking the right image / boot arguments while attempting this install , and fail, so wanted to clearly put the steps down here:

If you instead want to install Ubuntu 16.04 LTS on POWER9 I’ve already written instructions for that in previous post: 

  1. First grab the DVD (2.9 GB) from RHEL website. For the purposes of this post we’ll use evaluation version:

> Go to RHEL downloads page:—7/7.4/ppc64le/product-software

> Login and Get Download Link “Red Hat Enterprise Linux for Power 9”


NOTE1: DO-NOT get link for generic “RHEL big-endian for ppc64” and “RHEL little-endian ppc64”) rather get specific version of download provided for POWER9

NOTE:2 : Download the expansive DVD ( 2.9GB binary as RHEL calls it ) , just the basic ISO file is NOT enough for netboot, since there is no public mirror for repo / package data, that I’m aware of.

See picture below, get the second file. In my case file name was “rhel-alt-server-7.4-ppc64le-dvd.iso”. If you are missing the alt, you are getting the wrong file:

Screen Shot 2018-01-19 at 4.28.54 PM

2.  Once you get the download link for this image, wget it to local (apache) http server (obviously, this is different than the POWER machine you are doing the install) and mount that iso to a folder on the http server path

Log into

ssh root@<http-server>

cd /var/www/html/

mkdir rhel9

cd rhel9

wget <hyperlink to rhel-alt-server-7.4-ppc64le-boot.iso>

mount -o loop rhel-alt-server-7.4-ppc64le-dvd.iso\?_auth_\=1516572534_bc1326574be188a5e87bb59dfcd70b8e rhel9

See picture below for help:

Screenshot from 2018-01-21 20-45-36

3. Now get to main petitboot menu and add a new boot option by  “n” button. Fill in the new boot option menu with following mock links as example :

Kernel:  http://10.127.xx.xx/rhel/ppc/ppc64/vmlinuz

Inirdrd: http://10.127.xx.xx/rhel/ppc/ppc64/initrd.img

boot-arguments:    root=live:http://10.127.xx.xx/rhel/LiveOS/squashfs.img

4. Click save and execute the new boot option you just created “User item 1”:

Screen Shot 2016-05-06 at 1.39.25 PM

5. That will boot to RHEL 7.4 Install menu in about 2 minutes or so. Select “Text mode” as I ran into some issues getting VNC install working:

6. You will be greeted with the following text menu screen. The only tricky item here is number 3, “Installation Source”.

Select http mirror and enter the mount location of your iso as the selection:

Screenshot from 2018-01-21 20-28-26

7. After completing all the necessary options 1 through 9 , press “b” to begin the installation begin the installation.

POWER9 RHEL 7.4 Barreleye Installation Done

8.Installation should be done in under 7-8 minutes as we are choosing to install from local mirror:

Screenshot from 2018-01-21 20-42-42

Don’t hesitate to comment below for any questions.

How to mine Monero (XMR) on Rackspace Barreleye G2 Server (IBM POWER9)

Previous post, I’ve documented process of mining ETH / Ethereum on GPUs on POWER processor based server (called Barreleye ). Barreleye G2 is a open-source hardware server developed by Rackspace / Google.

In this post I explain, how to mine crypto-currency on CPU and still be profitable:

A. You pick the IBM POWER8 / 9 processors because of vector instructions supported on this CPU, helping the hashing algorithms (Specifically cryptonight algo – basis for Monero). (In your leisure, read this cryptonight standard that explains the basis of the algorithm itself )

B. You mine #MONERO – XMR because of its relative GPU / ASIC resistance. Monero has become one of the most popular cryptocurrencies due to its excellent privacy. Its website describes it as “secure, private, untraceable currency.”

For my guide I tested mining speeds on Rackspace server called : Barreleye G2 .
Example Speeds:  Hash-rate on Barreleye server running 2 POWER8 /POWER9 (10 core chip) (~5 KH/s) —- Equivalent to 9 x GeForce 1070 Mining hash-rate (475 H/s)


Currently verified operating systems are :

A. 16.04

B. CentOS 7.x

Here are the steps to follow for mining Monero on Barreleye G2 on Ubuntu 16.04. The only changes for CentOS will be in installing AT10 (look below for what AT10 is) and dependency package names (usually different for debain vs RHEL based repos):

Before starting make sure your SMT level is at 4 and not 8 (4-way multi-threading yields the best results – Discovery through experimentation)

ppc64_cpu –smt=4

1. Clone the xmr-stak-power repository for POWER based on xmr-stack-cpu and cd into source directory

git clone

cd xmr-stak-power

      2. Building a binary for mining on POWER needs gcc 6.3.1 which is available for public via Advanced toolchain 10. Install AT10.0 (Follow instructions below)

2a. Change Source list to add AT10 path

nano /etc/apt/sources.list

add “deb xenial at10.0” at the end of the file

apt-get update

2b. Install the 4 AT10.0 packages and add them to the PATH:

apt-get install advance-toolchain-at10.0-runtime \
advance-toolchain-at10.0-devel \
advance-toolchain-at10.0-perf \
export PATH=/opt/at10.0/bin:$PATH

3. Install dependencies before building the mining binary:

apt install libmicrohttpd-dev -y
apt install libhwloc-dev -y
apt install libssl-dev -y

4. Configure, Make and Build the XMR-STAK-POWER binary

export LD_LIBRARY_PATH=’/usr/lib/powerpc64le-linux-gnu/’



make install

Binary will be built and saved as <source-directory>/bin/xmr-stak/power

5. Create a Monero wallet and make not of wallet address and private keys


6. Replace above the Wallet address and your email in following file <source-directory>/bin/config.txt in below format.

We are using supportxmr pool so use the below template for pool address.

Only replace your email / wallet address. Follow template for everything else.

“pool_address” : “”,
“wallet_address” : “4945WAJVEC6A3ZM8hwWMrV15VSJeeAvUv3fRbwwMajToCQ2usQa2tefGyx6PFQwXqMfpk7dVdxX6BBqZfYibx3JD3UKzrFk”,
“pool_password” : “MinerName:<email-address>”,

Please note that the default config file assumes you have

8. Start mining by executing the binary you previously built. This binary will refer to config.txt to determine your wallet address, the pool you are joining and number of CPUs in the server etc. ( It assumes you have 2 x 10 core POWER proc. If its different config, you need to change the config file to reflect that )

Use a screen session to run the mining binary so that binary runs in background (and you don’t have to baby sit it)

apt install screen


<hit enter for the message>


7. After mining for 5 minutes, login to using these default credentials ::

username:     <your-wallet-address>

password:     <your-email-address provided in above config.txt>

8. Change your password to your choice after getting in to the website.

9. At this point the Top Right Corner on shows your current hash-rate mined via the pool. Here’s test output using 2 Barreleye G2 servers:

Screen Shot 2017-11-20 at 4.00.15 AM

How to update openPOWER BIOS / host flash / boot firmware (without wiping boot preferences)

There are 2 ways you can update host / boot firmware in openPOWER machines

a) Via BMC console


In this post we will see the steps for updating the host firmware via BMC console:

Step1: Get to the BMC console (via ssh)

ssh root@<openBMC IP>

Step2: Now we want to make sure host power is off

obmcutil poweroff

Step3: Get the host flash binary into  /tmp location on BMC

cd /tmp

wget <link to host firmware>

Step4:  Start flashing the new firmware using ‘pflash’ utility. Once you know for sure host power is off ( you can check it via <obmcutil state>)

pflash -E -f -p  <flash-binary-name>.pnor

Step5: Once you see the flash progress reach 100% and complete, try to get the power on

obmcutil poweron

Step6: Check the console , if your host is booting successfully



Above steps (specifically the pflash command above) completely wipe out your flash chip (including boot preferences) and replace it with new host firmware. If you, instead wanted to save boot preferences then you need to back up the NVRAM partition using the flow below:

cd /tmp

pflash -P NVRAM -r nvram

pflash -E -f -p <flash-binary-name>.pnor  

pflash -e -P NVRAM -p nvram

rm -rf nvram

How to mine Ethereum on Rackspace BarreleyeG2 OpenPOWER server with Nvidia P100 GPUs

OpenPOWER CPUs and Nvidia GPUs represent the best in class computing Power per $

To test that, I thought I’ll give compiling Ethereum mining suite a try on our Barreleye G2 server. As background information Barreleye G2 is a server Rackspace is building in collaboration with Google, IBM and Ingrasys

Screen Shot 2017-09-02 at 9.56.50 PM

To compile Ethereum on Power9 with Nvidia P100 GPUs you have 2 options:

A)  Compile it for CUDA

B)  Compile it for OpenCL

In this post we will go with Option A (Compile it with CUDA). To do this you need to first get Nvidia Drivers and CUDA working on your openPOWER server. The link to blog post on how to get these working is here:

Once you have CUDA working (run an example CUDA application to make sure), install the build dependencies necessary for compiling ethereum. We will use cmake to configure and build.

sudo apt-get install libleveldb-dev libmicrohttpd-dev cmake

After looking around I realized that main github source for ethereum “ethereum/cpp-ethereum” no longer supports mining for GPUs. So instead I used: for my mining experiments with Nvidia P100 GPUs. I found the release “0.12.0.dev2” to be stable.

Download the latest official release zip and unzip



cd ethminer-0.12.0.dev2/

Make a build directory for your build activities and make that your current directory

mkdir build
cd build

Since I’m building for CUDA and NOT OpenCL, I made sure, my cmake configuration reflects that. This step 3-4 minutes on Barreleye G2 server


Once cmake configure is done, we get to the fun part, building:

cmake –build .

If your build from source looks good, your screen should roughly look like this:

Screen Shot 2017-09-02 at 8.00.00 PM

Make install the ethminer binary

sudo make install

At this point you have successfully built your ”ethminer” binary and it should reside in “/usr/local/bin/ethminer”

Play around with your newly built binary with command-line options:

/usr/local/bin/ethminer –help

Before actually mining you can benchmark / simulate to see how many Hash Rate (MH/s) can you hit

/usr/local/bin/ethminer -M -U

Screen Shot 2017-09-02 at 9.22.26 PM

Once you have simulated and your expected MH/s match your expectations. Its time to actually mine ethereum.

You need to create a ethereum wallet and get your own address to send your mined coins to. Go to the below address, create and save your credentials safely

Once you have you created <ethereum address> and think of <miner tag> to identify your machine (helpful you are mining via multiple machines)

/usr/local/bin/ethminer –farm-recheck 200 -U -S -FS -O <wallet address>.<tag>

/usr/local/bin/ethminer –farm-recheck 200 -U -S -FS -O 0x4ff2de61282aa5da02E5F8399DB7d47A66Be1465.barreleyeg2

Soon your will receive console output indicating completed shares and your Hash rate:

Screen Shot 2017-09-02 at 9.44.26 PM

Now how do you confirm / check the number of shares / amount of ethereum you have mine ? Got to below website and look up your wallet address from above:

You will be able to see details like below:

Screen Shot 2017-09-02 at 9.48.41 PM

You can use below command to check how much power your nvidia GPUs are consuming


Please comment below if you have questions or are interested in price of this server / setup OR profitability.

How to Install NVIDIA Drivers for P100 and CUDA on Barreleye G2 / Power9 (Work in Progress)

I’m trying to get Nvidia Telsa P100 working with CUDA on Barreleye G2 Server.  Barreleye G2 is Power9 / OpenPOWER server that Rackspace and Google are building.

Posting the process I’m using, here, to help others trying to do the same.

Before Installing CUDA, make sure you have the right drivers for Nvidia Devices:

You have 2 Alternatives for Installing Drivers for NVIDIA devices:

1) Install the Power8 Drivers from NVIDIA website for Power9:


dpkg -i nvidia-driver-local-repo-ubuntu1604-384.59_1.0-1_ppc64el.deb


2) Install the APT recommended Drivers:

sudo apt-get install ubuntu-drivers-common
sudo ubuntu-drivers devices

Based  on output recommendation from above command, install recommended driver. For Example:
sudo apt-get install nvidia-384

Installing CUDA on Power9 / Barreleye G2:

Install dependency packages for CUDA

sudo apt-get install build-essential

Install repository packages for CUDA specific to ppc and your build (in my case 16.04). For the other builds, lookup Nvidia website for specific deb package
sudo dpkg -i ./cuda-repo-ubuntu1604-8-0-local-ga2_8.0.54-1_ppc64el-deb

Update the APT Definitions (You need to do this for above repo package to take effect ):

sudo apt-get update

Install CUDA Libraries

sudo apt-get install cuda

Make CUDA Accessible to all users:

echo ‘export PATH=$PATH:/usr/local/cuda-8.0/bin’ | sudo tee
echo /usr/local/cuda-8.0/lib64 | sudo tee /etc/
sudo ldconfig

Check your Drivers and Packages are all working together and you see devices, modules and details of your hardware 

sudo dmesg | grep -i nvidia
sudo lsmod | grep nvidia
nvidia-smi –list-gpus

If things went okay, output for above 3 commands, should be meaningful.

Now lets check if CUDA install went okay by building and running CUDA applications:

mkdir ~/samples
cp -r /usr/local/cuda-8.0/samples/ ~/samples/
cd ~/samples/samples/7_CUDALibraries/simpleCUFFT

In my case, CUDA application fails to run with following errors. I’m trying to resolve if this is Power9 / PPC OS related error or Nvidia device error. Will edit this post with details. Process for install should remain same neverthless.

root@ubuntu:~/samples/samples/7_CUDALibraries/simpleCUFFT# ./simpleCUFFT
[simpleCUFFT] is starting…
GPU Device 0: “Tesla P100-PCIE-16GB” with compute capability 6.0

[ 3457.484507] Severe Machine check interrupt [[Not recovered]
[ 3457.485099] Initiator: CPU
[ 3457.485332] Error type: Real address [Load/Store (foreign)]
[ 3457.485762] Effective address: 00003fff9e49208c
Bus error (core dumped)


How to Install Ubuntu Xenial 16.04 LTS on Power9 Machines

If you want to install Original LTS Ubuntu 16.04 (with initial LTS Kernel 4.4) on Power9, you were / are out of luck. This is because full kernel support for Power9 got in from 4.10 on wards.

But to some jubilation, 16.04.3 LTS got released today (08/03/17) with support for 17.04 Kernel (4.10). So work around to install Ubuntu LTS on Power9 is to use 16.04.03 LTS HWE kernel / initrd instead of ones I indicated in my previous blog post:



Again, Follow the same procedure in previous blog post  BUT with NEW kernel / initrd links.

Enjoy the first LTS port on Power9