EPYC / Skylake vs Power9 STREAM Memory bandwidth comparison via Zaius / Barreleye G2

Using stream benchmark for measuring memory bandwidth is a industry standard practice and I followed the same. For the x86 systems, to be unbiased, I picked the ‘Stream Triad’ results from a reputable benchmarking org (Anandtech).

Power9 CPU Config used for STREAM testing:

root@ubuntu:/home/ubuntu# lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 176
Thread(s) per core: 4
Core(s) per socket: 22
Socket(s): 2
NUMA node(s): 2
Model: 2.2 (pvr 004e 1202)
Model name: POWER9, altivec supported

Memory Config used for STREAM testing:

16x   16GiB RDIMM DDR4 2666 MHz (0.4ns)

Theoretical Memory bandwidth:

Theoretical Memory Bandwidth Calculation on Barreleye G2:


= 8*8*2.666*2 = 341.248 GB/s

Compiler and run instructions for measurement:

wget http://www.cs.virginia.edu/stream/FTP/Code/stream.c

gcc -m64 -O3 -fopenmp -DSTREAM_ARRAY_SIZE=536895856 -DNTIMES=20 -mcmodel=large stream.c -o stream



Stream Application Barreleye G2 – 2 x22 core (2400 MHz)


Barreleye G2 2x 22 core (2666 MHz)


AMD EPYC 32c 7601 (Anandtech)      2x Intel Skylake 8176 (Anandtech)
Stream Copy (MB/s) 217909.8 241641.7
Stream Add (MB/s) 240561.6 253784
Stream Scale (MB/s) 245069.7 268929.6
Stream Triad (MB/s) 247078.8 270000.4 207000 165000


Pictorial Representation of results:

Screen Shot 2018-07-19 at 9.41.34 AM.png

How to Netboot install RHEL 7.4 on POWER9 / Barreleye G2

RHEL 7.4 as I’m aware is the only GA’ed enterprise OS on POWER9 and hence Barreleye G2. I’ve seen couple of folks have issues picking the right image / boot arguments while attempting this install , and fail, so wanted to clearly put the steps down here:

If you instead want to install Ubuntu 16.04 LTS on POWER9 I’ve already written instructions for that in previous post: 

  1. First grab the DVD (2.9 GB) from RHEL website. For the purposes of this post we’ll use evaluation version:

> Go to RHEL downloads page:


> Login and Get Download Link “Red Hat Enterprise Linux for Power 9”


NOTE1: DO-NOT get link for generic “RHEL big-endian for ppc64” and “RHEL little-endian ppc64”) rather get specific version of download provided for POWER9

NOTE:2 : Download the expansive DVD ( 2.9GB binary as RHEL calls it ) , just the basic ISO file is NOT enough for netboot, since there is no public mirror for repo / package data, that I’m aware of.

See picture below, get the second file. In my case file name was “rhel-alt-server-7.4-ppc64le-dvd.iso”. If you are missing the alt, you are getting the wrong file:

Screen Shot 2018-01-19 at 4.28.54 PM

2.  Once you get the download link for this image, wget it to local (apache) http server (obviously, this is different than the POWER machine you are doing the install) and mount that iso to a folder on the http server path

Log into

ssh root@<http-server>

cd /var/www/html/

mkdir rhel9

cd rhel9

wget <hyperlink to rhel-alt-server-7.4-ppc64le-boot.iso>

mount -o loop rhel-alt-server-7.4-ppc64le-dvd.iso\?_auth_\=1516572534_bc1326574be188a5e87bb59dfcd70b8e rhel9

See picture below for help:

Screenshot from 2018-01-21 20-45-36

3. Now get to main petitboot menu and add a new boot option by  “n” button. Fill in the new boot option menu with following mock links as example :

Kernel:  http://10.127.xx.xx/rhel/ppc/ppc64/vmlinuz

Inirdrd: http://10.127.xx.xx/rhel/ppc/ppc64/initrd.img

boot-arguments:    root=live:http://10.127.xx.xx/rhel/LiveOS/squashfs.img

4. Click save and execute the new boot option you just created “User item 1”:

Screen Shot 2016-05-06 at 1.39.25 PM

5. That will boot to RHEL 7.4 Install menu in about 2 minutes or so. Select “Text mode” as I ran into some issues getting VNC install working:

6. You will be greeted with the following text menu screen. The only tricky item here is number 3, “Installation Source”.

Select http mirror and enter the mount location of your iso as the selection:

Screenshot from 2018-01-21 20-28-26

7. After completing all the necessary options 1 through 9 , press “b” to begin the installation begin the installation.

POWER9 RHEL 7.4 Barreleye Installation Done

8.Installation should be done in under 7-8 minutes as we are choosing to install from local mirror:

Screenshot from 2018-01-21 20-42-42

Don’t hesitate to comment below for any questions.

How to mine Ethereum on Rackspace BarreleyeG2 OpenPOWER server with Nvidia P100 GPUs

OpenPOWER CPUs and Nvidia GPUs represent the best in class computing Power per $

To test that, I thought I’ll give compiling Ethereum mining suite a try on our Barreleye G2 server. As background information Barreleye G2 is a server Rackspace is building in collaboration with Google, IBM and Ingrasys

Screen Shot 2017-09-02 at 9.56.50 PM

To compile Ethereum on Power9 with Nvidia P100 GPUs you have 2 options:

A)  Compile it for CUDA

B)  Compile it for OpenCL

In this post we will go with Option A (Compile it with CUDA). To do this you need to first get Nvidia Drivers and CUDA working on your openPOWER server. The link to blog post on how to get these working is here:

Once you have CUDA working (run an example CUDA application to make sure), install the build dependencies necessary for compiling ethereum. We will use cmake to configure and build.

sudo apt-get install libleveldb-dev libmicrohttpd-dev cmake

After looking around I realized that main github source for ethereum “ethereum/cpp-ethereum” no longer supports mining for GPUs. So instead I used: https://github.com/ethereum-mining/ethminer for my mining experiments with Nvidia P100 GPUs. I found the release “0.12.0.dev2” to be stable.

Download the latest official release zip and unzip

wget https://github.com/ethereum-mining/ethminer/archive/v0.12.0.dev2.zip

unzip v0.12.0.dev2.zip

cd ethminer-0.12.0.dev2/

Make a build directory for your build activities and make that your current directory

mkdir build
cd build

Since I’m building for CUDA and NOT OpenCL, I made sure, my cmake configuration reflects that. This step 3-4 minutes on Barreleye G2 server


Once cmake configure is done, we get to the fun part, building:

cmake –build .

If your build from source looks good, your screen should roughly look like this:

Screen Shot 2017-09-02 at 8.00.00 PM

Make install the ethminer binary

sudo make install

At this point you have successfully built your ”ethminer” binary and it should reside in “/usr/local/bin/ethminer”

Play around with your newly built binary with command-line options:

/usr/local/bin/ethminer –help

Before actually mining you can benchmark / simulate to see how many Hash Rate (MH/s) can you hit

/usr/local/bin/ethminer -M -U

Screen Shot 2017-09-02 at 9.22.26 PM

Once you have simulated and your expected MH/s match your expectations. Its time to actually mine ethereum.

You need to create a ethereum wallet and get your own address to send your mined coins to. Go to the below address, create and save your credentials safely


Once you have you created <ethereum address> and think of <miner tag> to identify your machine (helpful you are mining via multiple machines)

/usr/local/bin/ethminer –farm-recheck 200 -U -S us1.ethermine.org:4444 -FS eu1.ethermine.org:4444 -O <wallet address>.<tag>

/usr/local/bin/ethminer –farm-recheck 200 -U -S us1.ethermine.org:4444 -FS eu1.ethermine.org:4444 -O 0x4ff2de61282aa5da02E5F8399DB7d47A66Be1465.barreleyeg2

Soon your will receive console output indicating completed shares and your Hash rate:

Screen Shot 2017-09-02 at 9.44.26 PM

Now how do you confirm / check the number of shares / amount of ethereum you have mine ? Got to below website and look up your wallet address from above:


You will be able to see details like below:

Screen Shot 2017-09-02 at 9.48.41 PM

You can use below command to check how much power your nvidia GPUs are consuming


Please comment below if you have questions or are interested in price of this server / setup OR profitability.