How to mine Monero (XMR) on Rackspace Barreleye G2 Server (IBM POWER9)

Previous post, I’ve documented process of mining ETH / Ethereum on GPUs on POWER processor based server (called Barreleye ). Barreleye G2 is a open-source hardware server developed by Rackspace / Google.

In this post I explain, how to mine crypto-currency on CPU and still be profitable:

A. You pick the IBM POWER8 / 9 processors because of vector instructions supported on this CPU, helping the hashing algorithms (Specifically cryptonight algo – basis for Monero). (In your leisure, read this cryptonight standard that explains the basis of the algorithm itself )

B. You mine #MONERO – XMR because of its relative GPU / ASIC resistance. Monero has become one of the most popular cryptocurrencies due to its excellent privacy. Its website describes it as “secure, private, untraceable currency.”

For my guide I tested mining speeds on Rackspace server called : Barreleye G2 .
Example Speeds:  Hash-rate on Barreleye server running 2 POWER8 /POWER9 (10 core chip) (~5 KH/s) —- Equivalent to 9 x GeForce 1070 Mining hash-rate (475 H/s)

 

Currently verified operating systems are :

A. 16.04

B. CentOS 7.x

Here are the steps to follow for mining Monero on Barreleye G2 on Ubuntu 16.04. The only changes for CentOS will be in installing AT10 (look below for what AT10 is) and dependency package names (usually different for debain vs RHEL based repos):

Before starting make sure your SMT level is at 4 and not 8 (4-way multi-threading yields the best results – Discovery through experimentation)

ppc64_cpu –smt=4

1. Clone the xmr-stak-power repository for POWER based on xmr-stack-cpu and cd into source directory

git clone https://github.com/agangidi53/xmr-stak-power

cd xmr-stak-power

      2. Building a binary for mining on POWER needs gcc 6.3.1 which is available for public via Advanced toolchain 10. Install AT10.0 (Follow instructions below)

2a. Change Source list to add AT10 path

nano /etc/apt/sources.list

add “deb ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/ubuntu xenial at10.0” at the end of the file

apt-get update

2b. Install the 4 AT10.0 packages and add them to the PATH:

apt-get install advance-toolchain-at10.0-runtime \
advance-toolchain-at10.0-devel \
advance-toolchain-at10.0-perf \
advance-toolchain-at10.0-mcore-libs
export PATH=/opt/at10.0/bin:$PATH

3. Install dependencies before building the mining binary:

apt install libmicrohttpd-dev -y
apt install libhwloc-dev -y
apt install libssl-dev -y

4. Configure, Make and Build the XMR-STAK-POWER binary

export LD_LIBRARY_PATH=’/usr/lib/powerpc64le-linux-gnu/’

cmake

make

make install

Binary will be built and saved as <source-directory>/bin/xmr-stak/power

5. Create a Monero wallet and make not of wallet address and private keys

Use https://moneroaddress.org/

6. Replace above the Wallet address and your email in following file <source-directory>/bin/config.txt in below format.

We are using supportxmr pool so use the below template for pool address.

Only replace your email / wallet address. Follow template for everything else.

“pool_address” : “pool.supportxmr.com:5555”,
“wallet_address” : “4945WAJVEC6A3ZM8hwWMrV15VSJeeAvUv3fRbwwMajToCQ2usQa2tefGyx6PFQwXqMfpk7dVdxX6BBqZfYibx3JD3UKzrFk”,
“pool_password” : “MinerName:<email-address>”,

Please note that the default config file assumes you have

8. Start mining by executing the binary you previously built. This binary will refer to config.txt to determine your wallet address, the pool you are joining and number of CPUs in the server etc. ( It assumes you have 2 x 10 core POWER proc. If its different config, you need to change the config file to reflect that )

Use a screen session to run the mining binary so that binary runs in background (and you don’t have to baby sit it)

apt install screen

screen

<hit enter for the message>

<source-director>/bin/xmr-stak/power

7. After mining for 5 minutes, login to supportxmr.com using these default credentials ::

username:     <your-wallet-address>

password:     <your-email-address provided in above config.txt>

8. Change your password to your choice after getting in to the website.

9. At this point the Top Right Corner on supportxmr.com shows your current hash-rate mined via the pool. Here’s test output using 2 Barreleye G2 servers:

Screen Shot 2017-11-20 at 4.00.15 AM

Advertisements
How to mine Monero (XMR) on Rackspace Barreleye G2 Server (IBM POWER9)

How to setup Megaraid Volume in Petitboot using Storcli

If this is your first time bringing up a OpenPOWER server and you wanted a to setup a megaRAID volume before you do an install: You should be reading this.

In order to  do this

  1. Power on the server and wait for Petitboot menu to show on your VGA or Out of Band console. This should take 4-5 mins .
  2. When should get to Petitboot menu and then press “Exit to Shell” option as show below:

Screen Shot 2016-06-09 at 11.31.40 AM

3. Once in petitboot shell, you can access a plugged in thumb drive with storcli64 binary or scp / wget the same binary from Petitboot shell. I am providing the binary corresponding to latest here:

4.  change ‘storcli64’ permissions :   “chmod 777 storcli64”

5.  Now you can use this storcli64 binary to create a VD that you can subsequently install. Here is a dictionary of Storcli commands you can use for doing the same:

Show all physical drives:

./storcli64 /c0 /eall /sall show

Show existing VD

 ./storcli64 /c0 /vall show

Delete ALL Existing VD

./storcli64 /c0 /vall del force

Configure: Add everything to RAID 0 ( Based on number of drives you have, command changes, in my case it was 14 drives)

 ./storcli64 /c0 add vd type=r0 drives=0:0-14

Show new VD

 ./storcli64 /c0 /vall show

Initiate:

 ./storcli64 /c0 /vall start init

Check if everything is online:

./storcli64 /c0 /eall /sall show

 

How to setup Megaraid Volume in Petitboot using Storcli

Optimizing CPU Frequency on OpenPOWER distributions

Before we go on to look at optimizing the CPU frequency on Power systems, it is important to know what frequency you are running at. For this make sure you have powerpc-utils package installed.

On Fedora 

dnf install powerpc-utils

On CentOS

yum install powerpc-utils

On Ubuntu:

apt-get install powerpc-utils

Once you install this package, you can run the following command to know the frequency your CPU can run at:

sudo ppc64_cpu  --frequency

STEP 1 of Optimization:

To optimize CPU frequency for OpenPOWER distributions for ppc64le we first need install the kernel module specific to POWER-PC Architecture and then change the governor to “performance” enhancing mode.

On Fedora and some other openPOWER distributions this kernel module is not installed by default. For example: On Barreleye server, because of absence of this module on Fedora 23, Kernel doesn’t have enough info to scale the CPU very well. And hence puts CPU at a defensive “2.0 Ghz” instead of full “3.5 Ghz”.

Here is how you load the kernel module specific to PPC64LE to make sure CPU scales.

sudo modprobe powernv-cpufreq
STEP 2:
Next step is to Change the governor to the one that gives you best performance . Here is the list of default governors for different OS and what it can to be changed to for best performance. Please note the performance governor may not necessarily be needed for your workload, but is good to to have for getting best performance out of the system.
OS / Distribution DEFAULT CPU Frequency scaling Governor on PPC BEST CPU governor for  performance on PPC Linux Package needed to set governer Linux command for setting governor to performance
Fedora None (since builds are missing kernel module for frequency scaling) performance cpufrequtils cpufreq-set -g performance
CentOS conservative performance cpufrequtils cpupower frequency-set -g performance
Ubuntu ondemand performance cpufrequtils cpupower frequency-set -g performance

 

In my specific example, once we load this module on Fedora, and set the governor to “performance” Barreleye  (openPOWER server) starts to run at the full frequency.
Before loading the module:
[root@localhost ~]# sudo ppc64_cpu  –frequency
min:    2.063 GHz (cpu 126)
max:    2.063 GHz (cpu 0)
avg:    2.063 GHz
 After loading the module:
[live@localhost ~]$ sudo ppc64_cpu  –frequency
min:    3.494 GHz (cpu 120)
max:    3.494 GHz (cpu 7)
avg:    3.494 GHz
Optimizing CPU Frequency on OpenPOWER distributions

How to set openBMC to use static address

Good number of openPOWER servers run openBMC. OpenBMC as of today does NOT support network IPMI but it does support host-BMC IPMI. We can use this to set static IP on BMC using standard IPMI tool commands

Set openBMC to static IP from Host / Petitboot:

To set the BMC running openBMC to static address, run following ipmitool commands from petitboot / host.

   ipmitool lan set 1 ipsrc static

   ipmitool lan set 1 ipaddr 10.127.xx.xx

   ipmitool lan set 1 netmask 255.255.252.0

   ipmitool lan set 1 defgw ipaddr 10.127.xx.xx

   ipmitool raw 0x06 0x40

You can print lan credentials using:

ipmitool lan print

Remember default credentials of BMC is    root   and 0penBmc

Set openBMC to get static IP from openBMC linux shell:

You can use standard linux “ip” command to set the IP address to static

To check interfaces
ip a
To force eth0 down
ip link set eth0 down
To force eth0 up
ip link set eth0 up
To set static ip
ip addr add 192.168.1.1 dev eth0
To set up routing:
ip route add 192.168.1.0/24 via 192.168.1.1 dev eth0

How to set openBMC to use static address

How to set OpenPOWER server to boot from any device

OpenPOWER servers have Petitboot bootloader to manage the bootup options / preferences.

Since petitboot is relatively new I am putting together this guide to help walk through on how to set up devices to boot from any device (network or disk).

Step 1. 

Power on the server and wait for Petitboot to come up via the VGA console (or OOB)

Step 2:

Select and click “system configuration” using the key-board arrows and enter key.

Screen Shot 2016-06-09 at 11.14.06 AM

Setup 3:

You will find below menu to customize the boot up options. In order to boot from any device, select and press enter on ” Clear and boot Any” option as below

Screen Shot 2016-06-09 at 11.29.28 AM

Setup 4:

Once you finish step 3, you will see that Boot Order in the first line of the screen short below changes to “(0) Any Device”

Screen Shot 2016-06-09 at 11.29.57 AM

Step 5:

Now let’s save this boot config. For saving the boot order we have to go all the way down to the boot menu with down arrow key and select and press enter on “OK”

Screen Shot 2016-06-09 at 11.30.27 AM

Step 6:

At this point you will be returned to petitboot menu and you have successfully set to boot from any device. From your next power on your server will automatically boot into an image that it finds first.

Power-off, Unrack and rack your server (or hard reboot) to test that “boot from any device” preference is working. When you un-rack and re-rack your server, you will have to wait few minutes (for BMC orange light to go off) before you press the host power on button.

Screen Shot 2016-06-09 at 11.31.40 AM

How to set OpenPOWER server to boot from any device

How to Install Ubuntu Xenial 16.04 on PPC64LE / OpenPOWER

Here’s how to install Ubuntu 16.04 on PPC64LE server. In my case Barreleye, our openPOWER server is running OPAL as BIOS firmware with Petitboot bootloader:

Here are steps through install:

1) Login to Out of band management and get into host console

  • Logging into host OOB console  is important since Ubuntu 16.04 PPC64EL installer doesn’t have  ASPEED VGA / Graphics drivers. So you cannot see the install progress through the VGA console.
  • In your case this might be a ipmi based command to get to host console
  • In my case since we are running a openBMC firmware for our out of band needs :
    • a) SSH into BMC
    • b) Typed the command:       obmc-console-client

2) Power on the server and get to Petitboot bootloader. Which should look like below:

Screen Shot 2016-05-06 at 1.37.12 PM

2) Now type “n” to have add a new boot option.

We are netboot installing 16.04. So let’s grab and add the kernel and initrd url’s for Xenial (16.04 code name) respectively:

Kernel:

http://ports.ubuntu.com/ubuntu-ports/dists/xenial/main/installer-ppc64el/current/images/netboot/ubuntu-installer/ppc64el/vmlinux

Initrd:

http://ports.ubuntu.com/ubuntu-ports/dists/xenial/main/installer-ppc64el/current/images/netboot/ubuntu-installer/ppc64el/initrd.gz

Your screen should look like this:

Screen Shot 2016-05-06 at 1.38.58 PM

3) Press OK and you should return to Petitboot main screen. The boot option you just added should come up as “User Item 1”:

Screen Shot 2016-05-06 at 1.39.25 PM

4) Select and enter “User Item 1” for starting the install. Following which you will go through standard Ubuntu Install process, that you should be familiar with.

 

Thanks I hope this post helps you Install not just Ubuntu 16.04 but any OS of your choice using Petitboot.

 

How to Install Ubuntu Xenial 16.04 on PPC64LE / OpenPOWER

How to build openBMC binary for Barreleye

Barreleye is the openPOWER server that Rackspace helped build.  It runs openBMC for it’s management.

In my work at Rackspace, I often build custom openBMC builds for the sake of testing stability / new features.

Here’s how you can build your own openBMC binary for Barreleye.

If you are building on Ubuntu you need to install following packages before you build openBMC:

sudo apt-get install -y git build-essential libsdl1.2-dev texinfo gawk chrpath diffstat

If you are building on Fedora 23 you need to install the following packages before you build openBMC

sudo dnf install -y git patch diffstat texinfo chrpath SDL-devel bitbake
sudo dnf groupinstall "C Development Tools and Libraries"

 

Build Process:

First, clone the openBMC repository. If you are cloning form master this is how you’d do it:

git clone https://github.com/openbmc/openbmc

Then run the following commands:

cd openbmc
TEMPLATECONF=meta-openbmc-machines/meta-openpower/meta-rackspace/meta-barreleye/conf . oe-init-build-env
bitbake obmc-phosphor-image

 

If you run it a laptop with decent power, it takes about 2-3 hours. If you run it on a Barreleye (this is where I make you jealous) it takes about 35 minutes to build the binary.

The built binaries are in openbmc/build/tmp/deploy/images/barreleye location

In my case I use the barreleye-20160818194255.all.tar image and untar it to get an file named ‘image-bmc’ about 32MB in size .

You can  use a flash programmer to flash your BMC chip with this new version or use the openBMC online upgrade process.

If you wanna rebuild your binary, then you will have to first clear your config before you run above 3 commands again.

This is the command to clear the config:

rm -rf build/conf

Comment here for any issues you run into, while building the binary or if you want a link to a latest binary.

 

How to build openBMC binary for Barreleye