Zaius / Barreleye G2 – Power9 and AST2500 running OpenBMC
Part of integrating a new openPOWER Zaius / Barreleye G2 in a companies’ environment comes with a couple of new-ness.:
A) Not all openPOWER servers have APSS which provides the external power measurement for CPU, DIMMs and other components.
B) Not all openPOWER servers fully support IPMIoverLAN because of leveraging openBMC. OpenBMC does this better, it provides you API access to all the sensors / management controls.
However to mitigate this newness, I’ve document I’ve provided below information to help you in your journey.
Power Rail Information via Host:
Commands:
apt install lm-sensors
sensors
Output:
Chip 0 : 70.00 W
Chip 0 Vdd: 14.00 W
Chip 0 Vdn: 21.00 W
Chip 8 : 79.00 W
Chip 8 Vdd: 19.00 W
Chip 8 Vdn: 25.00 W
Section2
REST API BMC Access:
OpenBMC – Host Code Update
After build, you will end up with a set of image files in:
tmp/deploy/images//.
The images corresponds to components that can be updated on the BMC.
The whole flash image for BMC
image-bmc → flash--
The small initramfs image is used for early init and flash management
image-initramfs → core-image-minimal-initramfs-palmetto.cpio.lzma.u-boot
The OpenBMC kernel cuImage (combined kernel and device tree)
image-kernel → cuImage
The read-only OpenBMC file system
image-rofs → obmc-phosphor-image-palmetto.squashfs-xz
The read-write file system for persistent changes to the OpenBMC file system
image-rwfs → rwfs.jffs2
The OpenBMC boot-loader.
image-u-boot → u-boot.bin
Preparing for BMC code Update
The BMC normally runs with the read-write and read-only file systems mounted, which means these images may be read (and written, for the read-write file-system) at any time. Because the updates are distributed as complete file system images, these file-systems have to be unmounted to replace them with new images. To unmount these file-systems all applications must be stopped.
By default an orderly reboot will stop all applications and unmount the root file-system, and the images copied into the /run/initramfs directory will be applied at that point before restarting. This also applies to the shutdown and halt commands — they will also write the flash before stopping.
As an alternative, an option can be parsed by the init script in the initramfs to copy the required contents of these file-systems into RAM so that the images can be applied while the rest of the application stack is running and progress can be monitored over the network. The update script can then be called to write the images while the
system is operational and its progress output monitored.
Update from the OpenBMC Shell
To update from the OpenBMC shell, follow the steps in this section. It is recommended that the BMC be prepared for the update first as shown below :
fw_setenv openbmconce copy-files-to-ram copy-base-filesystem-to-ram reboot
Copy one or more of these image-* files to the directory:
/run/initramfs/
(preserving the filename), then run the update script to apply the images:
/run/initramfs/update
then reboot to finish applying:
reboot
During the reboot process the update script will be invoked after the
file systems are unmounted to complete the update process.
Some optional features are available, see the help for more details:
/run/initramfs/update –help
Update via REST
An OpenBMC system can download an update image from a TFTP server, and apply updates, controlled via REST. The general procedure is:
- Prepare system for update
- Configure update settings
- Initiate update
- Check flash status
- Reboot the BMC
Prepare system for update
Perform a POST to invoke the PrepareForUpdate method of the /flash/bmc object:
curl -b cjar -k -H "Content-Type: application/json" -X POST \
-d '{"data": ["<TFTP server IP address>", "<filename>"]}' \
https://bmc/org/openbmc/control/flash/bmc/action/prepareForUpdate
This will setup the u-boot environment and reboot the BMC. If no other images were pending the BMC should return in about 2 minutes.
Configure update settings
There are a few settings available to control the update process:
preserve_network_settings Preserve network settings, only needed if updating the whole flash
restore_application_defaults Update (clear) the read-write file system
update_kernel_and_apps_only Update kernel and initramfs
clear_persistent_files Ignore the persistent file list when resetting applications defaults
auto_apply Attempt to write the images by invoking the Apply method after the images are unpacked.
To configure the update settings, perform a REST PUT to /control/flash/bmc/attr/<setting>. For example:
curl -b cjar -k -H "Content-Type: application/json" -X PUT \
-d '{"data": 1}' \
https://bmc/org/openbmc/control/flash/bmc/attr/preserve_network_settings
Initiate update
Perform a POST to invoke the updateViaTftp method of the /flash/bmc object:
curl -b cjar -k -H "Content-Type: application/json" -X POST \
-d '{"data": ["<TFTP server IP address>", "<filename>"]}' \
https://bmc/org/openbmc/control/flash/bmc/action/updateViaTftp
Check flash status
You can query the progress of the download and image verification with a simple GET request:
curl -b cjar -k https://bmc/org/openbmc/control/flash/bmc
Or perform a POST to invoke the GetUpdateProgress method of the /flash/bmc object:
curl -b cjar -k -H "Content-Type: application/json" -X POST \
-d '{"data": []}' \
https://bmc/org/openbmc/control/flash/bmc/action/GetUpdateProgress
Note: the status will not advance from Writing images to flash without calling the GetUpdateProgress method.
If the status is Image ready to apply then you can either initiate a reboot or call the Apply method to start the process of writing the flash.
Reboot the BMC
To start using the new images, reboot the BMC using the warmReset method of the BMC control object:
curl -b cjar -k -H "Content-Type: application/json" -X POST \
-d '{"data": []}' \
https://bmc/org/openbmc/control/bmc0/action/warmReset
Host Code Update
The host firmware (or “BIOS”) can be updated in a similar method. Because the BMC does not use the host firmware it is updated when the download is completed. This assumes the host is not accessing its firmware.
Perform a POST request to call the updateViaTftp method of /control/flash/bios (instead of /control/flash/bmc used above). To initiate the update:
curl -b cjar -k -H "Content-Type: application/json" -X POST \
-d '{"data": ["<TFTP server IP address>", "<filename>"]}' \
https://bmc/org/openbmc/control/flash/bios/action/updateViaTftp
And to check the flash status:
curl -b cjar -k https://bmc/org/openbmc/control/flash/bios
Serial Over LAN(host console)
The console infrastructure allows multiple shared connections to a single host UART. UART data from the host is outputted to all connections, and input from any connection is sent to the host.
Remote Console Connections
To connect to an OpenBMC console session remotely, just ssh to your BMC on port 2200. Use the same login credentials as you would for a normal ssh session:
$ ssh -p 2200 [user]@[bmc-hostname]
bmc-hostname is BMC IP address.
Local Console Connections
If you’re already logged into an OpenBMC machine, you can start a console session directly, using:
$ obmc-console-client
To exit from a console, type:
return ~
Note that if you’re on an ssh connection, you’ll need to ‘escape’ the ~ character, by entering it twice.
Logging
Console logs are kept in:
/var/log/obmc-console.log
This log is limited in size, and will wrap after hitting that limit (currently set at 16kB)
REST API
The primary management interface for OpenBMC is REST.
Host Management with OpenBMC
Accessible over REST.
Inventory
The system inventory structure is under the /xyz/openbmc_project/inventory hierarchy. Each event is a separate object under this structure, referenced by name.
In OpenBMC the inventory is represented as a path which is hierarchical to the physical system topology. Items in the inventory are referred to as inventory items and are not necessarily FRUs. If the system contains one chassis, a motherboard, and a CPU on the motherboard, then the path to that inventoryitem would be:
inventory/system/chassis/motherboard0/cpu0
The properties associated with an inventory item are specific to that item with the addition of these common properties:
version: a code version associated with this item
present: indicates whether this item is present in the system (True/False)
The usual list and enumerate REST queries allow the system inventory structure to be accessed. For example, to enumerate all inventory items and their properties:
curl -b jcar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/inventory/enumerate
Confirm BMC version:
curl -b jcar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/org/openbmc/inventory/system/chassis/motherboard/bmc
Check the status of the CPU0:
curl -b cjar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/inventory/system/chassis/motherboard/cpu0
Check the status of the CPU0 core<item>:
curl -b cjar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/inventory/system/chassis/motherboard/cpu0/enumerate
curl -b cjar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/inventory/system/chassis/motherboard/cpu0/corex
(e.g. core0 ~ core23)
Check the status of the CPU1:
curl -b cjar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/inventory/system/chassis/motherboard/cpu1
Check the status of the CPU1 core<item>:
curl -b cjar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/inventory/system/chassis/motherboard/cpu1/enumerate
curl -b cjar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/inventory/system/chassis/motherboard/cpu1/corex
(e.g. core0 ~ core23)
Check the status of the dimm<instance>:
curl -b cjar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/inventory/system/chassis/motherboard/dimmx
(e.g. dimm0 ~ dimm31)
Check the information of motherboard
curl -b cjar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/inventory/system/chassis/motherboard
Check the information of FPDB
curl -b cjar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/inventory/system/chassis/FPDB
Check the information of EXP
curl -b cjar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/inventory/system/chassis/EXP
Check the information of BP
curl -b cjar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/inventory/system/chassis/BP
Check the information of pcie
curl -b cjar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/inventory/system/chassis/BP
Check the information of pcie, shows 7 pcie slot
curl -b cjar -k -X GET -H "Content-Type:application/json"
/xyz/openbmc_project/inventory/system/chassis/pcie_card_xxx
(xxx should be pe0~pe4, e2a and e2b)
Sensors
The system inventory structure is under the /xyz/openbmc_project/sensors hierarchy.
This interface allows monitoring of system attributes like temperature or altitude, and are represented similar to the inventory, by object paths under the top-level sensors object name. The path categorizes the sensor and shows what the sensor represents, but does not necessarily represent the physical topology of the system.
The following sensor hierarchies are recognized:
- Temperature: Unit must be “DegreesC”.
- Voltage: Unit must be “Volts”.
- Fan_tach: Unit must be “RPMS”.
For example, all temperature sensors are under sensors/temperature whereas ambient temperature sensors would be sensors/temperature/ + /ambient_mb, /ambient_fpdb, /ambient_bp.
These are the common properties associated with all sensors:
- Value: current value of sensor
- Units: units of value
- WarningHigh & WarningLow: High & Low threshold for a warning error
- CriticalHigh & CriticalLow: High & Low threshold for a critical error
All Sensor value properties are read-only.
To enumerate all sensors in the system:
curl -b jcar -k -X GET https://BMC_IP/xyz/openbmc_project/sensors/enumerate
Check the temperature of ambient sensor on Motherboard:
curl -b cjar -k -X GET https://BMC_IP/xyz/openbmc_project/sensors/temperature/ambient_mb
Check the temperature of ambient sensor on FPDB:
curl -b cjar -k -X GET https://BMC_IP/xyz/openbmc_project/sensors/temperature/ambient_fpdb
Check the temperature of ambient sensor on BP:
curl -b cjar -k -X GET https://BMC_IP/xyz/openbmc_project/sensors/temperature/ambient_bp
Check the temperature of cpu cores:
curl -b cjar -k -X GET https://BMC_IP/xyz/openbmc_project/sensors/temperature/p0_core7_temp
(e.g. cpu0/core7)
Check the voltage of power sequencer(UCD90160):
curl -b cjar -k -X GET https://BMC_IP/xyz/openbmc_project/sensors/voltage/voutx
(e.g. vout1 ~ vout13)
Check the voltage of ADC:
curl -b cjar -k -X GET https://BMC_IP/xyz/openbmc_project/sensors/voltage/iio_hwmon/<item>
Check the status of the Fan_tach:
curl -b cjar -k -X GET https://BMC_IP/xyz/openbmc_project/sensors/fan_tach/tach/fanxL
(e.g. fan1L ~ fan5L)
curl -b cjar -k -X GET https://BMC_IP/xyz/openbmc_project/sensors/fan_tach/tach/fanxH
(e.g. fan1H ~ fan5H)
Check the current speed of fans (PWM)
curl -b cjar -k -X GET https://BMC_IP//xyz/openbmc_project/sensors/pwm/speed/fanx
State
The system inventory structure is under the /xyz/openbmc_project/state hierarchy.
The OpenBMC is an interface to control and track the states of the different software
entities in a system. These interfaces will be the mechanism by which you determine the state of their corresponding instances, as well as reboot the BMC and hosts, turn power
on and off to the chassis. The interfaces are designed in a way to support many to many
mappings of each interface.
There are three states to track and control on a BMC based server. The states below in () represents the actual parameter name as found in /xyz/openbmc_project/state/+/bmcX,/hostY,/chassisZ
where X,Y,Z are the instances (in most cases 0). For all three states, the software tracks a current state, and a requested transition.
1. BMC : The BMC has either started all required systemd services and reached it’s required target(Ready) or it’s on it’s way there (NotReady). Users can request a (Reboot) or (None).
2. Host : The host is either (Off), (Running), or it’s (Quiesced). Running simply implies that the processors are executing instructions. Users can request the host be in a (Off), (On), or (Reboot) state. More details on different Reboot options below. Quiesced means the host OS is in a quiesce state and the system should be checked for errors.
3. Chassis : The chassis is either (Off) or (On) This represents the state of power to the chassis. The Chassis being on is a pre-req to the Host being running. Users can request for the chassis to be (Off) or (On). A transition to one or the other is implied by the transition not matching the current state.
To enumerate all state items :
BMC
Check the status of the BMC:
curl -b jcar -k -X GET https://BMC_IP/xyz/openbmc_project/state/bmc<instance>
Change the BMC transition states:
curl -b cjar -k -X PUT -H "Content-Type:application/json" -d '{"data":
"xyz.openbmc_project.State.BMC.Transition.None" }'
https://BMC_IP/xyz/openbmc_project/state/bmc<instance>/attr/RequestedBMCTransition
Host
Check the status of the Host:
curl -b jcar -k -X GET https://BMC_IP/xyz/openbmc_project/state/host<instance>
Change the Host transition states:
curl -b cjar -k -X PUT -H "Content-Type:application/json" -d '{"data":
"xyz.openbmc_project.State.Host.Transition.On" }'
https://BMC_IP/xyz/openbmc_project/state/host<instance>/attr/RequestedHostTransition
Chassis
Check the status of the Chassis:
curl -b jcar -k -X GET https://BMC_IP/xyz/openbmc_project/state/chassis<instance>
Change the Chassis transition states:
curl -b cjar -k -X PUT -H "Content-Type:application/json" -d '{"data":
"xyz.openbmc_project.State.Chassis.Transition.On" }'
https://BMC_IP/xyz/openbmc_project/state/chassis<instance>/attr/RequestedPowerTransition
Control
The system inventory structure is under the /xyz/openbmc_project/control hierarchy. Implement to specify power transition behavior on a BMC reset. The implementation may choose to only enforce the policy on a power loss or on both a power loss and BMC reboot.
To enumerate all controls in the system:
curl -b jcar -k -X GET https://BMC_IP/xyz/openbmc_project/led/physical/enumerate
The policy to adopt after the BMC reset:
1. AlwaysOn: Perform a complete power on process.
2. AlwaysOff: Remain powered off.
3. Restore: Restore power to last requested state recorded before the BMC was reset.
Check the status of the Power Restore Policy:
curl -b cjar -k -X GET https://BMC_IP/xyz/openbmc_project/control/host0/power_restore_policy
Change Power Restore Policy states:
curl -b cjar -k -X PUT -H "Content-Type:application/json" -d '{"data":
"xyz.openbmc_project.Control.Power.RestorePolicy.Policy.AlwaysOn" }'
https://BMC_IP/xyz/openbmc_project/control/host0/power_restore_policy/attr/PowerRestorePolicy
Led
LEDs that are present in the system can be managed by: BMC/Hardware controller/Operating system and always recommended that external users use only the LED groups. Describing the Physical LED here just for documenting it and strictly NOT to be used outside of the firmware code.
The system inventory structure is under the /xyz/openbmc_project/led/physical hierarchy. Each event is a separate object under this structure, referenced by name.
To enumerate all physical LED in the system:
curl -b jcar -k -X GET https://BMC_IP/xyz/openbmc_project/led/physical/enumerate
There are four kinds of LEDs to track and control on a BMC based server as follows:
- Attention
- Onboard
- Platform
- System
Control states of physical LED can be in:
- On: LED is in solid on state
- Off: LED is in off state
- Blink: LED is blinking once per second.
Check the status of the Attention LED:
curl -b cjar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/led/physical/Attention
Change a Attention LED states:
curl -b cjar -k -X PUT -H "Content-Type: application/json" -d '{"data":
"xyz.openbmc_project.Led.Physical.Action.On" }'
https://BMC_IP/xyz/openbmc_project/led/physical/Attention/attr/state
Check the status of the Onboard LED:
curl -b cjar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/led/physical/Onboard
Change a Attention LED states:
curl -b cjar -k -X PUT -H "Content-Type: application/json" -d '{"data":
"xyz.openbmc_project.Led.Physical.Action.On" }'
https://BMC_IP/xyz/openbmc_project/led/physical/Onboard/attr/state
Check the status of the Platform LED:
curl -b cjar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/led/physical/Platform
Change a Attention LED states:
curl -b cjar -k -X PUT -H "Content-Type: application/json" -d '{"data":
"xyz.openbmc_project.Led.Physical.Action.On" }'
https://BMC_IP/xyz/openbmc_project/led/physical/Platform/attr/state
Check the status of the System LED:
curl -b cjar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/led/physical/System
Change a Attention LED states:
curl -b cjar -k -X PUT -H "Content-Type: application/json" -d '{"data":
"xyz.openbmc_project.Led.Physical.Action.On" }'
https://BMC_IP/xyz/openbmc_project/led/physical/System/attr/state
Logging
The event log structure is under the /xyz/openbmc_project/logging/entry hierarchy. Each event is a separate object under this structure, referenced by number.
BMC and host firmware on POWER-based servers can report logs to the BMC. Typically, these logs are reported in cases where host firmware cannot start the OS, or cannot reliably log to the OS.
The properties associated with an error log are as follows:
- Id: The error event entry id number.
- Timestamp: Commit timestamp of the error event entry in milliseconds since 1970.
- Severity: The severity of the error event entry.
- Message: The message description of the error event entry.
- AdditionalData: Additional information in the form of metadata field strings VAR=val
- Resolved: Error resolution status.
To list all reported events:
curl -b cjar -k -X GET https://BMC_IP/xyz/openbmc_project/logging/entry/
{
"data": [
"/xyz/openbmc_project/logging/entry/1",
"/xyz/openbmc_project/logging/entry/2"
],
"message": "200 OK",
"status": "ok"
}
To read a specific event log:
curl -b cjar -k -X GET https://BMC_IP/xyz/openbmc_project/logging/entry/1
{
"data":{
"AdditionalData":[
"_PID=1389"
],
"Id": 1,
"Message": "org.open_power.Host.Boot.Error.WatchdogTimedOut",
"Resolved": 0,
BARRELEYE-G2 BMC FW Specification
"Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
"Timestamp": 1510895953373,
"associations":[]
},
"message": "200 OK",
"status": "ok"
}
Implement to allow the deletion of individual entries, but cannot clear all logs.
To delete an event log (log 1 in his example), call the delete method on the event:
curl -b cjar -k -X DELETE https://BMC_IP/xyz/openbmc_project/logging/entry/1
Network
The system inventory structure is under the /xyz/openbmc_project/network hierarchy.
A Network Manager is a daemon which handles network management operations. It must implement the xyz/openbmc_project/network/SystemConfiguration .
When the network manager daemon comes up, it should create objects implementing physical link/virtual interfaces such as xyz/openbmc_project/network/+/EthernetInterface or /VLANInterface on the system.
IP address(v4 and v6) objects must be children objects of the physical/virtual interface object.
These are the interface associated to all network:
- SystemConfiguration: This describes the system specific parameters.
- EthernetInterface: This describes the interface specific parameters.
- IP: This describes the ip address specific parameters.
- IPProtocol: This describes the IP protocol type(IPv4/IPv6).
- VLANInterface: This describes the VLAN specific properties.
- Bond: This describes the interface bonding parameters.
To enumerate all networks in the system:
curl -b jcar -k -X GET https://BMC_IP/xyz/openbmc_project/network/enumerate
Interface objects can be physical as well as virtual.
If the object is physical interface, it can’t be deleted, but if it is a virtual interface object it can be deleted.
curl -b cjar -k -X Delete https://BMC_IP/xyz/openbmc_project/network/<interfacename>
There can be multiple ip address objects under an interface object. These objects can be deleted by the delete function or be added by the add function.
To delete IPv4 object:
curl -b cjar -k -X Delete https://BMC_IP/xyz/openbmc_project/network/eth0/ipv4/<id>
To add IPv4 object and ip address:
curl -b cjar -k -X POST -H "Content-Type:application/json" -d '{"data":
["xyz.openbmc_project.Network.IP.Protocol.IPv4", " Address", PrefixLength, " Gateway"] }'
https://BMC_IP/xyz/openbmc_project/network/eth0/action/IP
To delete IPv6 object:
curl -b cjar -k -X Delete https://BMC_IP/xyz/openbmc_project/network/eth0/ipv6/<id>
Have the system configuration related parameters at/xyz/openbmc_project/network/config
Software Versions
The system inventory structure is under the /xyz/openbmc_project/software hierarchy.
All version identifiers are implementation specific strings. No format should be assumed. Some software versions are a collection of images, each with their own version identifiers.
The xyz/openbmc_project/software/ExtendedVersion interface can be added to any software.
To enumerate all software in the system:
curl -b jcar -k -X GET https://BMC_IP/xyz/openbmc_project/software/enumerate
Check the status of the aggregation version:
curl -b cjar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/software/<id>
Time
The system inventory structure is under the /xyz/openbmc_project/time hierarchy.
To enumerate all time in the system:
curl -b jcar -k -X GET https://BMC_IP/xyz/openbmc_project/time/enumerate
Check the status of the time owner:
curl -b cjar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/time/owner
The possible owners of the time are as follows:
- BMC: BMC alone owns system time.
- Host: Host alone owns system time.
- Both: BMC and host own system time.
- Split: BMC and host maintain their own time.
Change the method of the owner:
curl -b cjar -k -X PUT -H "Content-Type:application/json" -d '{"data":
"xyz.openbmc_project.Time.Owner.Owners.BMC" }'
https://BMC_IP/xyz/openbmc_project/time/owner/attr/TimeOwner
Check the status of the time synchronization method:
curl -b cjar -k -X GET -H "Content-Type:application/json"
https://BMC_IP/xyz/openbmc_project/time/sync_method
The possible methods of time synchronization are as follows:
- NTP: Sync by using the Network Time Protocol.
- Manual: Sync time manually.
To change the time synchronization method:
curl -b cjar -k -X PUT -H "Content-Type:application/json" -d '{"data":
"xyz.openbmc_project.Time.Synchronization.Method.NTP" }'
https://BMC_IP/xyz/openbmc_project/time/sync_method/attr/TimeSyncMethod
openpower blog 