How to enable and use the serial terminal of a HP Microserver Gen9?

The ILO port on the HP server by default asks for an IP via DHCP. So you just need to plug it into a network that has a DHCP server running. It will announce itself with a host name like ILOCZ12345678 which should make it easier to find in your router's DHCP lease table, or in e.g. journalctl/syslog if you run a DHCP server (e.g. isc-dhcp-server) on a system like Ubuntu.

After you've found the IP (say 1.2.3.4), you can connect to ILO's GUI interface via https://1.2.3.4. But that's not what we're looking for, we want the serial console. So instead, use ILO's SSH interface: ssh [email protected]. The default password is on the little paper tag that comes with the Microserver.

Note this also works when the server is shut down as long as it has a power cable stuck into it (that is the point of ILO).

In there, type ? to see all options. Excerpt:

POWER    : Control server power.
UID      : Control Unit-ID light.
ONETIMEBOOT: Access One-Time Boot setting.
NMI      : Generate an NMI.
VM       : Virtual media commands.
LANGUAGE : Command to set or get default language
VSP      : Invoke virtual serial port.
TEXTCONS : Invoke Remote Text Console.
TESTTRAP : Sends a test SNMP trap to the configured alert destinations.

We choose VSP to get the serial port; TEXTCONS would probably also be nice, but you'd have to buy the Advanced ILO license to use it.

Now, once we do that, with a running system, it just hangs at

</>hpiLO-> VSP

Virtual Serial Port Active: COM2

Starting virtual serial port.
Press 'ESC (' to return to the CLI Session.

That's because by default, Ubuntu (I'm using 16.04 here) doesn't make a login prompt available via the serial port. We'll change that later. First we reboot to see that the BIOS config already works.

Seeing it boot via the serial port

In another ILO ssh session, reboot using:

</>hpiLO-> power reset

status=0
status_tag=COMMAND COMPLETED
Sun Feb  5 20:05:59 2017

Server resetting .......

After some waiting time, you'll notice that the session in which VSP is running has cleared its screen.

Then appears:

                            4096 MB Installed                                   

ProLiant System BIOS - J06 (11/02/2015)                                         
Copyright 1982, 2015 Hewlett-Packard Development Company, L.P.                  



1 Processor(s) detected, 2 total cores enabled, Hyperthreading is not supported
Proc 1:  Intel(R) Celeron(R) CPU G1610T @ 2.30GHz      
HP Power Profile Mode: Custom
Power Regulator Mode: Dynamic Power Savings

Redundant ROM Detected - This system contains a valid backup System ROM.
Inlet Ambient Temperature: 18C/64F

HP SmartMemory authenticated in all populated DIMM slots.

HP AHCI SATA Controller (v0.90)            
Copyright (c) 2011, Hewlett-Packard Development Company, L.P.
HP AHCI SATA Controller (v0.84)             WD30EFRX-68EUZN0
   Port3: S.M.A.R.T Capable Hard Disk - WDC WD30EFRX-68EUZN0
Copyright (c) 2011, Hewlett-Packard Development Company, L.P.
                                                                  <F9 = Setup>
iLO 4 Standard                           
iLO 4 v2.50 Sep 23 2016 192.168.178.106

Broadcom NetXtreme Ethernet Boot Agent
Copyright (C) 2000-2015 Broadcom Corporation
All rights reserved.
Press Ctrl-S to enter Configuration Menu

That's the POST screen. Wait a bit and see:

 Press "F9" key for ROM-Based Setup Utility
 Press "F10" key for Intelligent Provisioning
 Press "F11" key for Default Boot Override Options
 Press "F12" key for Network Boot
 For access via BIOS Serial Console
 Press "ESC+9" for ROM-Based Setup Utility
 Press "ESC+0" for Intelligent Provisioning
 Press "ESC+!" for Default Boot Override Options
 Press "ESC+@" for Network Boot

If you want to configure the BIOS, now is your time: This screen appears only for 1 second or so. The F* keys typically don't work, they need to be escaped: You literally have to press the ESC key and then the number/symbol as it says above. E.g. ESC and then 5 is for F5.

Now, without further settings, we get silence at the serial port, because as mentioned above, neither GRUB2 nor the Ubuntu login terminals output by default to the serial port in 16.04.

So we configure the server (now via SSH) to use the serial port.

Making GRUB2 use the serial port

I follwed the instructions at https://www.hiroom2.com/2016/06/06/ubuntu-16-04-grub2-and-linux-with-serial-console/ changing /etc/default/grub as described (- means comment out the line with #, + means add the line):

DEFAULT=0
-GRUB_HIDDEN_TIMEOUT=0
-GRUB_HIDDEN_TIMEOUT_QUIET=true
-GRUB_TIMEOUT=10
+GRUB_TIMEOUT=1
 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
-GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
-GRUB_CMDLINE_LINUX=""
+GRUB_CMDLINE_LINUX_DEFAULT=""
+GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0,115200"

-#GRUB_TERMINAL=console
+GRUB_TERMINAL="console serial"
+GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

The timeout of 10 seconds is set because the serial connection redraws the screen extremely slowly, and we want enough time to be able to cancel the timeout by pressing e.g. key-down in GRUB.

After changing the file as described, run sudo update-grub, and reboot.

You should now see in your VSP session the following after the POST screen:

error: serial port `com2' isn't found
error: terminal `serial' isn't found
error: terminal `serial' isn't found
error: no video mode activated

We ignore this (if you know how to get rid of this, please comment). After a shor time, this appears:

 GNU GRUB  version 2.02~beta2-36ubuntu3.6

 +----------------------------------------------------------------------------+
 |*Ubuntu                                                                     |
 | Advanced options for Ubuntu                                                |
 | Memory test (memtest86+)                                                   |
 | Memory test (memtest86+, serial console 115200)                            |
 |                                                                            |
 |                                                                            | 
 +----------------------------------------------------------------------------+

      Use the  and  keys to select which entry is highlighted.          
      Press enter to boot the selected OS, `e' to edit the commands       
      before booting or `c' for a command-line.                           

That's a normal GRUB screen, and you can move the * pointer with the up and down keys.

Making the Ubuntu login use the serial port

The following bits are inspired by http://0pointer.de/blog/projects/serial-console.html

sudo cp /lib/systemd/system/[email protected] /etc/systemd/system/[email protected]

Now edit /etc/systemd/system/[email protected]:

Change the line

ExecStart=-/sbin/agetty --keep-baud 115200,38400,9600 %I $TERM

to

ExecStart=-/sbin/agetty %I 115200 vt100-nav

We especially care to set the mode vt100-nav -- I found that the login doesn't appear without this change.

I'm not sure why we have to use ttyS1 here while in GRUB we configured ttyS0 -- but it doesn't work for me to use S0 here. If somebody knows why, please comment.

Start the tty via:

sudo systemctl start [email protected]

You should now see a login prompt in your VSP session. Make this happen automatically at boot using:

sudo systemctl enable [email protected]

And reboot. You should now see all the output in VSP, from POST to GRUB2 to the login prompt.

Use textcons from the ILO ssh interface. This may require the ILO Advanced license, but at the same time, there's a very low barrier to obtaining one.

Most organizations aren't interested in making the virtual serial port work because it's time consuming.

Google for an ILO license
or obtain a demo license from HP
or just pay $25 for an ILO key from Amazon

In RHEL7 "error: terminal `serial' isn't found" is due to the fact that Anaconda does not place a "serial" device driver in the RAM image. It is possible to rebuild the RAM disk with the missing driver but Grub seems to function sufficiently without it.