Peter Evans's questions

I ran a yum update on a Centos 7 server that previously worked well, then tried to restart it with the command reboot.

The server is a google cloud compute instance.

It would not come up.

The output on the serial console is:

BdsDxe: loading Boot0003 "CentOS Linux" from HD(1,GPT,C8B5C89A-17ED-480E-B9F6-0E811C16FEE3,0x800,0x64000)/\EFI\cent
os\shimx64.efi                                                                                                     
BdsDxe: starting Boot0003 "CentOS Linux" from HD(1,GPT,C8B5C89A-17ED-480E-B9F6-0E811C16FEE3,0x800,0x64000)/\EFI\cen
tos\shimx64.efi                                                                                                    
                                                                                                                   
UEFI: Attempting to start image.                                                                                   
Description: CentOS Linux                                                                                          
FilePath: HD(1,GPT,C8B5C89A-17ED-480E-B9F6-0E811C16FEE3,0x800,0x64000)/\EFI\centos\shimx64.efi                     
OptionNumber: 3.                                                                                                   
                                                                                                                   
!!!! X64 Exception Type - 0D(#GP - General Protection)  CPU Apic ID - 00000000 !!!!                                
ExceptionData - 0000000000000000                                                                                   
RIP  - 00000000BF2E6D5C, CS  - 0000000000000038, RFLAGS - 0000000000010006                                         
RAX  - 00000000BFF67E50, RCX - 000000000000001F, RDX - 00000000BFF39F28                                            
RBX  - 00000000BF352FB0, RSP - 00000000BFF39E50, RBP - 00000000000023C0                                            
RSI  - 00000000BFF39F28, RDI - 0000000000000068                                                                    
R8   - 00000000BDB79018, R9  - 000000000000211E, R10 - 00000000BDC59010                                            
R11  - 00000000BFF39E9A, R12 - 00000000BE6E63E8, R13 - 00000000BF339018                                            
R14  - 00000000BEE21018, R15 - 0000000000000068                                                                    
DS   - 0000000000000030, ES  - 0000000000000030, FS  - 0000000000000030                                            
GS   - 0000000000000030, SS  - 0000000000000030                                                                    
CR0  - 0000000080010033, CR2 - 0000000000000000, CR3 - 00000000BF401000                                            
CR4  - 0000000000000668, CR8 - 0000000000000000                                                                    
DR0  - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000                                            
DR3  - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400                                            
GDTR - 00000000BF3EEA98 0000000000000047, LDTR - 0000000000000000                                                  
IDTR - 00000000BEE1F018 0000000000000FFF,   TR - 0000000000000000                                                  
FXSAVE_STATE - 00000000BFF39AB0                                                                                    
!!!! Find image based on IP(0xBF2E6D5C) /build/work/af60adde42b1d1ad5be2a01e4924bb905248/google3/blaze-out/k8-opt/g
enfiles/third_party/edk2/ovmf_x64_csm_debug_workspace_dir/ovmf_x64_csm_debug_edk2_files_dir/Build/OvmfX64/DEBUG_CLA
NG38/X64/OvmfPkg/8254TimerDxe/8254Timer/DEBUG/Timer.dll (ImageBase=00000000BF2E5000, EntryPoint=00000000BF2E6AB5) !
!!!                                                                                                                

I have no idea what I did wrong.

How can I avoid experiencing this in future? :)

Adding a new node-pool to an existing cluster is failing with no nodes registering.

The command used to add the node-pool is as follows (project name changed):

gcloud container --project my-project node-pools create hm-pool --cluster ds-cluster-west4 --zone europe-west4-c --node-version 1.16.9-gke.2 --machine-type n1-highmem-4 --image-type COS --disk-type pd-standard --disk-size 100 --metadata disable-legacy-endpoints=true --scopes logging-write,monitoring,pubsub,service-control,service-management,storage-full,taskqueue,trace --num-nodes 2 --enable-autoupgrade --enable-autorepair --max-surge-upgrade 1 --max-unavailable-upgrade 0

I am getting the following error messages

This will enable the autorepair feature for nodes. Please see https://cloud.google.com/kubernetes-engine/docs/node-auto-repair for more information on node autorepairs.
Creating node pool hm-pool...done.
ERROR: (gcloud.container.node-pools.create) Operation [<Operation
 clusterConditions: [<StatusCondition
 message: 'All cluster resources were brought up, but: only 0 nodes out of 2 have registered; cluster may be unhealthy.'>]
 detail: 'All cluster resources were brought up, but: only 0 nodes out of 2 have registered; cluster may be unhealthy.'
 endTime: '2020-06-04T15:17:05.810921209Z'
 name: 'operation-1591282299021-26295b28'
 nodepoolConditions: []
 operationType: OperationTypeValueValuesEnum(CREATE_NODE_POOL, 7)
 selfLink: 'https://container.googleapis.com/v1/projects/473462597806/zones/europe-west4-c/operations/operation-1591282299021-26295b28'
 startTime: '2020-06-04T14:51:39.021046271Z'
 status: StatusValueValuesEnum(DONE, 3)
 statusMessage: 'All cluster resources were brought up, but: only 0 nodes out of 2 have registered; cluster may be unhealthy.'
 targetLink: 'https://container.googleapis.com/v1/projects/473462597806/zones/europe-west4-c/clusters/ds-cluster-west4/nodePools/hm-pool'
 zone: 'europe-west4-c'>] finished with error: All cluster resources were brought up, but: only 0 nodes out of 2 have registered; cluster may be unhealthy.

On the console I see the message "Auto-repairing nodes in node pool hm-pool." and I see that hm-pool is updating. There are 0 nodes in the pool.

What am I doing wrong?

After setting up NFS on a server, rpcinfo -p servername from a client is returning the error message

rpcinfo: couldn't find a suitable transport

Connectivity is fine, checked with tcpdump. On the server hosts.allow contains the necessary entries. /etc/netconfig contains the following

# This config file is part of some non-installed packages (like nfs-utils)
#
# In case you install a tool that requires this file, please comment out the following lines:
#
# from https://highon.coffee/blog/security-harden-centos-7/
#
# udp6       tpi_clts      v     inet6    udp     -       -
# tcp6       tpi_cots_ord  v     inet6    tcp     -       -

udp        tpi_clts      v     inet     udp     -       -
tcp        tpi_cots_ord  v     inet     tcp     -       -
#udp6      tpi_clts      v     inet6    udp     -       -
#tcp6      tpi_cots_ord  v     inet6    tcp     -       -
rawip      tpi_raw       -     inet      -      -       -
local      tpi_cots_ord  -     loopback  -      -       -
unix       tpi_cots_ord  -     loopback  -      -       -

Any ideas?

I am trying to run a snakemake pipeline on a kubernetes cluster (GKE). The job is being initiated from a GCE VM. Sometimes it works, mostly it doesn't.

Steps I took were

gcloud container clusters get-credentials snakemake-k8s-demo
kubectl delete pod $(kubectl get pods | grep snakejob|colprint 1)
snakemake --kubernetes --container-image eu.gcr.io/scailyte-is/snakemake-gsdk --use-conda --default-remote-provider GS --default-remote-prefix xxxxxx-snakemake-test-1 --jobs 2

This first try worked very well.

I then deleted the files created by the snakemake pipeline and ran the identical job again without changing anything.

The job failed with the following error message:

HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /storage/v1/b/xxxxxxx-snakemake-test-1/o?projection=noAcl (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f3ee35159d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

According to the Google Cloud Status Dashboard, there are no problems with the Google Cloud Storage.

Subsequent attempts failed in the same way.

Any tips for a resolution gratefully accepted.

I recently carried out a recovery test on a Google Compute Engine VM with a Centos 7 OS. The VM has five partitioned disks managed by LVM. The server has been around for a while so there are about ten file systems distributed over the disks. The snapshots were all created within one or two seconds of each other in the early hours of the morning.

It would appear that the recovery went well, df output looks like what I would expect.

Was I just lucky?

I would normally expect to have trouble when restoring from ostensibly unsynchronized snapshots.

I'm wondering: do I need to use a "proper" logical-volume-aware backup system to ensure the consistency of restored file systems? Or failing that, should I ensure that each file systems is on its own single disk? Or am I worrying unnecessarily?