Photon OS General Troubleshooting

• 1: Photon Code
• 2: Package Management
• 3: Network Configuration
• 4: Cloud-init
• 5: Open-vm-tools/Vmtoolsd

1 - Photon Code

Photon is an RPM based Linux distribution similar to variants like CentOS and Fedora. With RPM based distributions granular updates as opposed to updating the whole OS image is possible.

##SPEC File The “Recipe” for creating an RPM package is a spec file. The Photon code base’s SPECS folder hast the following directory structure:

SourceRoot

       SPECS
            linux
                patch1
                patch2
                linux.spec

##To Check if a Package is Signed Run the following commands to check if the package is signed:

#check if a package is signed
rpm -q linux --qf '%{NAME}-%{VERSION}-%{RELEASE} %{SIGPGP:pgpsig} %{SIGGPG:pgpsig}\n'
linux-4.19.79-2.ph3 RSA/SHA1, Thu 31 Oct 2019 10:05:05 AM UTC, Key ID c0b5e0ab66fd4949 (none)
 
#or
rpm -qi linux | grep "Signature"
Signature   : RSA/SHA1, Thu 31 Oct 2019 10:05:05 AM UTC, Key ID c0b5e0ab66fd4949
 
#Last 8 chars of Key ID: 66fd4949
#See if it matches the version of any of the gpg keys installed.
rpm -qa | grep gpg-pubkey | xargs -n1 rpm -q --queryformat "%{NAME} %{VERSION} %{PACKAGER}\n"
gpg-pubkey 66fd4949 VMware, Inc. -- Linux Packaging Key -- linux-packages@vmware.com
gpg-pubkey 3e1ba8d5 Google Cloud Packages RPM Signing Key gc-team@google.com

##To Check if Your Image Has Vulnerabilities Use the security scanners to find security issues. Alternatively The tdnf updateinfo info command displays all the applicable security updates the host needs.

##To Check if a CVE is Fixed The Photon team fix the vulnerabilities and then publish the advisories to (https://github.com/vmware/photon/wiki/Security-Advisories).

##To Check if Security Updates are Available Use the tdnf updateinfo info, tdnf update --security or tdnf update ---sec-severity <level> commands to check if security updates are available. For example:

#check if there are any security updates
root@photon-9a8c05dd97e9 [ ~ ]# tdnf updateinfo
70 Security notice(s)
 
#check if there are security updates for libssh2. note this is relative to what is installed in local
root@photon-9a8c05dd97e9 [ ~ ]# tdnf updateinfo list libssh2
patch:PHSA-2020-3.0-0047 Security libssh2-1.9.0-2.ph3.x86_64.rpm
patch:PHSA-2019-3.0-0025 Security libssh2-1.9.0-1.ph3.x86_64.rpm
patch:PHSA-2019-3.0-0009 Security libssh2-1.8.2-1.ph3.x86_64.rpm
patch:PHSA-2019-3.0-0008 Security libssh2-1.8.0-2.ph3.x86_64.rpm
 
#show details of all the libssh2 updates
root@photon-9a8c05dd97e9 [ ~ ]# tdnf updateinfo info libssh2
       Name : libssh2-1.9.0-2.ph3.x86_64.rpm
  Update ID : patch:PHSA-2020-3.0-0047
       Type : Security
    Updated : Wed Jan 15 10:48:25 2020
Needs Reboot: 0
Description : Security fixes for {'CVE-2019-17498'}
       Name : libssh2-1.9.0-1.ph3.x86_64.rpm
  Update ID : patch:PHSA-2019-3.0-0025
       Type : Security
    Updated : Sat Aug 17 16:14:35 2019
Needs Reboot: 0
Description : Security fixes for {'CVE-2019-13115'}
       Name : libssh2-1.8.2-1.ph3.x86_64.rpm
  Update ID : patch:PHSA-2019-3.0-0009
       Type : Security
    Updated : Sat Apr 13 03:34:22 2019
Needs Reboot: 0
Description : Security fixes for {'CVE-2019-3859', 'CVE-2019-3862', 'CVE-2019-3861', 'CVE-2019-3857', 'CVE-2019-3858', 'CVE-2019-3863', 'CVE-2019-3860', 'CVE-2019-3856'}
       Name : libssh2-1.8.0-2.ph3.x86_64.rpm
  Update ID : patch:PHSA-2019-3.0-0008
       Type : Security
    Updated : Fri Mar 29 16:04:18 2019
Needs Reboot: 0
Description : Security fixes for {'CVE-2019-3855'}
 
 
 
#install all security updates >= score 9.0 (CVSS_v3.0_Severity)
root@photon-9a8c05dd97e9 [ ~ ]# tdnf update --sec-severity 9.0
Upgrading:
apache-tomcat                  noarch          8.5.50-1.ph3         photon-updates    9.00M 9440211
bash                           x86_64          4.4.18-2.ph3         photon-updates    3.16M 3315720
bzip2                          x86_64          1.0.8-1.ph3          photon-updates  124.99k 127990
bzip2-libs                     x86_64          1.0.8-1.ph3          photon-updates   74.31k 76096
file                           x86_64          5.34-2.ph3           photon-updates   43.02k 44056
file-libs                      x86_64          5.34-2.ph3           photon-updates    5.21M 5458536
git                            x86_64          2.23.1-2.ph3         photon-updates   24.34M 25519969
glib                           x86_64          2.58.0-4.ph3         photon-updates    3.11M 3265152
libseccomp                     x86_64          2.4.0-2.ph3          photon-updates  315.79k 323368
libssh2                        x86_64          1.9.0-2.ph3          photon-updates  238.41k 244136
linux-esx                      x86_64          4.19.97-2.ph3        photon-updates   12.68M 13299655
 
Total installed size:  58.28M 61114889

2 - Package Management

TDNF is the default package manager for Photon OS. The standard syntax for tdnf commands is the same as that for DNF and YUM. TDNF reads YUM repositories from /etc/yum.repos.d/.

To find the main configuration file and see its contents, run the following command:

cat /etc/tdnf/tdnf.conf
[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=true
repodir=/etc/yum.repos.d
cachedir=/var/cache/tdnf

Repositories have a .repo file extension, The following repositories are available in /etc/yum.repos.d/ :

ls /etc/yum.repos.d/
lightwave.repo
photon-extras.repo
photon-iso.repo
photon-updates.repo
photon.repo

Use the tdnf repolist command to list the repositories. Tdnf filters the results by their status enabled, disabled, and all. Running the tdnf repolist command without arguments displays the enabled repositories.

#tdnf repolist

repo id repo name status
photon-extras        VMware Photon Extras 3.0(x86_64) enabled
photon-debuginfo VMware Photon Linux debuginfo 3.0(x86_64)enabled
photon                    VMware Photon Linux 3.0(x86_64) enabled
photon-updates     VMware Photon Linux 3.0(x86_64) Updates enabled
root@photon-75829bfd01d0 [ ~ ]#

The following repositories are important for Photon:

• photon-updates : This repo contains RPM updates for CVE/version and updates/others fixes.
• photon-debuginfo : This repo contains information about RPMs with debug symbols.
• photon : This repo generally contains the RPM versions packaged with the released ISO.

To check the local cache data from the repository, run the following command:

# ls -l /var/cache/tdnf/photon
total 12
-r--r----- 1 root root 0 Apr 3 22:34 lastrefresh
drwxr-x--- 2 root root 4096 Apr 3 22:34 repodata
drwxr-x--- 4 root root 4096 Feb 4 14:31 rpms
drwxr-x--- 2 root root 4096 Apr 3 22:34 solvcache

##Usage The tdnf command can be used in the following ways:

#tdnf repolist --refresh : This command is used to refresh the repolist. Generally there is a cache of the repo data stored in the local VM.

#tdnf install <rpm name> : This command is used to install a RPM. This command installs the latest version of the RPM.

#tdnf install <pkg-name>-<verison>-<release>.<photon-release> : This command is used to install a particular RPM version. For example, run # tdnf install systemd-239-11.ph3.

#tdnf list systemd : This command is used to list the available RPM versions in the repository.

#tdnf makecache : This command updates the cached binary metadata for all known repositories.

tdnf clean all : This command cleans up temporary files, data, and metadata. It takes the argument all.

#tdnf list systemd

Refreshing metadata for: 'VMware Photon Linux 3.0(x86_64)'

systemd.x86_64                                                                       239-15.ph3                                            @System

systemd.x86_64                                                                       239-11.ph3                                     photon-updates

systemd.x86_64                                                                       239-12.ph3                                     photon-updates

systemd.x86_64                                                                       239-13.ph3                                     photon-updates

systemd.x86_64                                                                       239-14.ph3                                     photon-updates

systemd.x86_64                                                                       239-15.ph3                                     photon-updates

systemd.x86_64                                                                       239-17.ph3                                     photon-updates

systemd.x86_64                                                                       239-18.ph3                                     photon-updates

systemd.x86_64                                                                       239-19.ph3                                     photon-updates

systemd.x86_64                                                                       239-10.ph3                                             photon

systemd.x86_64                                                                       239-10.ph3                                             photon

root@photon-4a0e7f2307d4 [ /WS/photon-dev/photon ]#

Here, @System indicates that the particular RPM is installed in the VM.

To upgrade/downgrade RPMs run the following commands:

#tdnf upgrade <pkg-name>

#tdnf downgrade <pkg-name>

After upgrade/downgrade the dependent packages must be manually upgraded/downgraded as well. Use the #tdnf remove <pkg-name> command to remove packages and # tdnf clean all to clear cached packages, metadata, dbcache, plugins and expire-cache.

#RPM RPM is an open source package management system capable of building software from source into easily distributable packages. It is used for installing, updating and uninstalling packaged software. RPM can also be used to query detailed information about the packaged software and to check if a particular package is installed or not.

You can do the following operation using the RPM binaries:

• Install/Upgrade/Downgrade/Remove RPMs from a virtual machine.
• Check the version of the packages installed.
• Check the package contents.
• Check the dependencies of a package.
• Find the source package of a file.

To find the package that contains a particular binary, run rpm -q —whatprovides <binary/file path> command.

##Usage The rpm command can be used in the following ways:

• rpm -ivh <rpm file path> : This command installs the RPM in a virtual machine.
• rpm -Uvh <rpm file path> : This command is used to upgrade/downgrade the RPM.
• rpm -e <rpm file path> : This command uninstalls the RPM from the virtual machine.
• rpm -qp <rpm file path> --provides : This displays the libraries provided by the RPM.
• rpm -qp <rpm file path> --requires : This displays the binaries/libraries required to install a particular rpm.
• rpm -qa : This displays a list of all installed packages.
• rpm -ql <package file.rpm> : This command lists all files in the package file.

3 - Network Configuration

systemd-networkd is a system daemon that manages network configurations. It detects and configures network devices as they appear. It can also create virtual network devices.

##Configuration Examples All configurations are stored as foo.network in the /etc/systemd/network/, /lib/systemd/network/ and /run/systemd/network/ folder. Use the networkctl list command to list all the devices on the system.

After making changes to a configuration file, restart the systemd-networkd.service if version is < 245, for other version run the following commands:

root@photon [ /home/sus ]# networkctl reload
root@photon [ /home/sus ]# networkctl reconfigure eth0

Note:

• The options mentioned in the configuration files are case sensitive.
• Set DHCP=yes to accept IPv4 and IPv6 DHCP requests.
• Set DHCP=ipv4 to accept IPv4 DHCP requests.
• Set LinkLocalAddressing=no to disable IPv6. Please do not disable IPv6 via sysctl. When LinkLocalAddressing=no in the .network file, the kernel drops addresses starting with fe80, for example fe80::20c:29ff:fe4c:7eca. If IPv6LL address is not available networkd will not start IPv6 configurations.

To link network configurations using DHCPv4 (IPv6 disabled), run the following command:

/etc/systemd/network/20-eth0.network
[Match]
Name=eth0

[Network]
LinkLocalAddressing=no

DHCP=ipv4

To link network configurations using DHCPv6, run the following command:

/etc/systemd/network/20-eth0.network
[Match]
Name=eth0

[Network]
IPv6AcceptRA=yes

DHCP=ipv6

To link network configurations using a static IP address, run the following command:

/etc/systemd/network/20-wired.network
[Match]
Name=enp1s0

[Network]
Address=10.1.10.9/24
Gateway=10.1.10.1
DNS=10.1.10.1

Here Address= can be used more than once to configure multiple IPv4 or IPv6 addresses.

A .link file can be used to rename an interface. For example, set a predictable interface name for a Ethernet adapter based on its MAC address by running the following command:

/etc/systemd/network/10-test0.link
[Match]
MACAddress=12:34:56:78:90:ab

[Link]
Description=my custom name
Name=test123

##Configuration Files Configuration files are located in /usr/lib/systemd/network/ folder, the volatile runtime network directory in /run/systemd/network/ folder and the local administration network directory in /etc/systemd/network/ folder. Configuration files in /etc/systemd/network/ folder have the highest priority.

There are three types of configuration files and they use a format similar to systemd unit files.

• .network : These files apply a network configuration to a matching device.

• .netdev : These files are used to create a virtual network device for a matching environment.

• .link : When a network device appears, udev looks for the first matching .link file. These link files follow the following rules:

• Only if all conditions in the [Match] section are matched, the profile will be activated.

• An empty [Match] section means the profile can apply to any case (can be compared to the * wild card)

• All configuration files are collectively sorted and processed in lexical order, regardless of the directory it resides in.

• Files with identical names replace each other.

##Dupliate Matches If we have multiple configuration files matching an interface, the first (in lexical order) network file matching a given device is applied. All other files are ignored even if they match. The following is an example of matching configuration files:

builder@localhost [ ~ ]$ cat /etc/systemd/network/10-eth0.network
[Match]
Name=eth0
[Network]
DHCP=yes
  
builder@localhost [ ~ ]$ cat /etc/systemd/network/99-dhcp-en.network
[Match]
Name=e*
 
[Network]
DHCP=yes
IPv6AcceptRA=no

##Network Files These files are used to set network configuration variables for servers and containers. .network files have the following sections:

###[Match]

###[Link]

• MACAddress= : Used to spoof MAC address.
• MTUBytes= : Setting a larger MTU value (For example: when using jumbo frames) can significantly speed up your network transfers.
• Multicast : Enables the use of multicast on interface(s).

###[Network]

###[Address] Address= option is mandatory unless DHCP is used.

###[Route]

• Gateway= option is mandatory unless DHCP is used.
• Destination= option defines the destination prefix of the route, possibly followed by a slash and the prefix length. If Destination is not present in [Route] section it is treated as a default route. Note: You can add the Address= and Gateway= keys in the [Network] section as a short-hand, if the [Address] section contains only an Address key and [Route] section contains only a Gateway key.

###DHCP

###[DHCPServer] The following is an example of a DHCP server configuration which works well with hostapd to create a wireless hotspot. IPMasquerade adds the firewall rules for NAT and IPForward enables packet forwarding.

/etc/systemd/network/wlan0.network
[Match]
Name=wlan0

[Network]
Address=10.1.1.1/24
DHCPServer=true
IPMasquerade=true
IPForward=true

[DHCPServer]
PoolOffset=100
PoolSize=20
EmitDNS=yes
DNS=9.9.9.9

##Netdev Files These files create virtual network devices. They have the following two sections:

###[Match]

• Host= : The host name.
• Virtualization= : Checks if it is running in a virtual environment.

###[NetDev]

• Name= : The interface’s name. This is a mandatory field.
• Kind= : For example: bridge, bond, vlan, veth, sit, etc. This is a mandatory field.

##Link Files These files are an alternative to custom udev rules and will be applied by udev as the device appears. They have the following two sections:

###[Match]

• MACAddress= : The MAC address.
• Host= : The host name.
• Virtualization= : Checks if it is running in a virtual environment.
• Type= : the device type. For example: vlan.

###[Link]

• MACAddressPolicy= : Persistent or random addresses.
• MACAddress= : The MAC address. Note: The system /usr/lib/systemd/network/99-default.link file is sufficient for most cases.

##Debugging Systemd-networkd The log can be generated by creating a drop-in config. For example:

# /etc/systemd/system/systemd-networkd.service.d/override.conf
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

4 - Cloud-init

Cloud-init is mixture of Python and Shell scripts that initialize cloud instances of Linux machines. Cloud-init performs boot time configuration of a system. We can configure users, hostname, host network, write files to disk, manage packages, run custom scripts and so on.

##DataSources Datasource is the source of configuration data for cloud-init that is typically given by a user (For example: userdata) or obtained from the cloud that created the configuration drive (For example: metadata). Userdata includes files, YAML configuration files and shell scripts. Metadata includes server name, instance id, display name and other cloud specific details.

Currently there are two datasources used in Photon OS, it’s usage is described in the following sections:

• DataSourceOVF - Used for GuestOS customization in vSphere.
• VMwareGuestInfo - Used to read meta, user, and vendor data from VMware vSphere’s GuestInfo interface and initialize the system.

###DataSourceOVF The OVF (Open Virtualization Format) Datasource provides a datasource for reading data from an OVF transport ISO. The vmtoolsd service extracts the customization spec cab file from the OVF and calls either cloud-init or the GuestOS customization scripts. The disable_vmware_customization flag in /etc/cloud/cloud.cfg file determines if GOSC scripts or cloud-init is used.

• disable_vmware_customization: false : Cloud-init is used for Guest OS customization
• disable_vmware_customization: true : GuestOS customization scripts is used for Guest OS customization

Note: The default value for disable_vmware_customization is set to true in the /etc/cloud/cloud.cfg file

###VMwareGuestInfo VMwareGuestInfo data source is configured by setting guestinfo properties on a VM. This can be set by performing one of the following:

• Using the vmware-rpctool provided by open-vmtools.
• Modifying the vmx file to set the guestinfo properties.

##Debugging Cloud-init Failures Cloud-init has four services which are started in the following sequence:

1. cloud-init-local - This service locates local data sources and applies networking configurations provided n the metadata (If there is no metadata it applies Fallback). Use $ systemctl status cloud-init-local command to check its status.
2. cloud-init - This service processes any user-data that is found and runs the cloud_init_modules in /etc/cloud/cloud.cfg. Use $ systemctl status cloud-init command to check its status.
3. cloud-config - This service runs the cloud_config_modules in /etc/cloud/cloud.cfg file. Use $ systemctl status cloud-config command to check its status.
4. cloud-final - This service runs any script that a user is accustomed to running after logging into a system (For example: package installations, configs, user-scripts) and runs cloud_final_modules in /etc/cloud/cloud.cfg file. Use $ systemctl status cloud-final command to check its status.

Cloud-init logs are available in the /var/log/cloud-init.log file. Logs for GuestOS customization using DataSourceOVF are available in the /var/log/vmware-imc/toolsDeployPkg.log and /var/log/cloud-init.log files.

To analyze the cloud-init boot time performance, run the following commands:

• $ cloud-init analyze blame - The blame command prints in descending order, the units that took the longest to run. This output is useful for observe where cloud-init is spending its time during execution.
• $ cloud-init analyze show - The show command prints a list of units, the time they started and how long they took to complete. It also prints a summary of total time per boot.
• $ cloud-init analyze dump - The dump command dumps the cloud-init logs for the analyze modules and displays a list of dictionaries that can be consumed for other reporting needs.
• $ cloud-init status - To know the overall status of clouf-init.

Cloud-init doesn’t configure the network if /etc/cloud/cloud.cfg.d/99-disable-networking-config.cfg file is present and has the following content:

• network:Item
• config: disabled

Take a backup of /etc/cloud/cloud.cfg.d/99-disable-networking-config.cfg file and remove it from it’s location. Reconfigure the machine using metadata, userdata and vendordata. Once the configurations are done copy the backup file to the same location. Cloud-init will push it’s fallback configuration when service is restarted or rebooted and there is no local datasource to configure. To avoid this /etc/cloud/cloud.cfg.d/99-disable-networking-config.cfg file is required.

##Run Cloud-init Manually To run cloud-init manually, run the following commands:

/usr/bin/cloud-init -d init  (-d for debug)
/usr/bin/cloud-init -d modules (run all modules)
/usr/bin/cloud-init --file <config-yaml-file-path> init (if you want to run cloud-init with a configuration yaml file)

When cloud-init is running, to force it to run with all configs engaged run the following command:

rm -rf /var/lib/cloud/*

For more information about cloud-init, see https://cloudinit.readthedocs.io/en/latest/index.htmlhttps://cloudinit.readthedocs.io/en/latest/index.html

For more information about cloud-init CLI, see https://cloudinit.readthedocs.io/en/latest/topics/cli.htmlhttps://cloudinit.readthedocs.io/en/latest/topics/cli.html

Note:Include the cloud-init log tarball and the vmtoolsd logs when you raise an issue.

1. Collect cloud-init log tarball by running the cloud-init collect-logs command.
2. Collect the vmtoolsd logs from /var/log/vmware-imc/toolsDeployPkg.log file.
3. Attach the collected logs to the issue ticket.

5 - Open-vm-tools/Vmtoolsd

Vmtoolsd is a systemd service, using which we can set guestinfo properties metadata, userdata and vendordata etc., which in turn are consumed by cloud-init. VMwareGuestInfo Datasource uses this guestinfo properties and applies them to the system.

vmware-rpctool is a utility provided by open-vm-tools to set metadata, userdata and vendordata. vmware-rpctool provides info.set and info.get options to set and get the guestinfo properties respectively.

##Debugging To check the status of the vmtoolsd service (vmtoolsd is dependant on vgauthd), run the following commands:

$ systemctl status vmtoolsd vgauthd

$ journalctl -u vmtoolsd

$ journalctl -u vgauthd

To set and get metadata, userdata and vendordata, run the following commands:

$ /usr/bin/vmware-rpctool 'info-get guestinfo.metadata'

$ /usr/bin/vmware-rpctool 'info-get guestinfo.userdata'

$ /usr/bin/vmware-rpctool 'info-get guestinfo.vendordata'

A YAML file can be used as input to the rpctool using following commands:

vmware-rpctool "info-set guestinfo.userdata.encoding base64"
vmware-rpctool "info-set guestinfo.metadata.encoding base64"

vmware-rpctool "info-set guestinfo.metadata ${metadata file contents}"

vmware-rpctool "info-set guestinfo.userdata ${userdata file contents}"

Note:Include the cloud-init log tarball and the vmtoolsd logs when you raise an issue.

1. Collect cloud-init log tarball by running the cloud-init collect-logs command.
2. Collect the vmtoolsd logs from /var/log/vmware-imc/toolsDeployPkg.log file.
3. Attach the logs collected to the issue ticket.