Introduction

• 1: Systemd and TDNF
• 2: The Root Account and the `sudo` and `su` Commands
• 3: Checking the Version and Build Number
• 4: General Best Practices
• 5: Photon OS Logs
• 6: Troubleshooting Progression

1 - Systemd and TDNF

By using systemd, Photon OS adopts a contemporary Linux standard to bootstrap the user space and concurrently start services, an architecture that differs from traditional Linux systems such as SUSE Linux Enterprise Server 11.

A traditional Linux system contains an initialization system called SysVinit. With SLES 11, for instance, SysVinit-style init programs control how the system starts up and shuts down. Init implements system runlevels. A SysVinit runlevel defines a state in which a process or service runs. In contrast to a SysVinit system, systemd defines no such runlevels. Instead, systemd uses a dependency tree of targets to determine which services to start when.

Because the systemd commands differ from those of an init.d-based Linux system, a section later in this guide illustrates how to troubleshoot by using systemctl commands instead of init.d-style commands.

Tdnf keeps the operating system as small as possible while preserving yum’s robust package-management capabilities. On Photon OS, tdnf is the default package manager for installing new packages. Since troubleshooting with tdnf differs from using yum, a later section of this guide describes how to solve problems with packages and repositories by using tdnf commands.

2 - The Root Account and the `sudo` and `su` Commands

The Troubleshooting Guide assumes that you are logged in to Photon OS with the root account and running commands as root. The sudo program comes with the full version of Photon OS. On the minimal version, you must install sudo with tdnf if you want to use it. As an alternative to installing sudo on the minimal version, you can switch users as needed with the su command to run commands that require root privileges.

3 - Checking the Version and Build Number

To check the version and build number of Photon OS, concatenate /etc/photon-release.

Example:

cat /etc/photon-release
VMware Photon Linux 1.0
PHOTON_BUILD_NUMBER=a6f0f63

The build number in the results maps to the commit number on the VMware Photon OS GitHub commits page.

4 - General Best Practices

When troubleshooting, it is recommended that you follow some general best practices:

• Take a snapshot. Before you do anything to a virtual machine running Photon OS, take a snapshot of the VM so that you can restore it if need be.

• Make a backup copy. Before you change a configuration file, make a copy of the original file. For example: cp /etc/tdnf/tdnf.conf /etc/tdnf/tdnf.conf.orig

• Collect logs. Save the log files associated with a Photon OS problem. Include not only the log files on the guest but also the vmware.log file on the host. The vmware.log file is in the host’s directory that contains the VM.

• Know what is in your toolbox. View the man page for a tool before you use it so that you know what your options are. The options can help focus the command’s output on the problem you’re trying to solve.

• Understand the system. The more you know about the operating system and how it works, the better you can troubleshoot.

5 - Photon OS Logs

On Photon OS, all the system logs except the installation logs and the cloud-init logs are written into the systemd journal. The journalctl command queries the contents of the systemd journal.

The installation log files and the cloud-init log files reside in /var/log. If Photon OS is running on a virtual machine in a VMware hypervisor, the log file for the VMware tools, vmware-vmsvc.log, also resides in /var/log.

##Journalctl Journalctl is a utility to query and display logs from journald and systemd’s logging service. Since journald stores log data in a binary format instead of a plain text format, journalctl is the standard way of reading log messages processed by journald.

Journald is a service provided by systemd. To see the staus of the daemon, run the following commands:

# systemctl status systemd-journald
● systemd-journald.service - Journal Service
Loaded: loaded (/lib/systemd/system/systemd-journald.service; static; vendor preset: enabled)
Active: active (running) since Tue 2020-04-07 14:33:41 CST; 2 days ago
Docs: man:systemd-journald.service(8)
man:journald.conf(5)
Main PID: 943 (systemd-journal)
Status: "Processing requests..."
Tasks: 1 (limit: 4915)
Memory: 18.0M
CGroup: /system.slice/systemd-journald.service
└─943 /lib/systemd/systemd-journald



Apr 07 14:33:41 photon-4a0e7f2307d4 systemd-journald[943]: Journal started
Apr 07 14:33:41 photon-4a0e7f2307d4 systemd-journald[943]: Runtime journal (/run/log/journal/b8cebc61a6cb446a968ee1d4c5bbbbd5) is 8.0M, max 1.5G, 1.5G free.
Apr 07 14:33:41 photon-4a0e7f2307d4 systemd-journald[943]: Time spent on flushing to /var is 88.263ms for 1455 entries.
Apr 07 14:33:41 photon-4a0e7f2307d4 systemd-journald[943]: System journal (/var/log/journal/b8cebc61a6cb446a968ee1d4c5bbbbd5) is 40.0M, max 4.0G, 3.9G free.
root@photon-4a0e7f2307d4 [ ~ ]#

The following command are related to journalctl:

• journalctl : This command displays all the logs after the system has booted up. journalctl splits the results into pages, similar to the less command in Linux. You can navigate using the arrow keys, the Page Up, Page Down keys or the Space bar. To quit navigation, press the q key.
• journalctl -b : This command displays the logs for the current boot.

The following commands pull logs based on a time range:

• journalctl --since "1 hour ago" : This command displays the journal logs from the past 1 hour.
• journalctl --since "2 days ago" : This command displays the logs generated in the past 2 days.
• journalctl --since "2020-03-25 00:00:00" --until "2020-04-09 00:00:00" : This command displays the logs generated between the mentioned time frame.

To traverse for logs in the reverse order, run the following command:

• journalctl -r : This command displays the logs in reverse order.

Note: If you add -r at the end of a command, the logs are displayed in the reverse order. For example: journalctl -u unit.service -r

To pull logs related to a particular daemon, run the following command:

• journalctl -u unit.service : This command displays logs for a specific service. mention the name of the service instead of unit. This command helps when a service is not behaving properly or when there are crash/core dumps.

To see Journal logs by their priority, run the following command:

• journalctl -p "emerg".."crit : This command displays logs emerg to critical. For example: core dumps.

Journalctl can print log messages to the console as they are added, like the Linux tail command. Add the -f switch to follow a specific service or daemon.

journalctl -u unit.service -f

To list the boots of the system, run the following command:

journalctl --list-boots

You can maintain the journalctl logs manually, by running the following vacuum commands:

• journalctl --vacuum-time=2d : This command retains the logs from the last 2 days.
• journalctl --vacuum-size=500M : This command helps retain logs with a maximum size of 500 MB.

You can configure Journald using the conf file located at /etc/systemd/journald.conf. Run the following command to configure the file:

# cat /etc/systemd/journald.conf

# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=10000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=no
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K
root@photon-4a0e7f2307d4 [ ~ ]#

By default rotate is disabled in Photon. Once the changes are made to the conf file, for the changes to take effect you must restart the systemd-journald by running the systemctl restart systemd-journald command.

Cloud-init Logs

Cloud-init is the industry standard multi-distribution method for cross-platform cloud instance initialisation.

If there are with the Cloud-init behaviour, we can debug them by looking at the logs. Run the following command to look at Cloud-init logs:

journalctl -u cloud-init

For better understanding/debugging, You can also look at logs from the following locations:

• /var/log/cloud-init.log : This log contains information from each stage of Cloud-init.
• /var/log/cloud-init-output.log : This log contains errors, warnings, etc..

Syslog

Syslog is the general standard for logging system and program messages in the Linux environment.

Photon provides the following two packages to support syslog:

• syslog-ng : syslog-ng is syslog with some advanced next gen features. It supports TLS encryption, TCP for transport with other existing features. Configurations can be added to the /etc/syslog-ng/syslog-ng.conf file.
• rsyslog : The official RSYSLOG website defines the utility as “the rocket-fast system for log processing”. rsyslog supports some advanced features like relp, imfile, omfile, gnutls protocols. Configurations can be added to the /etc/rsyslog.conf file. You can configure the required TLS certificates by editing the conf file.

Logs for RPMS on Photon

Logs for a particular RPM can be checked in the following ways:

• If the RPM provides a daemon, we can see the status of daemon by running systemctl command and check logs using journactl -u <service name> command.
• For additional logs, check if a conf file is provided by the RPM by running the rpm -ql <rpm name> | grep conf command and find the file path of the log file. You can also check the /var/log folder.

6 - Troubleshooting Progression

If you encounter a problem running an application or appliance on Photon OS and you suspect it involves the operating system, you can troubleshoot by proceeding as follows.

1. Check the services running on Photon OS:

   systemctl status

2. Check your application log files for errors. For VMware applications, see Location of Log Files for VMware Products.)

3. Check the service controller or service monitor for your application or appliance.

4. Check the network interfaces and other aspects of the network service with systemd-network commands.

5. Check the operating system log files:

   journalctl

   Next, run the following commands to view all services according to the order in which they were started:

   systemd-analyze critical-chain

6. Use the troubleshooting tool that you think is most likely to help with the issue at hand. For example, use strace to identify the location of the failure.