The Troubleshooting Guide covers the basics of troubleshooting systemd, packages, network interfaces, services such as SSH and Sendmail, the file system, and the Linux kernel. The guide also includes information about the tools that you can use for troubleshooting with examples, how to access the logs, and best practices.
This the multi-page printable view of this section. Click here to print.
- 1: Systemd and TDNF
- 2: The Root Account and the `sudo` and `su` Commands
- 3: Checking the Version and Build Number
- 4: General Best Practices
- 5: Photon OS Logs
- 6: Troubleshooting Progression
1 - Systemd and TDNF
By using systemd, Photon OS adopts a contemporary Linux standard to bootstrap the user space and concurrently start services, an architecture that differs from traditional Linux systems such as SUSE Linux Enterprise Server 11.
A traditional Linux system contains an initialization system called SysVinit. With SLES 11, for instance, SysVinit-style init programs control how the system starts up and shuts down. Init implements system runlevels. A SysVinit runlevel defines a state in which a process or service runs. In contrast to a SysVinit system, systemd defines no such runlevels. Instead, systemd uses a dependency tree of targets to determine which services to start when.
Because the systemd commands differ from those of an init.d-based Linux system, a section later in this guide illustrates how to troubleshoot by using systemctl commands instead of init.d-style commands.
Tdnf keeps the operating system as small as possible while preserving yum’s robust package-management capabilities. On Photon OS, tdnf is the default package manager for installing new packages. Since troubleshooting with tdnf differs from using yum, a later section of this guide describes how to solve problems with packages and repositories by using tdnf commands.
2 - The Root Account and the `sudo` and `su` Commands
The Troubleshooting Guide assumes that you are logged in to Photon OS with the root account and running commands as root. The
sudo program comes with the full version of Photon OS. On the minimal version, you must install
sudo with tdnf if you want to use it. As an alternative to installing
sudo on the minimal version, you can switch users as needed with the
su command to run commands that require root privileges.
3 - Checking the Version and Build Number
To check the version and build number of Photon OS, concatenate
cat /etc/photon-release VMware Photon Linux 1.0 PHOTON_BUILD_NUMBER=a6f0f63
The build number in the results maps to the commit number on the VMware Photon OS GitHub commits page.
4 - General Best Practices
When troubleshooting, it is recommended that you follow some general best practices:
Take a snapshot. Before you do anything to a virtual machine running Photon OS, take a snapshot of the VM so that you can restore it if need be.
Make a backup copy. Before you change a configuration file, make a copy of the original file. For example:
cp /etc/tdnf/tdnf.conf /etc/tdnf/tdnf.conf.orig
Collect logs. Save the log files associated with a Photon OS problem. Include not only the log files on the guest but also the
vmware.logfile on the host. The
vmware.logfile is in the host’s directory that contains the VM.
Know what is in your toolbox. View the man page for a tool before you use it so that you know what your options are. The options can help focus the command’s output on the problem you’re trying to solve.
Understand the system. The more you know about the operating system and how it works, the better you can troubleshoot.
5 - Photon OS Logs
On Photon OS, all the system logs except the installation logs and the cloud-init logs are written into the systemd journal. The
journalctl command queries the contents of the systemd journal.
The installation log files and the cloud-init log files reside in
/var/log. If Photon OS is running on a virtual machine in a VMware hypervisor, the log file for the VMware tools,
vmware-vmsvc.log, also resides in
Journalctl is a utility to query and display logs from journald and systemd’s logging service. Since journald stores log data in a binary format instead of a plain text format,
journalctl is the standard way of reading log messages processed by journald.
Journald is a service provided by systemd. To see the staus of the daemon, run the following commands:
# systemctl status systemd-journald ● systemd-journald.service - Journal Service Loaded: loaded (/lib/systemd/system/systemd-journald.service; static; vendor preset: enabled) Active: active (running) since Tue 2020-04-07 14:33:41 CST; 2 days ago Docs: man:systemd-journald.service(8) man:journald.conf(5) Main PID: 943 (systemd-journal) Status: "Processing requests..." Tasks: 1 (limit: 4915) Memory: 18.0M CGroup: /system.slice/systemd-journald.service └─943 /lib/systemd/systemd-journald Apr 07 14:33:41 photon-4a0e7f2307d4 systemd-journald: Journal started Apr 07 14:33:41 photon-4a0e7f2307d4 systemd-journald: Runtime journal (/run/log/journal/b8cebc61a6cb446a968ee1d4c5bbbbd5) is 8.0M, max 1.5G, 1.5G free. Apr 07 14:33:41 photon-4a0e7f2307d4 systemd-journald: Time spent on flushing to /var is 88.263ms for 1455 entries. Apr 07 14:33:41 photon-4a0e7f2307d4 systemd-journald: System journal (/var/log/journal/b8cebc61a6cb446a968ee1d4c5bbbbd5) is 40.0M, max 4.0G, 3.9G free. root@photon-4a0e7f2307d4 [ ~ ]#
The following command are related to
journalctl: This command displays all the logs after the system has booted up.
journalctlsplits the results into pages, similar to the
lesscommand in Linux. You can navigate using the arrow keys, the Page Up, Page Down keys or the Space bar. To quit navigation, press the q key.
journalctl -b: This command displays the logs for the current boot.
The following commands pull logs based on a time range:
journalctl --since "1 hour ago": This command displays the journal logs from the past 1 hour.
journalctl --since "2 days ago": This command displays the logs generated in the past 2 days.
journalctl --since "2020-03-25 00:00:00" --until "2020-04-09 00:00:00": This command displays the logs generated between the mentioned time frame.
To traverse for logs in the reverse order, run the following command:
journalctl -r: This command displays the logs in reverse order.
Note: If you add
-r at the end of a command, the logs are displayed in the reverse order. For example:
journalctl -u unit.service -r
To pull logs related to a particular daemon, run the following command:
journalctl -u unit.service: This command displays logs for a specific service. mention the name of the service instead of
unit. This command helps when a service is not behaving properly or when there are crash/core dumps.
To see Journal logs by their priority, run the following command:
journalctl -p "emerg".."crit: This command displays logs emerg to critical. For example: core dumps.
Journalctl can print log messages to the console as they are added, like the Linux
tail command. Add the
-f switch to follow a specific service or daemon.
journalctl -u unit.service -f
To list the boots of the system, run the following command:
You can maintain the journalctl logs manually, by running the following
journalctl --vacuum-time=2d: This command retains the logs from the last 2 days.
journalctl --vacuum-size=500M: This command helps retain logs with a maximum size of 500 MB.
You can configure Journald using the conf file located at /etc/systemd/journald.conf. Run the following command to configure the file:
# cat /etc/systemd/journald.conf
# This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by # the Free Software Foundation; either version 2.1 of the License, or # (at your option) any later version. # # Entries in this file show the compile time defaults. # You can change settings by editing this file. # Defaults can be restored by simply deleting this file. # # See journald.conf(5) for details. [Journal] #Storage=auto #Compress=yes #Seal=yes #SplitMode=uid #SyncIntervalSec=5m #RateLimitIntervalSec=30s #RateLimitBurst=10000 #SystemMaxUse= #SystemKeepFree= #SystemMaxFileSize= #SystemMaxFiles=100 #RuntimeMaxUse= #RuntimeKeepFree= #RuntimeMaxFileSize= #RuntimeMaxFiles=100 #MaxRetentionSec= #MaxFileSec=1month #ForwardToSyslog=no #ForwardToKMsg=no #ForwardToConsole=no #ForwardToWall=yes #TTYPath=/dev/console #MaxLevelStore=debug #MaxLevelSyslog=debug #MaxLevelKMsg=notice #MaxLevelConsole=info #MaxLevelWall=emerg #LineMax=48K root@photon-4a0e7f2307d4 [ ~ ]#
rotate is disabled in Photon. Once the changes are made to the conf file, for the changes to take effect you must restart the systemd-journald by running the
systemctl restart systemd-journald command.
Cloud-init is the industry standard multi-distribution method for cross-platform cloud instance initialisation.
If there are with the Cloud-init behaviour, we can debug them by looking at the logs. Run the following command to look at Cloud-init logs:
journalctl -u cloud-init
For better understanding/debugging, You can also look at logs from the following locations:
- /var/log/cloud-init.log : This log contains information from each stage of Cloud-init.
- /var/log/cloud-init-output.log : This log contains errors, warnings, etc..
Syslog is the general standard for logging system and program messages in the Linux environment.
Photon provides the following two packages to support syslog:
- syslog-ng : syslog-ng is syslog with some advanced next gen features. It supports TLS encryption, TCP for transport with other existing features. Configurations can be added to the
- rsyslog : The official RSYSLOG website defines the utility as “the rocket-fast system for log processing”. rsyslog supports some advanced features like relp, imfile, omfile, gnutls protocols. Configurations can be added to the
/etc/rsyslog.conffile. You can configure the required TLS certificates by editing the conf file.
Logs for RPMS on Photon
Logs for a particular RPM can be checked in the following ways:
- If the RPM provides a daemon, we can see the status of daemon by running
systemctlcommand and check logs using
journactl -u <service name>command.
- For additional logs, check if a conf file is provided by the RPM by running the
rpm -ql <rpm name> | grep confcommand and find the file path of the log file. You can also check the /var/log folder.
6 - Troubleshooting Progression
If you encounter a problem running an application or appliance on Photon OS and you suspect it involves the operating system, you can troubleshoot by proceeding as follows.
Check the services running on Photon OS:
Check your application log files for errors. For VMware applications, see Location of Log Files for VMware Products.)
Check the service controller or service monitor for your application or appliance.
Check the network interfaces and other aspects of the network service with
Check the operating system log files:
Next, run the following commands to view all services according to the order in which they were started:
Use the troubleshooting tool that you think is most likely to help with the issue at hand. For example, use
straceto identify the location of the failure.