This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

File System Troubleshooting

1: Checking Disk Space
2: Adding a Disk and Partitioning It
3: Expanding Disk Partition
4: List Disk Partitions with 'fdisk'
5: File System Consistency Check Tool
6: Fixing File System Errors When fsck Fails

Photon OS includes commands to check and troubleshoot file systems.

1 - Checking Disk Space

One of the first simple steps to take while troubleshooting is to check how much disk space is available by running the df command:

df -h

2 - Adding a Disk and Partitioning It

If the df command shows that the file system is indeed nearing capacity, you can add a new disk on the fly and partition it to increase capacity.

Add a new disk.
For example, you can add a new disk to a virtual machine by using the VMware vSphere Client. After adding a new disk, check for the new disk by using fdisk. In the following example, the new disk is named /dev/sdb:

fdisk -l
Device        Start      End  Sectors Size Type
/dev/sda1      2048 16771071 16769024   8G Linux filesystem
/dev/sda2  16771072 16777182     6111   3M BIOS boot

Disk /dev/sdb: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Partition it with the parted wizard.
The command to partition the disk on Photon OS is as follows:

parted /dev/sdb

Use the parted wizard to create it as follows:

mklabel gpt
mkpart ext3 1 1024

Create a file system on the partition:

mkfs -t ext3 /dev/sdb1

Make a directory where you will mount the new file system:

mkdir /newdata

Open /etc/fstab and add the new file system with the options that you require:

	#system mnt-pt  type    options dump    fsck
	/dev/sda1       /       ext4    defaults,barrier,noatime,noacl,data=ord$
	/dev/cdrom      /mnt/cdrom      iso9660 ro,noauto       0       0
	/dev/sdb1       /newdata        ext3    defaults        0		0

Mount it using the following command:

mount /newdata

Verify the results:

df -h
	Filesystem      Size  Used Avail Use% Mounted on
	/dev/root       7.8G  4.4G  3.1G  59% /
	devtmpfs        172M     0  172M   0% /dev
	tmpfs           173M     0  173M   0% /dev/shm
	tmpfs           173M  664K  172M   1% /run
	tmpfs           173M     0  173M   0% /sys/fs/cgroup
	tmpfs           173M   36K  173M   1% /tmp
	tmpfs            35M     0   35M   0% /run/user/0
	/dev/sdb1       945M  1.3M  895M   1% /newdata

3 - Expanding Disk Partition

If you require more space, you can expand the last partition of your disk after resizing the disk.

The commands in this section assume sda as disk device.

After the disk is resized in the virtual machine, use the following command to enable the system to recognize the new disk ending boundary without rebooting:

echo 1 > /sys/class/block/sda/device/rescan

Install the parted package to resize the disk partition by running the following command to install it:

tdnf install parted.
parted /dev/sda

GNU Parted 3.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.

List all partitions available to fix the GPT and check the last partition number:

(parted) print

Warning: Not all of the space available to /dev/sda appears to be used, you can
fix the GPT to use all of the space (an extra 4194304 blocks) or continue with
the current setting? 
Fix/Ignore?

Press `f` to fix the GPT layout.

Model: VMware Virtual disk (scsi)
Disk /dev/sda: 34.4GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
1      1049kB  3146kB  2097kB                     bios_grub
2      3146kB  8590MB  8587MB  ext4

In this case we have the partition 2 as last, then we extend the partition to 100% of the remaining size:

(parted) resizepart 2 100%

Expand the filesystem to the new size:

resize2fs /dev/sda2
	resize2fs 1.42.13 (17-May-2015)
	Filesystem at /dev/sda2 is mounted on /; on-line resizing required
	old_desc_blocks = 1, new_desc_blocks = 2
	The filesystem on /dev/sda2 is now 8387835 (4k) blocks long.

The new space is already available in the system:

df -h
	Filesystem      Size  Used Avail Use% Mounted on
	/dev/root        32G  412M   30G   2% /
	devtmpfs       1001M     0 1001M   0% /dev
	tmpfs          1003M     0 1003M   0% /dev/shm
	tmpfs          1003M  252K 1003M   1% /run
	tmpfs          1003M     0 1003M   0% /sys/fs/cgroup
	tmpfs          1003M     0 1003M   0% /tmp
	tmpfs           201M     0  201M   0% /run/user/0

4 - List Disk Partitions with 'fdisk'

The fdisk command manipulates the disk partition table. You can, for example, use fdisk to list the disk partitions so that you can identify the root Linux file system.

The following example shows /dev/sda1 to be the root Linux partition:

fdisk -l
	Disk /dev/ram0: 4 MiB, 4194304 bytes, 8192 sectors
	Units: sectors of 1 * 512 = 512 bytes
	Sector size (logical/physical): 512 bytes / 4096 bytes
	I/O size (minimum/optimal): 4096 bytes / 4096 bytes

	...
	
	Disk /dev/sda: 8 GiB, 8589934592 bytes, 16777216 sectors
	Units: sectors of 1 * 512 = 512 bytes
	Sector size (logical/physical): 512 bytes / 512 bytes
	I/O size (minimum/optimal): 512 bytes / 512 bytes
	Disklabel type: gpt
	Disk identifier: 3CFA568B-2C89-4290-8B52-548732A3972D

	Device        Start      End  Sectors Size Type
	/dev/sda1      2048 16771071 16769024   8G Linux filesystem
	/dev/sda2  16771072 16777182     6111   3M BIOS boot

5 - File System Consistency Check Tool

You can manually check the file system by using the file system consistency check tool, fsck, after you unmount the file system.

The Photon OS file system includes btrfs and ext4. The default root file system is ext4, which you can see by looking at the file system configuration file, /etc/fstab:

cat /etc/fstab
	#system mnt-pt  type    options dump    fsck
	/dev/sda1       /       ext4    defaults,barrier,noatime,noacl,data=ordered     1       1
	/dev/cdrom      /mnt/cdrom      iso9660 ro,noauto       0       0

The 1 in the fifth column, under fsck, indicates that fsck checks the file system when the system boots.

You can also perform a read-only check without unmounting it:

fsck -nf /dev/sda1
	fsck from util-linux 2.27.1
	e2fsck 1.42.13 (17-May-2015)
	Warning!  /dev/sda1 is mounted.
	Warning: skipping journal recovery because doing a read-only filesystem check.
	Pass 1: Checking inodes, blocks, and sizes
	Pass 2: Checking directory structure
	Pass 3: Checking directory connectivity
	Pass 4: Checking reference counts
	Pass 5: Checking group summary information
	Free blocks count wrong (1439651, counted=1423942).
	Fix? no
	Free inodes count wrong (428404, counted=428397).
	Fix? no
	/dev/sda1: 95884/524288 files (0.3% non-contiguous), 656477/2096128 blocks

The inodes count might be wrong because the file system is mounted and in use.

To fix errors, you must first unmount the file system and then run fsck again:

umount /dev/sda1
umount: /: target is busy

You can find information about processes that use the device by using lsof or fuser.

lsof | grep ^jbd2/sd
	jbd2/sda1   99                root  cwd       DIR                8,1     4096          2 /
	jbd2/sda1   99                root  rtd       DIR                8,1     4096          2 /
	jbd2/sda1   99                root  txt   unknown                                        /proc/99/exe

The above example indicates that file system is in use.

6 - Fixing File System Errors When fsck Fails

Sometimes when fsck runs during startup, it encounters an error that prevents the system from fully booting until you fix the issue by running fsck manually. This error might occur when Photon OS is the operating system for a VM running an appliance.

If fsck fails when the computer boots and an error message says to run fsck manually, you can troubleshoot by restarting the VM, altering the GRUB edit menu to enter emergency mode before Photon OS fully boots, and running fsck.

Perform the following steps:

Take a snapshot of the virtual machine.
Restart the virtual machine running Photon OS.
When the Photon OS splash screen appears as it restarts, type the letter e quickly to go to the GNU GRUB edit menu.
Note: You must type e quickly as Photon OS reboots quickly. Also, in VMware vSphere or VMware Workstation Pro, you might have to give the console focus by clicking in its window before it will register input from the keyboard.
In the GNU GRUB edit menu, go to the end of the line that starts with linux, add a space, and then add the following code exactly as it appears below:
systemd.unit=emergency.target
Type F10.
In the bash shell, run one of the following commands to fix the file system errors, depending on whether sda1 or sda2 represents the root file system:
e2fsck -y /dev/sda1
or
e2fsck -y /dev/sda2
Restart the virtual machine. If the virtual machine fails to boot and finds any error then follow the steps below to recover.
Log in to the root shell: Command>shell root@vc701-w4#
To know about the error, execute the following command: journalctl -b 0 | grep -i “failed to start”
Below is output of above command:

Output for the journalctl -b 0 | grep -i “failed to start” command

Referring to the Failed to start the file system check on /dev/log_vg/log error in the screenshot above, if the partition type is logical volume, then the device mapper modules create a device-special file /dev/dm-X to which symbolic links with the original names points to /dev/mapper/log_vg-log or /dev/log_vg/log. Here log_vg is volume group and log is logical volume name.
Execute the lsblk command to confirm the device type. Below is the output of lsblk command. Here log_vg-log is associated with the device sde and type lvm. Also, note that it is not mounted.

lsblk command output

Execute the following command to fix the file system errors: e2fsck /dev/log_vg/log
Restart the virtual machine.