How To Fix Incorrect Copying Progress In Ubuntu/Linux?
If you are a Linux user for a while, you might have noticed a strange behavior in Ubuntu. I have been seeing this behavior since Ubuntu 16.04. I have tried different Linux distributions and found the same behavior on every single one. Every time I copy files to external drives, the copying progress shows completely wrong progress.
When Does Incorrect Copying Progress Happen?
At the starting point, when I copy a big file (1GB-4GB), the Nautilus file manager shows insane transfer speed. The progress bar reaches 90%-100% in a matter of seconds. After completion, when I try to unmount the drive, it freezes for several minutes. The system was still writing the data on the drive.
If I unplug the drive at this moment, the file will be corrupted. I have to wait until the writing process is completed. And how much time does it take? I don’t know. This is annoying.
Why Does Incorrect Copying Progress Happen?
There are two types of computer memory: Volatile and Non-volatile. Volatile memory like RAM is faster but limited in size and requires power to retain data.
Non-volatile memories like HDDs and SSDs can retain data even after power loss. But these are slower than RAM.
RAM is fast enough to access any specific address called by a process at any time. But for non-volatile memories, such an algorithm increases latency and makes the system unresponsive. So the Linux system uses the asynchronous approach as much as possible. Linux puts disk writes into the cache in RAM, and asynchronously flushes them to disk over time. This algorithm has a positive effect on speeding up disk I/O but has some caveats.
When the cache is empty, and a process attempts to do a write operation, it immediately receives feedback. But, when the flush command is called out, there is a significant delay before the data is transferred from RAM to HDD.
Ubuntu allocates 20% of RAM by default for file caches, so when I have more RAM, the cache can hold the entire file.
That’s why you see insane transfer speed. The entire file is in the cache and later on flushed to the USB drive slowly resulting in longer unmounting time.
How To Fix This Incorrect Copying Progress?
The problem here is directly connected to dirty memory and how the Linux kernel deals with it. Dirty memory is memory written in the cache but not saved on the main disk. When the cache size is big, the file cache in memory holds big dirty data and slowly flushes buffered writes to the main disk. If we reduce the file cache size, then the system will write those dirty data into the disk more frequently.
These parameters can be tuned to change the behavior of the Linux Kernel with the file cache.
| dirty_ratio | Maximum percentage of dirty memory |
| dirty_bytes | Same as dirty_ratio but specified in bytes |
| dirty_background_ratio | Percentage of dirty memory at which background writeback will start |
| dirty_background_bytes | Same as dirty_background_ratio but specified in bytes |
Above I have mentioned that Ubuntu uses 20% of RAM for file cache. That means if I have 32GB of RAM and the dirty_ratio is 20 then the file cache size would be over 6GB. This eventually shows wrong progress while copying and takes longer unmounting time.
We can reduce the file cache size to 48Mb and ask the Linux Kernel to start writing to the drive when the cache has more than 16Mb of dirty data using the command:
sudo bash -c 'echo $((16*1024*1024)) > /proc/sys/vm/dirty_background_bytes' sudo bash -c 'echo $((48*1024*1024)) > /proc/sys/vm/dirty_bytes'
After that, the copying progress will show the correct speed, and the unmounting process will take a few seconds.
You can make this setting persistent by adding the following lines to the /etc/sysctl.conf file:
vm.dirty_background_bytes = 16777216 vm.dirty_bytes = 50331648
Open the terminal and open the /etc/sysctl.conf file using the following command.
sudo gedit /etc/sysctl.conf
You can use any text editor you want. Now go to the bottom of the file and add these two lines.
vm.dirty_background_bytes = 16777216 vm.dirty_bytes = 50331648
Save the file and reboot.
Theoretically, this setting can reduce system performance. But in my daily tasks, I do not face any major performance hit. So I am happy with this solution.
Conclusion
This fix can improve your overall user experience. If you face any difficulties setting everything up, or if you have any questions or suggestions, please comment down below. If you have a better solution to this weird behavior, let us know in the comment section.
I will be glad to hear from you. I will update the post if I find a better solution than this one. until then, stay safe. Have a nice day.
