Instructobit
If you have an application that relies on detecting, moving or copying files on a host, it can be quite dangerous to simply access these files without first confirming they are not being manipulated by another process or are currently in the process of being copied or written to. In such a situation, you'll first want to check that this is not the case, wait if necessary and the only proceed with accessing the necessary file.
What is the best way to measure changes to a file in Python?
Method 1:
The most accurate way to measure changes to a file is to directly compare the new version of the file to the old version. This can be done by storing a checksum of the initial file, waiting over a predetermined polling period, then comparing the old checksum to the new one. This will accurately measure any changes made to the file.
The issue with this method is that for larger files it can take quite some time and processing power to calculate a checksum of the file. If you are working with files that are larger than just a few megabytes, you could run in to issues such as high CPU usage, long wait times or running out of memory entirely. So for larger files you may want to consider another method.
Method 2:
The next best way to measure whether a file is in the process of being modified is to analyze its size over time, this can be be done by either reading the file's properties from the operating system, or to open the file and measure its size from there.
The issue with using the file properties from the operating system is that it may not be accurate depending on the operating system you are using. For example, when you copy a large file in Windows, the created file will show its final size throughout the copying process, not its current size. So to create a platform independent solution you won't be able to go this route.
A better solution is to open the file in read-only mode and calculate the file's size. The difficult part about this is to avoid reading the entire file into memory in order to measure its size, as this could make the process extremely slow for larger files, this can however be avoided using some file mechanic trickery.
Detecting changed made to large files using size comparison
To measure the size of a large file in Python and not suffer any serious performance issues is not as difficult as it may seem, all you really need to do is make use of the file object's read/write pointer. Simply open the file in Python and move the read/write pointer to the end of the file using the seek() method, then retrieve its position using the tell() method, this position is equal to the size of the file in bytes.
To set the pointer to the end of a file you can use seek(0, 2), this means set the pointer to position 0 relative to the end of the file (0 = absolute positioning, 1 = relative to the start of the file, 2 = relative to the end of the file).
def get_file_size(filepath):
# open the file in read only
with open(filepath, "r") as file:
# move pointer to the end of the file
file.seek(0, 2)
# retrieve the current position of the pointer
# this will be the file's size in bytes
size = file.tell()
return size
# if the function reaches this statement it means an error occurred within the above context handler
return False
Certain operating systems may not allow you to open the file (even just in reading mode) while it is being written to, so if an error occurs relating to that, this function will return False, if this is the case you can assume the file is/was being modified.
Now that you can accurately determine the size of a file at a given point, you'll need to create a piece of code that measures a file's size at two different intervals. If the size differs during the two intervals, you know there has been a modification made to the file.
To wait a specified amount of time you can make use of the sleep() method which can be imported from the time module.
For this example, we'll measure the initial size of the file, wait 2 seconds, then measure it again and compare the two sizes to see if it changed.
from time import sleep
# include get_file_size() function here
initial_size = get_file_size("testfile.txt")
sleep(2)
final_size = get_file_size("testfile.txt")
# if the initial size is equal to the final size, the file has most likely
# not changed, unless they are both False.
if initial_size == final_size and initial_size != False:
print("File has not changed!")
else:
print("File has been modified!")
In the above example, if you run this application and leave the target file untouched, the script will let you know that it has not been modified. If you try modify the file during the poll time, it will let you know that it has been modified.
Note:
if both file sizes come back as False, then we can assume the files were in the middle of being written to while attempting to access them, assuming that you have set up your permissions on the file correctly,
Detecting changes to smaller files using an MD5 checksum
Measuring a file's size gives us a pretty good idea of whether a file has been modified, but if a file is changed in such as way that it remains the same size after being modified, such as a single character being replaced, then comparing sizes will be ineffective
Creating an MD5 hash of the entire file before and after a predetermined polling period can tell whether a file has been modified with a high amount of accuracy. Although this method may not be suitable for large files (unless performance and time is not an issue) it is much safer to use this method whenever possible.
To generate an MD5 checksum of a file, we can make use of the
which is a default package in Python. The md5() function generates a hash using a given string, then we can retrieve that hash using the hexdigest() or digest() function, the only difference between those two functions is that hexdigest() returns the hash in the form of hexadecimals instead of bytes.
In this example we create a function that generates an MD5 hash from the contents of a given file.
import hashlib
def get_file_md5(filepath):
# open the file in read only
with open(filepath, "rb") as file:
file_contents = file.read()
# generate and return the md5 hash
return hashlib.md5(file_contents).hexdigest()
# if the function reaches this statement it means an error occurred within the above context handler
return False
If you do want to detect modifications to larger files in this manner, consider making use of the update() function to feed your file in chunks to the MD5 function, this is more memory efficient.
Just as we did in the first method, to determine whether a file has been modified, all we need to do is call our comparison function at two intervals, then compare the output.
from time import sleep
import hashlib
# include get_file_md5() function here
initial_md5 = get_file_md5("testfile.txt")
sleep(2)
final_md5 = get_file_md5("testfile.txt")
# if the initial hash is equal to the final hash, the file not changed
# unless they are both False.
if initial_md5 == final_md5 and initial_md5 != False:
print("File has not changed!")
else:
print("File has been modified!")
Detecting any changes made to an entire directory
To compare all files within a directory, all you need to do is create a function to iterate over the contents of a folder, creating a hash or measuring its size and storing that result in a dictionary, if the item you are iterating over is a sub-directory, we can recursively call the same function.
import os
import hashlib
# include get_file_md5() function here
def scan_dir(path):
# list all items in the directory
filelist = os.listdir(path)
hashes = []
for file in filelist:
filepath = path + file
# generate a hash if it's a file and add it to the output list
if os.path.isfile(filepath):
file_hash = get_file_md5(filepath)
# return False if any files are found to be in use
if file_hash == False:
return False
hashes.append({filepath: file_hash})
# generate hashed for all files in a sub-directory, add the output to the list
elif os.path.isdir(filepath):
dir_contents = scan_dir(filepath + "/")
# return False if any files are found to be in use
if dir_contents == False:
return False
hashes.append({filepath: dir_contents})
return hashes
To see if anything has changed in the directory, all you need to do is compare the output of this function over a given interval just as we did in the first two examples
import os
import hashlib
# list the current directory
initial_list = scan_dir("./")
sleep(2)
final_list = scan_dir("./")
# if the initial list is equal to the final list, no files in the directory
# have changed, unless they are both False.
if initial_list == final_list and initial_list != False:
print("Directory has not changed!")
else:
print("Directory has been modified!")
If you have any questions feel free to leave a comment below!
