1Usage 2===== 3 4Test Scenarios 5-------------- 6There are several approaches to implementing tests using ``pyfakefs``. 7 8Patch using fake_filesystem_unittest 9~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 10If you are using the Python ``unittest`` package, the easiest approach is to 11use test classes derived from ``fake_filesystem_unittest.TestCase``. 12 13If you call ``setUpPyfakefs()`` in your ``setUp()``, ``pyfakefs`` will 14automatically find all real file functions and modules, and stub these out 15with the fake file system functions and modules: 16 17.. code:: python 18 19 from pyfakefs.fake_filesystem_unittest import TestCase 20 21 class ExampleTestCase(TestCase): 22 def setUp(self): 23 self.setUpPyfakefs() 24 25 def test_create_file(self): 26 file_path = '/test/file.txt' 27 self.assertFalse(os.path.exists(file_path)) 28 self.fs.create_file(file_path) 29 self.assertTrue(os.path.exists(file_path)) 30 31The usage is explained in more detail in :ref:`auto_patch` and 32demonstrated in the files ``example.py`` and ``example_test.py``. 33 34Patch using the PyTest plugin 35~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 36If you use `PyTest <https://doc.pytest.org>`__, you will be interested in 37the PyTest plugin in ``pyfakefs``. 38This automatically patches all file system functions and modules in a 39similar manner as described above. 40 41The PyTest plugin provides the ``fs`` fixture for use in your test. For example: 42 43.. code:: python 44 45 def my_fakefs_test(fs): 46 # "fs" is the reference to the fake file system 47 fs.create_file('/var/data/xx1.txt') 48 assert os.path.exists('/var/data/xx1.txt') 49 50Patch using fake_filesystem_unittest.Patcher 51~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 52If you are using other means of testing like `nose <http://nose2.readthedocs.io>`__, you can do the 53patching using ``fake_filesystem_unittest.Patcher`` - the class doing the actual work 54of replacing the filesystem modules with the fake modules in the first two approaches. 55 56The easiest way is to just use ``Patcher`` as a context manager: 57 58.. code:: python 59 60 from pyfakefs.fake_filesystem_unittest import Patcher 61 62 with Patcher() as patcher: 63 # access the fake_filesystem object via patcher.fs 64 patcher.fs.create_file('/foo/bar', contents='test') 65 66 # the following code works on the fake filesystem 67 with open('/foo/bar') as f: 68 contents = f.read() 69 70You can also initialize ``Patcher`` manually: 71 72.. code:: python 73 74 from pyfakefs.fake_filesystem_unittest import Patcher 75 76 patcher = Patcher() 77 patcher.setUp() # called in the initialization code 78 ... 79 patcher.tearDown() # somewhere in the cleanup code 80 81Patch using fake_filesystem_unittest.patchfs decorator 82~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 83This is basically a convenience wrapper for the previous method. 84If you want to use the fake filesystem for a single function, you can write: 85 86.. code:: python 87 88 from pyfakefs.fake_filesystem_unittest import patchfs 89 90 @patchfs 91 def test_something(fs): 92 # access the fake_filesystem object via fs 93 fs.create_file('/foo/bar', contents='test') 94 95Note the argument name ``fs``, which is mandatory. 96 97Don't confuse this with pytest tests, where ``fs`` is the fixture name (with 98the same functionality). If you use pytest, you don't need this decorator. 99 100You can also use this to make a single unit test use the fake fs: 101 102.. code:: python 103 104 class TestSomething(unittest.TestCase): 105 106 @patchfs 107 def test_something(self, fs): 108 fs.create_file('/foo/bar', contents='test') 109 110If you want to pass additional arguments to the patcher you can just 111pass them to the decorator: 112 113.. code:: python 114 115 @patchfs(allow_root_user=False) 116 def test_something(fs): 117 # now always called as non-root user 118 os.makedirs('/foo/bar') 119 120Patch using unittest.mock (deprecated) 121~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 122You can also use ``mock.patch()`` to patch the modules manually. This approach will 123only work for the directly imported modules, therefore it is not suited for testing 124larger code bases. As the other approaches are more convenient, this one is considered 125deprecated and will not be described in detail. 126 127.. _customizing_patcher: 128 129Customizing Patcher and TestCase 130-------------------------------- 131 132Both ``fake_filesystem_unittest.Patcher`` and ``fake_filesystem_unittest.TestCase`` 133provide a few arguments to handle cases where patching does not work out of 134the box. 135In case of ``fake_filesystem_unittest.TestCase``, these arguments can either 136be set in the TestCase instance initialization, or passed to ``setUpPyfakefs()``. 137 138.. note:: If you need these arguments in ``PyTest``, you can pass them using 139 ``@pytest.mark.parametrize``. Note that you have to also provide 140 `all Patcher arguments <http://jmcgeheeiv.github.io/pyfakefs/master/modules.html#pyfakefs.fake_filesystem_unittest.Patcher>`__ 141 before the needed ones, as keyword arguments cannot be used, and you have to 142 add ``indirect=True`` as argument. 143 Alternatively, you can add your own fixture with the needed parameters. 144 145 Examples for the first approach can be found below, and in 146 `pytest_fixture_param_test.py <https://github.com/jmcgeheeiv/pyfakefs/blob/master/pyfakefs/pytest_tests/pytest_fixture_param_test.py>`__. 147 The second approach is shown in 148 `pytest_fixture_test.py <https://github.com/jmcgeheeiv/pyfakefs/blob/master/pyfakefs/pytest_tests/pytest_fixture_test.py>`__ 149 with the example fixture in `conftest.py <https://github.com/jmcgeheeiv/pyfakefs/blob/master/pyfakefs/pytest_tests/conftest.py>`__. 150 We advice to use this example fixture code as a template for your customized 151 pytest plugins. 152 153modules_to_reload 154~~~~~~~~~~~~~~~~~ 155Pyfakefs patches modules that are imported before starting the test by 156finding and replacing file system modules in all loaded modules at test 157initialization time. 158This allows to automatically patch file system related modules that are: 159 160- imported directly, for example: 161 162.. code:: python 163 164 import os 165 import pathlib.Path 166 167- imported as another name: 168 169.. code:: python 170 171 import os as my_os 172 173- imported using one of these two specially handled statements: 174 175.. code:: python 176 177 from os import path 178 from pathlib import Path 179 180Additionally, functions from file system related modules are patched 181automatically if imported like: 182 183.. code:: python 184 185 from os.path import exists 186 from os import stat 187 188This also works if importing the functions as another name: 189 190.. code:: python 191 192 from os.path import exists as my_exists 193 from io import open as io_open 194 from builtins import open as bltn_open 195 196Initializing a default argument with a file system function is also patched 197automatically: 198 199.. code:: python 200 201 import os 202 203 def check_if_exists(filepath, file_exists=os.path.exists): 204 return file_exists(filepath) 205 206There are a few cases where automatic patching does not work. We know of at 207least one specific case where this is the case: 208 209If initializing a global variable using a file system function, the 210initialization will be done using the real file system: 211 212.. code:: python 213 214 from pathlib import Path 215 216 path = Path("/example_home") 217 218In this case, ``path`` will hold the real file system path inside the test. 219 220To get these cases to work as expected under test, the respective modules 221containing the code shall be added to the ``modules_to_reload`` argument (a 222module list). 223The passed modules will be reloaded, thus allowing pyfakefs to patch them 224dynamically. All modules loaded after the initial patching described above 225will be patched using this second mechanism. 226 227Given that the example code shown above is located in the file 228``example/sut.py``, the following code will work: 229 230.. code:: python 231 232 import example 233 234 # example using unittest 235 class ReloadModuleTest(fake_filesystem_unittest.TestCase): 236 def setUp(self): 237 self.setUpPyfakefs(modules_to_reload=[example.sut]) 238 239 def test_path_exists(self): 240 file_path = '/foo/bar' 241 self.fs.create_dir(file_path) 242 self.assertTrue(example.sut.check_if_exists(file_path)) 243 244 # example using pytest 245 @pytest.mark.parametrize('fs', [[None, [example.sut]]], indirect=True) 246 def test_path_exists(fs): 247 file_path = '/foo/bar' 248 fs.create_dir(file_path) 249 assert example.sut.check_if_exists(file_path) 250 251 # example using Patcher 252 def test_path_exists(): 253 with Patcher(modules_to_reload=[example.sut]) as patcher: 254 file_path = '/foo/bar' 255 patcher.fs.create_dir(file_path) 256 assert example.sut.check_if_exists(file_path) 257 258 # example using patchfs decorator 259 @patchfs(modules_to_reload=[example.sut]) 260 def test_path_exists(fs): 261 file_path = '/foo/bar' 262 fs.create_dir(file_path) 263 assert example.sut.check_if_exists(file_path) 264 265 266modules_to_patch 267~~~~~~~~~~~~~~~~ 268Sometimes there are file system modules in other packages that are not 269patched in standard pyfakefs. To allow patching such modules, 270``modules_to_patch`` can be used by adding a fake module implementation for 271a module name. The argument is a dictionary of fake modules mapped to the 272names to be faked. 273 274This mechanism is used in pyfakefs itself to patch the external modules 275`pathlib2` and `scandir` if present, and the following example shows how to 276fake a module in Django that uses OS file system functions: 277 278.. code:: python 279 280 class FakeLocks: 281 """django.core.files.locks uses low level OS functions, fake it.""" 282 _locks_module = django.core.files.locks 283 284 def __init__(self, fs): 285 """Each fake module expects the fake file system as an __init__ 286 parameter.""" 287 # fs represents the fake filesystem; for a real example, it can be 288 # saved here and used in the implementation 289 pass 290 291 @staticmethod 292 def lock(f, flags): 293 return True 294 295 @staticmethod 296 def unlock(f): 297 return True 298 299 def __getattr__(self, name): 300 return getattr(self._locks_module, name) 301 302 ... 303 # test code using Patcher 304 with Patcher(modules_to_patch={'django.core.files.locks': FakeLocks}): 305 test_django_stuff() 306 307 # test code using unittest 308 class TestUsingDjango(fake_filesystem_unittest.TestCase): 309 def setUp(self): 310 self.setUpPyfakefs(modules_to_patch={'django.core.files.locks': FakeLocks}) 311 312 def test_django_stuff(self) 313 ... 314 315 # test code using pytest 316 @pytest.mark.parametrize('fs', [[None, None, 317 {'django.core.files.locks': FakeLocks}]], indirect=True) 318 def test_django_stuff(fs): 319 ... 320 321 # test code using patchfs decorator 322 @patchfs(modules_to_patch={'django.core.files.locks': FakeLocks}) 323 def test_django_stuff(fs): 324 ... 325 326additional_skip_names 327~~~~~~~~~~~~~~~~~~~~~ 328This may be used to add modules that shall not be patched. This is mostly 329used to avoid patching the Python file system modules themselves, but may be 330helpful in some special situations, for example if a testrunner needs to access 331the file system after test setup. To make this possible, the affected module 332can be added to ``additional_skip_names``: 333 334.. code:: python 335 336 with Patcher(additional_skip_names=['pydevd']) as patcher: 337 patcher.fs.create_file('foo') 338 339Alternatively to the module names, the modules themselves may be used: 340 341.. code:: python 342 343 import pydevd 344 345 with Patcher(additional_skip_names=[pydevd]) as patcher: 346 patcher.fs.create_file('foo') 347 348There is also the global variable ``Patcher.SKIPNAMES`` that can be extended 349for that purpose, though this seldom shall be needed (except for own pytest 350plugins, as shown in the example mentioned above). 351 352allow_root_user 353~~~~~~~~~~~~~~~ 354This is ``True`` by default, meaning that the user is considered a root user 355if the real user is a root user (e.g. has the user ID 0). If you want to run 356your tests as a non-root user regardless of the actual user rights, you may 357want to set this to ``False``. 358 359Using convenience methods 360------------------------- 361While ``pyfakefs`` can be used just with the standard Python file system 362functions, there are few convenience methods in ``fake_filesystem`` that can 363help you setting up your tests. The methods can be accessed via the 364``fake_filesystem`` instance in your tests: ``Patcher.fs``, the ``fs`` 365fixture in PyTest, or ``TestCase.fs``. 366 367File creation helpers 368~~~~~~~~~~~~~~~~~~~~~ 369To create files, directories or symlinks together with all the directories 370in the path, you may use ``create_file()``, ``create_dir()`` and 371``create_symlink()``, respectively. 372 373``create_file()`` also allows you to set the file mode and the file contents 374together with the encoding if needed. Alternatively, you can define a file 375size without contents - in this case, you will not be able to perform 376standard I\O operations on the file (may be used to "fill up" the file system 377with large files). 378 379.. code:: python 380 381 from pyfakefs.fake_filesystem_unittest import TestCase 382 383 class ExampleTestCase(TestCase): 384 def setUp(self): 385 self.setUpPyfakefs() 386 387 def test_create_file(self): 388 file_path = '/foo/bar/test.txt' 389 self.fs.create_file(file_path, contents = 'test') 390 with open(file_path) as f: 391 self.assertEqual('test', f.read()) 392 393``create_dir()`` behaves like ``os.makedirs()``. 394 395Access to files in the real file system 396~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 397If you want to have read access to real files or directories, you can map 398them into the fake file system using ``add_real_file()``, 399``add_real_directory()``, ``add_real_symlink()`` and ``add_real_paths()``. 400They take a file path, a directory path, a symlink path, or a list of paths, 401respectively, and make them accessible from the fake file system. By 402default, the contents of the mapped files and directories are read only on 403demand, so that mapping them is relatively cheap. The access to the files is 404by default read-only, but even if you add them using ``read_only=False``, 405the files are written only in the fake system (e.g. in memory). The real 406files are never changed. 407 408``add_real_file()``, ``add_real_directory()`` and ``add_real_symlink()`` also 409allow you to map a file or a directory tree into another location in the 410fake filesystem via the argument ``target_path``. 411 412.. code:: python 413 414 from pyfakefs.fake_filesystem_unittest import TestCase 415 416 class ExampleTestCase(TestCase): 417 418 fixture_path = os.path.join(os.path.dirname(__file__), 'fixtures') 419 def setUp(self): 420 self.setUpPyfakefs() 421 # make the file accessible in the fake file system 422 self.fs.add_real_directory(self.fixture_path) 423 424 def test_using_fixture1(self): 425 with open(os.path.join(self.fixture_path, 'fixture1.txt') as f: 426 # file contents are copied to the fake file system 427 # only at this point 428 contents = f.read() 429 430You can do the same using ``pytest`` by using a fixture for test setup: 431 432.. code:: python 433 434 import pytest 435 import os 436 437 fixture_path = os.path.join(os.path.dirname(__file__), 'fixtures') 438 439 @pytest.fixture 440 def my_fs(fs): 441 fs.add_real_directory(fixture_path) 442 yield fs 443 444 def test_using_fixture1(my_fs): 445 with open(os.path.join(fixture_path, 'fixture1.txt') as f: 446 contents = f.read() 447 448When using ``pytest`` another option is to load the contents of the real file 449in a fixture and pass this fixture to the test function **before** passing 450the ``fs`` fixture. 451 452.. code:: python 453 454 import pytest 455 import os 456 457 @pytest.fixture 458 def content(): 459 fixture_path = os.path.join(os.path.dirname(__file__), 'fixtures') 460 with open(os.path.join(fixture_path, 'fixture1.txt') as f: 461 contents = f.read() 462 return contents 463 464 def test_using_file_contents(content, fs): 465 fs.create_file("fake/path.txt") 466 assert content != "" 467 468 469Handling mount points 470~~~~~~~~~~~~~~~~~~~~~ 471Under Linux and MacOS, the root path (``/``) is the only mount point created 472in the fake file system. If you need support for more mount points, you can add 473them using ``add_mount_point()``. 474 475Under Windows, drives and UNC paths are internally handled as mount points. 476Adding a file or directory on another drive or UNC path automatically 477adds a mount point for that drive or UNC path root if needed. Explicitly 478adding mount points shall not be needed under Windows. 479 480A mount point has a separate device ID (``st_dev``) under all systems, and 481some operations (like ``rename``) are not possible for files located on 482different mount points. The fake file system size (if used) is also set per 483mount point. 484 485Setting the file system size 486~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 487If you need to know the file system size in your tests (for example for 488testing cleanup scripts), you can set the fake file system size using 489``set_disk_usage()``. By default, this sets the total size in bytes of the 490root partition; if you add a path as parameter, the size will be related to 491the mount point (see above) the path is related to. 492 493By default, the size of the fake file system is considered infinite. As soon 494as you set a size, all files will occupy the space according to their size, 495and you may fail to create new files if the fake file system is full. 496 497.. code:: python 498 499 from pyfakefs.fake_filesystem_unittest import TestCase 500 501 class ExampleTestCase(TestCase): 502 503 def setUp(self): 504 self.setUpPyfakefs() 505 self.fs.set_disk_usage(100) 506 507 def test_disk_full(self): 508 with open('/foo/bar.txt', 'w') as f: 509 self.assertRaises(OSError, f.write, 'a' * 200) 510 511To get the file system size, you may use ``get_disk_usage()``, which is 512modeled after ``shutil.disk_usage()``. 513 514Pausing patching 515~~~~~~~~~~~~~~~~ 516Sometimes, you may want to access the real filesystem inside the test with 517no patching applied. This can be achieved by using the ``pause/resume`` 518functions, which exist in ``fake_filesystem_unittest.Patcher``, 519``fake_filesystem_unittest.TestCase`` and ``fake_filesystem.FakeFilesystem``. 520There is also a context manager class ``fake_filesystem_unittest.Pause`` 521which encapsulates the calls to ``pause()`` and ``resume()``. 522 523Here is an example that tests the usage with the pyfakefs pytest fixture: 524 525.. code:: python 526 527 from pyfakefs.fake_filesystem_unittest import Pause 528 529 def test_pause_resume_contextmanager(fs): 530 fake_temp_file = tempfile.NamedTemporaryFile() 531 assert os.path.exists(fake_temp_file.name) 532 fs.pause() 533 assert not os.path.exists(fake_temp_file.name) 534 real_temp_file = tempfile.NamedTemporaryFile() 535 assert os.path.exists(real_temp_file.name) 536 fs.resume() 537 assert not os.path.exists(real_temp_file.name) 538 assert os.path.exists(fake_temp_file.name) 539 540Here is the same code using a context manager: 541 542.. code:: python 543 544 from pyfakefs.fake_filesystem_unittest import Pause 545 546 def test_pause_resume_contextmanager(fs): 547 fake_temp_file = tempfile.NamedTemporaryFile() 548 assert os.path.exists(fake_temp_file.name) 549 with Pause(fs): 550 assert not os.path.exists(fake_temp_file.name) 551 real_temp_file = tempfile.NamedTemporaryFile() 552 assert os.path.exists(real_temp_file.name) 553 assert not os.path.exists(real_temp_file.name) 554 assert os.path.exists(fake_temp_file.name) 555 556Troubleshooting 557--------------- 558 559Modules not working with pyfakefs 560~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 561 562Modules may not work with ``pyfakefs`` for several reasons. ``pyfakefs`` 563works by patching some file system related modules and functions, specifically: 564 565- most file system related functions in the ``os`` and ``os.path`` modules 566- the ``pathlib`` module 567- the build-in ``open`` function and ``io.open`` 568- ``shutil.disk_usage`` 569 570Other file system related modules work with ``pyfakefs``, because they use 571exclusively these patched functions, specifically ``shutil`` (except for 572``disk_usage``), ``tempfile``, ``glob`` and ``zipfile``. 573 574A module may not work with ``pyfakefs`` because of one of the following 575reasons: 576 577- It uses a file system related function of the mentioned modules that is 578 not or not correctly patched. Mostly these are functions that are seldom 579 used, but may be used in Python libraries (this has happened for example 580 with a changed implementation of ``shutil`` in Python 3.7). Generally, 581 these shall be handled in issues and we are happy to fix them. 582- It uses file system related functions in a way that will not be patched 583 automatically. This is the case for functions that are executed while 584 reading a module. This case and a possibility to make them work is 585 documented above under ``modules_to_reload``. 586- It uses OS specific file system functions not contained in the Python 587 libraries. These will not work out of the box, and we generally will not 588 support them in ``pyfakefs``. If these functions are used in isolated 589 functions or classes, they may be patched by using the ``modules_to_patch`` 590 parameter (see the example for file locks in Django above), and if there 591 are more examples for patches that may be useful, we may add them in the 592 documentation. 593- It uses C libraries to access the file system. There is no way no make 594 such a module work with ``pyfakefs`` - if you want to use it, you have to 595 patch the whole module. In some cases, a library implemented in Python with 596 a similar interface already exists. An example is ``lxml``, 597 which can be substituted with ``ElementTree`` in most cases for testing. 598 599A list of Python modules that are known to not work correctly with 600``pyfakefs`` will be collected here: 601 602- ``multiprocessing`` has several issues (related to points 1 and 3 above). 603 Currently there are no plans to fix this, but this may change in case of 604 sufficient demand. 605 606If you are not sure if a module can be handled, or how to do it, you can 607always write a new issue, of course! 608 609OS temporary directories 610~~~~~~~~~~~~~~~~~~~~~~~~ 611 612Tests relying on a completely empty file system on test start will fail. 613As ``pyfakefs`` does not fake the ``tempfile`` module (as described above), 614a temporary directory is required to ensure ``tempfile`` works correctly, 615e.g., that ``tempfile.gettempdir()`` will return a valid value. This 616means that any newly created fake file system will always have either a 617directory named ``/tmp`` when running on Linux or Unix systems, 618``/var/folders/<hash>/T`` when running on MacOs and 619``C:\Users\<user>\AppData\Local\Temp`` on Windows. 620 621User rights 622~~~~~~~~~~~ 623 624If you run pyfakefs tests as root (this happens by default if run in a 625docker container), pyfakefs also behaves as a root user, for example can 626write to write-protected files. This may not be the expected behavior, and 627can be changed. 628Pyfakefs has a rudimentary concept of user rights, which differentiates 629between root user (with the user id 0) and any other user. By default, 630pyfakefs assumes the user id of the current user, but you can change 631that using ``fake_filesystem.set_uid()`` in your setup. This allows to run 632tests as non-root user in a root user environment and vice verse. 633Another possibility is the convenience argument ``allow_root_user`` 634described above. 635