Python bindings: Fix editable install + Execute Python2.7 workflow tests (#2044)

* Python binding:
- Added missing `license` field in pyproject.toml file
- Fixed editable mode install and some more code cleanup in setup.py
- Refreshed README.md
- Replaced f-string formatter in tests with `format` method in order to be py2-compatible
- Fixed typos
- PEP8 fixes

* GitHub Action: Install Python2.7 and run tests for re-tagged wheels on native arch runners only

* Python bindings:
- Use #x formatter to format hex values

Author: Antelox

.github/workflows/build-wheels-publish.yml

@@ -160,14 +160,36 @@ jobs:
         run: |
           python -m pip install -U pip wheel && Get-ChildItem -Path wheelhouse/ -Filter *cp38*.whl | Foreach-Object {
             python -m wheel tags --python-tag='py2' --abi-tag=none $_.FullName
+            break
           }
+
       - name: '🚧 Python 2.7 wheels re-tagging - Non-Windows'
         if: matrix.os != 'windows-2019'
         env:
           PIP_BREAK_SYSTEM_PACKAGES: 1
         run: |
           python3 -m pip install -U pip wheel && python3 -m wheel tags --python-tag='py2' --abi-tag=none wheelhouse/*cp38*.whl
 
+      - uses: LizardByte/setup-python-action@master
+        if: (matrix.os == 'ubuntu-latest' && matrix.arch == 'x86_64' && matrix.cibw_build == 'cp38-manylinux') || matrix.os == 'macos-latest' || (matrix.os == 'windows-2019' && matrix.arch == 'AMD64')
+        with:
+          python-version: 2.7
+
+      - name: 'Python 2.7 tests - Windows'
+        if: matrix.os == 'windows-2019' && matrix.arch == 'AMD64'
+        run: |
+          C:\Python27\python.exe -m pip install -U pip pytest && Get-ChildItem -Path wheelhouse/ -Filter *py2*.whl | Foreach-Object {
+            C:\Python27\python.exe -m pip install $_.FullName
+            C:\Python27\python.exe -m pytest bindings/python/tests
+            break
+          }
+
+      # we install and test python2.7 wheels only on native arch
+      # NOTE: no python2.7 support for macos-13: https://github.com/LizardByte/setup-python-action/issues/2
+      - name: 'Python 2.7 tests - Non-Windows'
+        if: (matrix.os == 'ubuntu-latest' && matrix.arch == 'x86_64' && matrix.cibw_build == 'cp38-manylinux') || matrix.os == 'macos-latest'
+        run: python2 -m pip install wheelhouse/*py2*.whl && python2 -m pip install -U pip pytest && python2 -m pytest bindings/python/tests
+
       - uses: actions/upload-artifact@v4
         with:
           name: cibw-wheels-${{ matrix.os }}-${{ strategy.job-index }}-py38

bindings/python/README.md

@@ -25,27 +25,21 @@ Originally written by Nguyen Anh Quynh, polished and redesigned by elicn, mainta
 Install a prebuilt wheel from PyPI:
 
 ```bash
-pip3 install unicorn
+python3 -m pip install unicorn
 ```
 
 In case you would like to develop the bindings:
 
 ```bash
-# Python3
-DEBUG=1 THREADS=4 pip3 install --user -e .
+DEBUG=1 THREADS=4 python3 -m pip install --user -e .
 # Workaround for Pylance
-DEBUG=1 THREADS=4 pip3 install --user -e . --config-settings editable_mode=strict
-# Python2
-DEBUG=1 THREADS=4 pip install -e .
+DEBUG=1 THREADS=4 python3 -m pip install --user -e . --config-settings editable_mode=strict
 ```
 
 or install it by building it by yourself:
 
 ```bash
-# Python3
-THREADS=4 pip3 install --user .
-# Python2, unfortunately `pip2` doesn't support in-tree build
-THREADS=4 python3 setup.py install
+THREADS=4 python3 -m pip install --user .
 ```
 
 Explanations for arguments:
@@ -59,4 +53,5 @@ Note that you should setup a valid building environment according to docs/COMPIL
 
 ## Python2 compatibility
 
-By default, Unicorn python bindings will be maintained against Python3 as it offers more powerful features which improves developing efficiency. Meanwhile, Unicorn will only keep compatible with all features Unicorn1 offers regarding Python2 because Python2 has reached end-of-life for more than 3 years as the time of writing this README. While offering all features for both Python2 & Python3 is desirable and doable, it inevitably costs too much efforts to maintain and few users really rely on this. Therefore, we assume that if users still stick to Python2, previous Unicorn1 features we offer should be enough. If you really want some new features Unicorn2 offers, please check and pull request to `unicorn/unicorn_py2``. We are happy to review and accept!
+By default, Unicorn python bindings works with Python3.7 and above, as it offers more powerful features which improves developing efficiency compared to Python2. However, Unicorn will only keep compatible with all features Unicorn1 offers regarding Python2 because it has reached end-of-life for more than 3 years at the time of writing this README. While offering all features for both Python2 & Python3 is desirable and doable, it inevitably costs too much efforts to maintain and few users really rely on this. Therefore, we assume that if users still stick to Python2, previous Unicorn1 features should be enough. If you really want some new features Unicorn2 offers, please check and pull request to `unicorn/unicorn_py2`. We are happy to review and accept!
+Even though the build of wheel packages requires Python3, it's still possible to re-tag the wheel produced from Python3 with `py2` tag and then run `python2 -m pip install <retagged-wheel-py>`. For detailed commands please refer to our workflow files.

bindings/python/pyproject.toml

@@ -12,6 +12,7 @@ authors = [
 description = "Unicorn CPU emulator engine"
 readme = "README.md"
 keywords = ["emulation", "qemu", "unicorn"]
+license = { text = "BSD License" }
 classifiers = [
     'License :: OSI Approved :: BSD License',
     'Programming Language :: Python :: 2.7',

bindings/python/setup.py

@@ -8,9 +8,8 @@
 import subprocess
 import sys
 from setuptools import setup
-from setuptools.command.build import build
+from setuptools.command.build_py import build_py
 from setuptools.command.sdist import sdist
-from setuptools.command.bdist_egg import bdist_egg
 
 log = logging.getLogger(__name__)
 
@@ -134,7 +133,7 @@ def run(self):
         return super().run()
 
 
-class CustomBuild(build):
+class CustomBuild(build_py):
     def run(self):
         if 'LIBUNICORN_PATH' in os.environ:
             log.info("Skipping building C extensions since LIBUNICORN_PATH is set")
@@ -144,30 +143,7 @@ def run(self):
         return super().run()
 
 
-class CustomBDistEgg(bdist_egg):
-    def run(self):
-        self.run_command('build')
-        return super().run()
-
-
-cmdclass = {'build': CustomBuild, 'sdist': CustomSDist, 'bdist_egg': CustomBDistEgg}
-
-try:
-    from setuptools.command.develop import develop
-
-
-    class CustomDevelop(develop):
-        def run(self):
-            log.info("Building C extensions")
-            build_libraries()
-            return super().run()
-
-
-    cmdclass['develop'] = CustomDevelop
-except ImportError:
-    print("Proper 'develop' support unavailable.")
-
 setup(
-    cmdclass=cmdclass,
+    cmdclass={'build_py': CustomBuild, 'sdist': CustomSDist},
     has_ext_modules=lambda: True,  # It's not a Pure Python wheel
 )

bindings/python/tests/test_arm64.py

@@ -87,7 +87,8 @@ def test_arm64_read_sctlr():
 
 def test_arm64_hook_mrs():
     def _hook_mrs(uc, reg, cp_reg, _):
-        print(f">>> Hook MRS instruction: reg = 0x{reg:x}(UC_ARM64_REG_X2) cp_reg = {cp_reg}")
+        print(">>> Hook MRS instruction: reg = {reg:#x}(UC_ARM64_REG_X2) cp_reg = {cp_reg}".format(reg=reg,
+                                                                                                   cp_reg=cp_reg))
         uc.reg_write(reg, 0x114514)
         print(">>> Write 0x114514 to X")
 
@@ -111,7 +112,7 @@ def _hook_mrs(uc, reg, cp_reg, _):
         # Start emulation
         mu.emu_start(0x1000, 0x1000 + len(ARM64_MRS_CODE))
 
-        print(f">>> X2 = {mu.reg_read(UC_ARM64_REG_X2):x}")
+        print(">>> X2 = {reg:#x}".format(reg=mu.reg_read(UC_ARM64_REG_X2)))
 
     except UcError as e:
         print("ERROR: %s" % e)

bindings/python/tests/test_ctl.py

@@ -2,6 +2,8 @@
 # Sample code for Unicorn.
 # By Lazymio(@wtdcode), 2021
 
+import pytest
+import sys
 from unicorn import *
 from unicorn.x86_const import *
 from datetime import datetime
@@ -20,7 +22,9 @@ def test_uc_ctl_read():
 
     timeout = uc.ctl_get_timeout()
 
-    print(f">>> arch={arch} mode={mode} page size={page_size} timeout={timeout}")
+    print(">>> arch={arch} mode={mode} page size={page_size} timeout={timeout}".format(arch=arch, mode=mode,
+                                                                                       page_size=page_size,
+                                                                                       timeout=timeout))
 
 
 def time_emulation(uc, start, end):
@@ -31,6 +35,8 @@ def time_emulation(uc, start, end):
     return (datetime.now() - n).total_seconds() * 1e6
 
 
+# TODO: Check if worth adapting the ctl_request_cache method for py2 bindings
+@pytest.mark.skipif(sys.version_info < (3, 7), reason="requires python3.7 or higher")
 def test_uc_ctl_tb_cache():
     # Initialize emulator in X86-32bit mode
     uc = Uc(UC_ARCH_X86, UC_MODE_32)
@@ -52,7 +58,7 @@ def test_uc_ctl_tb_cache():
     # Now we request cache for all TBs.
     for i in range(8):
         tb = uc.ctl_request_cache(addr + i * 512)
-        print(f">>> TB is cached at {hex(tb[0])} which has {tb[1]} instructions with {tb[2]} bytes")
+        print(">>> TB is cached at {:#x} which has {} instructions with {} bytes".format(tb[0], tb[1], tb[2]))
 
     # Do emulation with all TB cached.
     cached = time_emulation(uc, addr, addr + len(code))
@@ -63,17 +69,22 @@ def test_uc_ctl_tb_cache():
 
     evicted = time_emulation(uc, addr, addr + len(code))
 
-    print(f">>> Run time: First time {standard}, Cached: {cached}, Cached evicted: {evicted}")
+    print(">>> Run time: First time {standard}, Cached: {cached}, Cached evicted: {evicted}".format(standard=standard,
+                                                                                                    cached=cached,
+                                                                                                    evicted=evicted))
 
 
 def trace_new_edge(uc, cur, prev, data):
-    print(f">>> Getting a new edge from {hex(prev.pc + prev.size - 1)} to {hex(cur.pc)}")
+    print(">>> Getting a new edge from {:#x} to {:#x}".format(prev.pc + prev.size - 1, cur.pc))
 
 
 def trace_tcg_sub(uc, address, arg1, arg2, size, data):
-    print(f">>> Get a tcg sub opcode at {hex(address)} with args: {arg1} and {arg2}")
+    print(">>> Get a tcg sub opcode at {address:#x} with args: {arg1} and {arg2}".format(address=address, arg1=arg1,
+                                                                                         arg2=arg2))
 
 
+# TODO: Check if worth adapting the hook_add method for py2 bindings
+@pytest.mark.skipif(sys.version_info < (3, 7), reason="requires python3.7 or higher")
 def test_uc_ctl_exits():
     uc = Uc(UC_ARCH_X86, UC_MODE_32)
     addr = 0x1000
@@ -110,15 +121,15 @@ def test_uc_ctl_exits():
     eax = uc.reg_read(UC_X86_REG_EAX)
     ebx = uc.reg_read(UC_X86_REG_EBX)
 
-    print(f">>> eax = {hex(eax)} and ebx = {hex(ebx)} after the first emulation")
+    print(">>> eax = {eax:#x} and ebx = {ebx:#x} after the first emulation".format(eax=eax, ebx=ebx))
 
     # This should stop at ADDRESS + 8, even though we don't provide an exit.
     uc.emu_start(addr, 0)
 
     eax = uc.reg_read(UC_X86_REG_EAX)
     ebx = uc.reg_read(UC_X86_REG_EBX)
 
-    print(f">>> eax = {hex(eax)} and ebx = {hex(ebx)} after the first emulation")
+    print(">>> eax = {eax:#x} and ebx = {ebx:#x} after the first emulation".format(eax=eax, ebx=ebx))
 
 
 if __name__ == "__main__":

bindings/python/tests/test_network_auditing.py

@@ -356,7 +356,7 @@ def hook_intr(uc, intno, user_data):
             fd = args[0]
             how = args[1]
 
-            msg = "fd(%d) is shutted down because of %d" % (fd, how)
+            msg = "fd(%d) is shut down because of %d" % (fd, how)
             fd_chains.add_log(fd, msg)
             print_sockcall(msg)
 

bindings/python/tests/test_x86.py

@@ -639,13 +639,16 @@ def test_x86_16():
 
 
 def mmio_read_cb(uc, offset, size, data):
-    print(f">>> Read IO memory at offset {hex(offset)} with {hex(size)} bytes and return 0x19260817")
+    print(">>> Read IO memory at offset {offset:#x} with {size:#x} bytes and return 0x19260817".format(offset=offset,
+                                                                                                       size=size))
 
     return 0x19260817
 
 
 def mmio_write_cb(uc, offset, size, value, data):
-    print(f">>> Write value {hex(value)} to IO memory at offset {hex(offset)} with {hex(size)} bytes")
+    print(">>> Write value {value:#x} to IO memory at offset {offset:#x} with {size:#x} bytes".format(value=value,
+                                                                                                      offset=offset,
+                                                                                                      size=size))
 
 
 def test_i386_mmio():
@@ -668,7 +671,7 @@ def test_i386_mmio():
         mu.emu_start(0x10000, 0x10000 + len(X86_MMIO_CODE))
 
         # now print out some registers
-        print(f">>> Emulation done. ECX={hex(mu.reg_read(UC_X86_REG_ECX))}")
+        print(">>> Emulation done. ECX={reg:#x}".format(reg=mu.reg_read(UC_X86_REG_ECX)))
 
     except UcError as e:
         print("ERROR: %s" % e)