Skip to content

Commit dea3c37

Browse files
AnteloxAntelox
authored
CI(full),CI(release): Python bindings: (#2100)
- Switched to the ubuntu-24.04-arm runner - Bumped Windows runner to windows-2022 and Visual Studio 17 2022 GENERATORS - Minor changes about checks in workflow jobs - Fixed the pagefile job (even though should not be really needed) - Refreshed the TO BE CHECKED regress tests to either update or remove the skip conditions - Added a test to check if the created sdist archive is ok
1 parent bf87b9a commit dea3c37

File tree

6 files changed

+43
-40
lines changed

6 files changed

+43
-40
lines changed

‎.github/workflows/build-wheels-publish.yml

+36-31
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ jobs:
4545
fail-fast: false
4646
matrix:
4747
include:
48-
# NOTE: aarch64 builds are super slow due to QEMU emulation. Making this to parallelize and speed up workflow
48+
# NOTE: Doing this to parallelize and speed up workflow
4949
# i686 - manylinux
5050
- { os: ubuntu-latest, arch: i686, cibw_build: 'cp38-manylinux*', cibw_skip: '' }
5151
# i686 - musllinux
@@ -55,13 +55,13 @@ jobs:
5555
# x86_64 - musllinux
5656
- { os: ubuntu-latest, arch: x86_64, cibw_build: 'cp38-musllinux*', cibw_skip: '' }
5757
# aarch64 - manylinux
58-
- { os: ubuntu-22.04, arch: aarch64, cibw_build: 'cp38-manylinux*', cibw_skip: '' }
58+
- { os: ubuntu-24.04-arm, arch: aarch64, cibw_build: 'cp38-manylinux*', cibw_skip: '' }
5959
# aarch64 - musllinux
60-
- { os: ubuntu-22.04, arch: aarch64, cibw_build: 'cp38-musllinux*', cibw_skip: '' }
60+
- { os: ubuntu-24.04-arm, arch: aarch64, cibw_build: 'cp38-musllinux*', cibw_skip: '' }
6161
- { os: macos-13, arch: x86_64, cibw_build: 'cp38*', cibw_skip: '' }
6262
- { os: macos-latest, arch: arm64, cibw_build: 'cp38*', cibw_skip: '' }
63-
- { os: windows-2019, arch: AMD64, cibw_build: 'cp38*', cibw_skip: '' }
64-
- { os: windows-2019, arch: x86, cibw_build: 'cp38*', cibw_skip: '' }
63+
- { os: windows-2022, arch: AMD64, cibw_build: 'cp38*', cibw_skip: '' }
64+
- { os: windows-2022, arch: x86, cibw_build: 'cp38*', cibw_skip: '' }
6565

6666
steps:
6767
- uses: actions/checkout@v4
@@ -79,15 +79,16 @@ jobs:
7979
8080
- name: "Increase pagefile to avoid OOM"
8181
if: runner.os == 'Windows'
82-
uses: al-cheb/configure-pagefile-action@v1.2
82+
uses: al-cheb/configure-pagefile-action@v1.4
8383
with:
8484
minimum-size: 8GB
85+
disk-root: "C:"
8586

8687
- name: '🛠️ Add msbuild to PATH'
8788
if: runner.os == 'Windows'
8889
uses: microsoft/setup-msbuild@v2
8990
with:
90-
vs-version: '16.5'
91+
vs-version: '17'
9192

9293
# https://cibuildwheel.pypa.io/en/stable/faq/#macos-building-cpython-38-wheels-on-arm64
9394
- uses: actions/setup-python@v5
@@ -108,7 +109,7 @@ jobs:
108109
arch: x64
109110

110111
- name: '🛠️ Set up QEMU'
111-
if: runner.os == 'Linux' && matrix.arch != 'x86_64'
112+
if: runner.os == 'Linux' && matrix.arch == 'i686'
112113
uses: docker/setup-qemu-action@v3
113114

114115
- name: '🚧 cibuildwheel run'
@@ -118,7 +119,7 @@ jobs:
118119
CIBW_BUILD: ${{ matrix.cibw_build }}
119120
CIBW_SKIP: ${{ matrix.cibw_skip }}
120121
CIBW_ARCHS: ${{ matrix.arch }}
121-
CIBW_ENVIRONMENT: DEBUG=${{ env.UNICORN_DEBUG }}
122+
CIBW_ENVIRONMENT: DEBUG=${{ env.UNICORN_DEBUG }} GENERATORS='Visual Studio 17 2022'
122123
CIBW_ENVIRONMENT_PASS_LINUX: DEBUG
123124
CIBW_TEST_EXTRAS: test
124125
CIBW_TEST_COMMAND: 'python -m unittest discover -v {project}/tests/regress "*.py"'
@@ -138,21 +139,21 @@ jobs:
138139
python3 -m wheel tags --python-tag='py2' --abi-tag=none wheelhouse/*cp38*.whl
139140
140141
- uses: LizardByte/[email protected]
141-
if: (matrix.os == 'ubuntu-latest' && matrix.arch == 'x86_64' && matrix.cibw_build == 'cp38-manylinux*') || matrix.os == 'macos-latest' || (matrix.os == 'windows-2019' && matrix.arch == 'AMD64')
142+
if: (runner.os == 'Linux' && (matrix.arch == 'x86_64' || matrix.arch == 'aarch64') && matrix.cibw_build == 'cp38-manylinux*') || (runner.os == 'macOS' && matrix.arch == 'arm64') || (runner.os == 'Windows' && matrix.arch == 'AMD64')
142143
with:
143144
python-version: 2.7
144145

145146
# we install and test python2.7 wheels only on native arch
146147
- name: 'Python 2.7 tests - Windows'
147-
if: matrix.os == 'windows-2019' && matrix.arch == 'AMD64'
148+
if: runner.os == 'Windows' && matrix.arch == 'AMD64'
148149
shell: bash
149150
run: |
150151
C:/Python27/python.exe -m pip install capstone==4.0.2 wheelhouse/*py2*.whl
151152
C:/Python27/python.exe -m unittest discover tests/regress "*.py"
152153
153154
# NOTE: no python2.7 support for macos-13: https://github.com/LizardByte/setup-python-action/issues/2
154155
- name: 'Python 2.7 tests - Non-Windows'
155-
if: (matrix.os == 'ubuntu-latest' && matrix.arch == 'x86_64' && matrix.cibw_build == 'cp38-manylinux*') || matrix.os == 'macos-latest'
156+
if: (runner.os == 'Linux' && (matrix.arch == 'x86_64' || matrix.arch == 'aarch64') && matrix.cibw_build == 'cp38-manylinux*') || (runner.os == 'macOS' && matrix.arch == 'arm64')
156157
run: |
157158
python2 -m pip install capstone==4.0.2 wheelhouse/*py2*.whl
158159
python2 -m unittest discover tests/regress "*.py"
@@ -170,7 +171,7 @@ jobs:
170171
fail-fast: false
171172
matrix:
172173
include:
173-
# NOTE: aarch64 builds are super slow due to QEMU emulation. Making this to parallelize and speed up workflow
174+
# NOTE: Doing this to parallelize and speed up workflow
174175
# i686 - manylinux
175176
- { os: ubuntu-latest, arch: i686, cibw_build: 'cp37-manylinux*', cibw_skip: '' }
176177
- { os: ubuntu-latest, arch: i686, cibw_build: 'cp39-manylinux*', cibw_skip: '' }
@@ -200,23 +201,23 @@ jobs:
200201
- { os: ubuntu-latest, arch: x86_64, cibw_build: 'cp312-musllinux*', cibw_skip: '' }
201202
- { os: ubuntu-latest, arch: x86_64, cibw_build: 'cp313-musllinux*', cibw_skip: '' }
202203
# aarch64 - manylinux
203-
- { os: ubuntu-22.04, arch: aarch64, cibw_build: 'cp37-manylinux*', cibw_skip: '' }
204-
- { os: ubuntu-22.04, arch: aarch64, cibw_build: 'cp39-manylinux*', cibw_skip: '' }
205-
- { os: ubuntu-22.04, arch: aarch64, cibw_build: 'cp310-manylinux*', cibw_skip: '' }
206-
- { os: ubuntu-22.04, arch: aarch64, cibw_build: 'cp311-manylinux*', cibw_skip: '' }
207-
- { os: ubuntu-22.04, arch: aarch64, cibw_build: 'cp312-manylinux*', cibw_skip: '' }
208-
- { os: ubuntu-22.04, arch: aarch64, cibw_build: 'cp313-manylinux*', cibw_skip: '' }
204+
- { os: ubuntu-24.04-arm, arch: aarch64, cibw_build: 'cp37-manylinux*', cibw_skip: '' }
205+
- { os: ubuntu-24.04-arm, arch: aarch64, cibw_build: 'cp39-manylinux*', cibw_skip: '' }
206+
- { os: ubuntu-24.04-arm, arch: aarch64, cibw_build: 'cp310-manylinux*', cibw_skip: '' }
207+
- { os: ubuntu-24.04-arm, arch: aarch64, cibw_build: 'cp311-manylinux*', cibw_skip: '' }
208+
- { os: ubuntu-24.04-arm, arch: aarch64, cibw_build: 'cp312-manylinux*', cibw_skip: '' }
209+
- { os: ubuntu-24.04-arm, arch: aarch64, cibw_build: 'cp313-manylinux*', cibw_skip: '' }
209210
# aarch64 - musllinux
210-
- { os: ubuntu-22.04, arch: aarch64, cibw_build: 'cp37-musllinux*', cibw_skip: '' }
211-
- { os: ubuntu-22.04, arch: aarch64, cibw_build: 'cp39-musllinux*', cibw_skip: '' }
212-
- { os: ubuntu-22.04, arch: aarch64, cibw_build: 'cp310-musllinux*', cibw_skip: '' }
213-
- { os: ubuntu-22.04, arch: aarch64, cibw_build: 'cp311-musllinux*', cibw_skip: '' }
214-
- { os: ubuntu-22.04, arch: aarch64, cibw_build: 'cp312-musllinux*', cibw_skip: '' }
215-
- { os: ubuntu-22.04, arch: aarch64, cibw_build: 'cp313-musllinux*', cibw_skip: '' }
211+
- { os: ubuntu-24.04-arm, arch: aarch64, cibw_build: 'cp37-musllinux*', cibw_skip: '' }
212+
- { os: ubuntu-24.04-arm, arch: aarch64, cibw_build: 'cp39-musllinux*', cibw_skip: '' }
213+
- { os: ubuntu-24.04-arm, arch: aarch64, cibw_build: 'cp310-musllinux*', cibw_skip: '' }
214+
- { os: ubuntu-24.04-arm, arch: aarch64, cibw_build: 'cp311-musllinux*', cibw_skip: '' }
215+
- { os: ubuntu-24.04-arm, arch: aarch64, cibw_build: 'cp312-musllinux*', cibw_skip: '' }
216+
- { os: ubuntu-24.04-arm, arch: aarch64, cibw_build: 'cp313-musllinux*', cibw_skip: '' }
216217
- { os: macos-13, arch: x86_64, cibw_build: 'cp*', cibw_skip: '*36* *38*' }
217218
- { os: macos-latest, arch: arm64, cibw_build: 'cp*', cibw_skip: '*36* *37* *38*' }
218-
- { os: windows-2019, arch: AMD64, cibw_build: 'cp*', cibw_skip: '*36* *38*' }
219-
- { os: windows-2019, arch: x86, cibw_build: 'cp*', cibw_skip: '*36* *38*' }
219+
- { os: windows-2022, arch: AMD64, cibw_build: 'cp*', cibw_skip: '*36* *38*' }
220+
- { os: windows-2022, arch: x86, cibw_build: 'cp*', cibw_skip: '*36* *38*' }
220221
if: ${{ inputs.fullMode == 1 || startsWith(github.ref, 'refs/tags') || contains(github.event.head_commit.message, 'CI(full)') }}
221222
steps:
222223
- uses: actions/checkout@v4
@@ -234,15 +235,16 @@ jobs:
234235
235236
- name: "Increase pagefile to avoid OOM"
236237
if: runner.os == 'Windows'
237-
uses: al-cheb/configure-pagefile-action@v1.2
238+
uses: al-cheb/configure-pagefile-action@v1.4
238239
with:
239240
minimum-size: 8GB
241+
disk-root: "C:"
240242

241243
- name: '🛠️ Add msbuild to PATH'
242244
if: runner.os == 'Windows'
243245
uses: microsoft/setup-msbuild@v2
244246
with:
245-
vs-version: '16.5'
247+
vs-version: '17'
246248

247249
- name: '🛠️ Win MSVC 32 dev cmd setup'
248250
if: runner.os == 'Windows' && matrix.arch == 'x86'
@@ -257,7 +259,7 @@ jobs:
257259
arch: x64
258260

259261
- name: '🛠️ Set up QEMU'
260-
if: runner.os == 'Linux' && matrix.arch != 'x86_64'
262+
if: runner.os == 'Linux' && matrix.arch == 'i686'
261263
uses: docker/setup-qemu-action@v3
262264

263265
- name: '🚧 cibuildwheel run'
@@ -267,7 +269,7 @@ jobs:
267269
CIBW_BUILD: ${{ matrix.cibw_build }}
268270
CIBW_SKIP: ${{ matrix.cibw_skip }}
269271
CIBW_ARCHS: ${{ matrix.arch }}
270-
CIBW_ENVIRONMENT: DEBUG=${{ env.UNICORN_DEBUG }}
272+
CIBW_ENVIRONMENT: DEBUG=${{ env.UNICORN_DEBUG }} GENERATORS='Visual Studio 17 2022'
271273
CIBW_ENVIRONMENT_PASS_LINUX: DEBUG
272274
CIBW_TEST_EXTRAS: test
273275
CIBW_TEST_COMMAND: 'python -m unittest discover -v {project}/tests/regress "*.py"'
@@ -293,6 +295,9 @@ jobs:
293295
cd bindings/python
294296
python3 -m pip install -U pip build
295297
python3 -m build --sdist
298+
python3 -m pip install dist/*.tar.gz
299+
cd ~
300+
python3 -c 'import unicorn; print(f"Unicorn version installed from sdist: {unicorn.__version__}")'
296301
297302
- uses: actions/upload-artifact@v4
298303
with:

‎tests/regress/arm_movr12_hang.py

+1-3
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,11 @@
1-
import platform
21
import regress
3-
import unittest
42
from unicorn import *
53
from unicorn.arm_const import *
64

75

86
class MovHang(regress.RegressTest):
97

10-
@unittest.skipIf(platform.machine().lower() == 'aarch64', reason='TO BE CHECKED!')
8+
# NOTE: This test was failing when workflow was using ubuntu-latest + qemu. Fixed once switched to native arm runner
119
def runTest(self):
1210
uc = Uc(UC_ARCH_ARM, UC_MODE_ARM)
1311
uc.mem_map(0x1000, 0x1000)

‎tests/regress/core_ctl.py

+1-2
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ def test_mode(self):
4444
self.assertEqual(UC_CPU_ARM_CORTEX_M0, uc.ctl_get_cpu_model())
4545

4646
@unittest.skipIf(sys.version_info < (3, 7), reason="requires python3.7 or higher")
47-
@unittest.skipIf(sys.platform == 'win32' or platform.machine().lower() not in ('x86_64', 'arm64'), 'TO BE CHECKED!')
47+
@unittest.skip('TO BE CHECKED!')
4848
def test_page_size(self):
4949
SIZE_4KB = 4 * 1024 ** 1
5050
SIZE_2MB = 2 * 1024 ** 2
@@ -78,7 +78,6 @@ def test_page_size(self):
7878
# are we still with the valid value?
7979
self.assertEqual(SIZE_2MB, uc.ctl_get_page_size())
8080

81-
@unittest.skipIf(platform.machine().lower() == 'aarch64', reason='TO BE CHECKED!')
8281
def test_timeout(self):
8382
MILLIS_1S = 1000
8483

‎tests/regress/init.py

+2-1
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ class Init(regress.RegressTest):
1414
def init_unicorn(self, ip, sp, counter):
1515
regress.logger.debug("[+] Emulating IP: %x SP: %x - Counter: %x" % (ip, sp, counter))
1616
self.emulator = Uc(UC_ARCH_X86, UC_MODE_64)
17+
self.emulator.ctl_set_tlb_mode(UC_TLB_VIRTUAL)
1718
self.emulator.mem_map(0x1000000, 2 * 1024 * 1024)
1819
self.emulator.mem_write(0x1000000, b"\x90")
1920
self.emulator.mem_map(0x8000000, 8 * 1024 * 1024)
@@ -55,7 +56,7 @@ def hook_mem_fetch_unmapped(self, uc, access, address, size, value, user_data):
5556
return True
5657

5758
@unittest.skipIf(sys.version_info < (3, 7), reason="requires python3.7 or higher")
58-
@unittest.skipIf(sys.platform == 'win32' or platform.machine().lower() not in ('x86_64', 'arm64'), 'TO BE CHECKED!')
59+
@unittest.skipIf(sys.platform == 'win32', 'TO BE CHECKED!')
5960
def runTest(self):
6061
ips = range(0x1000000, 0x1001000)
6162
sps = range(0x8000000, 0x8001000)

‎tests/regress/potential_memory_leak.py

-1
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,6 @@
1616
class MemoryLeak(regress.RegressTest):
1717

1818
@unittest.skipIf(sys.platform == 'win32', reason='Test for Unix only')
19-
@unittest.skipIf(platform.machine().lower() == 'aarch64', reason='TO BE CHECKED!')
2019
def test(self):
2120
if platform.system() == "Darwin":
2221
rusage_multiplier = 1

‎tests/regress/translator_buffer.py

+3-2
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
# By Mariano Graziano
22

3-
import platform
43
import regress
54
import struct
5+
import sys
66
import unittest
77
from unicorn import *
88
from unicorn.x86_const import *
@@ -17,6 +17,7 @@ def __page_aligned(address):
1717
self.unicorn_stack = stack
1818

1919
self.mu = Uc(UC_ARCH_X86, UC_MODE_64)
20+
self.mu.ctl_set_tlb_mode(UC_TLB_VIRTUAL)
2021

2122
regress.logger.debug("mapping code : %#x", __page_aligned(code))
2223
regress.logger.debug("mapping stack : %#x", __page_aligned(stack))
@@ -75,7 +76,7 @@ def init_unicorn(self, ip, sp, magic):
7576

7677
emu.emu(1)
7778

78-
@unittest.skipIf(platform.machine().lower() == 'aarch64', reason='TO BE CHECKED!')
79+
@unittest.skipIf(sys.version_info < (3, 7), reason="requires python3.7 or higher")
7980
def runTest(self):
8081
ip_base = 0x000fffff816a0000 # was: 0xffffffff816a0000
8182
sp_base = 0x000f88001b800000 # was: 0xffff88001b800000

0 commit comments

Comments
 (0)