-
-
Notifications
You must be signed in to change notification settings - Fork 33.9k
gh-136728: Refactor build.yml CI config and multissltests.py #136729
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
a104f8a
3fcbe0d
5d8ec9a
991c6b2
7b51499
6638127
4e0a8ca
1a90e0c
1fcb49f
dd969fb
1edce1c
3343120
6536fab
f783cc3
c8b80f5
25c294b
20aeb3a
450e836
c2202a7
cbe0cb7
cc48bf0
50ecd6b
fc2f0fe
9af7a20
57937a8
4ee6929
e8e044e
d4dc3dd
1e17ccd
7c44f37
84b7e69
3509fa5
7342890
305aff0
594a463
cf6758f
86d9045
8d46f96
579c5b4
8611f74
b9a4806
59ede34
ea3fd78
888d101
de22e71
d3d4cf9
b3f2d80
a1c6308
9d92ac1
5436289
5d1e78f
57d5699
1af21ea
f5e11fa
9976c2b
7726119
00e24b8
3a728e5
84fcdbd
61ee048
23abbf1
5225635
836b281
3ccc76f
3ca1f2a
c3bfe5d
fa9a425
0efbad6
c3febba
713684d
daa9aa4
f37f57d
6cb245d
b6b0e14
79c03ac
ef834de
23ad9c5
0aedf2f
7e3a5a7
aa8a43d
04899b8
469fe33
96ab379
3c4429f
c521597
7f6c16a
2d9f4e3
422ca07
1fb8e0e
d00d39f
faa2604
5d13335
513ae17
e5ad7b7
6b9a6c6
61fc72a
9712dc1
18f3c59
315f474
09ce592
8a2deea
61f2ad9
08a17ed
864c598
ef6f92a
0417dab
9825832
f7a03bb
136f6d8
b538c28
abdbe0b
e7c542d
6d05e55
2c39b9d
27434c6
3c56f9e
12283f6
ef3b882
6c53af1
9609574
6116d70
e6bfe4d
c99f766
68fcb95
bfac54d
4f9a8d0
1569275
240a6c3
7a572d9
04ace41
12d363b
d6f77e6
4d21297
7dae107
efb4e6c
dd750b3
71119a1
d745b60
841b748
d9c1235
54f1ed0
05406b2
faa3dc7
90c44bc
7b0a372
8735daf
e79c9b7
74bb3ca
51227b6
ff7d1ce
df35534
98e55d7
0a5c04a
4fb6a31
b866a1c
8565ddd
a1eedae
b3e4a34
b2827de
51a56a3
d043949
9a3263f
bfc3d8d
228d955
f11f5eb
67d3d03
6c9f7b4
c07e5ec
f3e069a
efaa56f
5462002
49c3b0a
8cf5c4d
cea2d24
aeb3403
e2f15ae
b54a1d2
dfeefbe
68a01f9
af9f783
fd6d41b
234a15d
dcdb23f
6d54b6a
39a2bcf
a9ca49d
a4086d7
ba10100
6d6c7ed
c696f33
61e0366
b852236
66bca38
499d3a8
1932127
e0fb278
9525911
78e868f
e7f5ffa
e2f0160
ce6bae9
aa8578d
718c15f
75d73c3
03e6457
d1282ef
515ae40
e22b685
23b9377
9633f95
620a5b9
265381b
9d13ca9
548526b
5f28aa2
dbd10a6
f3759d2
bd83a57
43c7658
7f50a5f
82f1154
66414c3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -243,78 +243,32 @@ jobs: | |
| free-threading: ${{ matrix.free-threading }} | ||
| os: ${{ matrix.os }} | ||
|
|
||
| build-ubuntu-ssltests-openssl: | ||
| name: 'Ubuntu SSL tests with OpenSSL' | ||
| runs-on: ${{ matrix.os }} | ||
| build-ubuntu-ssltests: | ||
| name: 'Ubuntu SSL tests' | ||
| runs-on: ubuntu-24.04 | ||
| timeout-minutes: 60 | ||
| needs: build-context | ||
| if: needs.build-context.outputs.run-ubuntu == 'true' | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| os: [ubuntu-24.04] | ||
| # Keep 1.1.1w in our list despite it being upstream EOL and otherwise | ||
| # unsupported as it most resembles other 1.1.1-work-a-like ssl APIs | ||
| # supported by important vendors such as AWS-LC. | ||
| openssl_ver: [1.1.1w, 3.0.18, 3.2.6, 3.3.5, 3.4.3, 3.5.4] | ||
| include: | ||
| # Keep 1.1.1w in our list despite it being upstream EOL and otherwise | ||
| # unsupported as it most resembles other 1.1.1-work-a-like ssl APIs | ||
| # supported by important vendors such as AWS-LC. | ||
| - { ssl: openssl, ssl_ver: 1.1.1w } | ||
| - { ssl: openssl, ssl_ver: 3.0.18 } | ||
| - { ssl: openssl, ssl_ver: 3.2.6 } | ||
| - { ssl: openssl, ssl_ver: 3.3.5 } | ||
| - { ssl: openssl, ssl_ver: 3.4.3 } | ||
| - { ssl: openssl, ssl_ver: 3.5.4 } | ||
| - { ssl: awslc, ssl_ver: 1.55.0 } | ||
| # See Tools/ssl/make_ssl_data.py for notes on adding a new version | ||
| env: | ||
| OPENSSL_VER: ${{ matrix.openssl_ver }} | ||
| MULTISSL_DIR: ${{ github.workspace }}/multissl | ||
| OPENSSL_DIR: ${{ github.workspace }}/multissl/openssl/${{ matrix.openssl_ver }} | ||
| LD_LIBRARY_PATH: ${{ github.workspace }}/multissl/openssl/${{ matrix.openssl_ver }}/lib | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| persist-credentials: false | ||
| - name: Runner image version | ||
| run: echo "IMAGE_OS_VERSION=${ImageOS}-${ImageVersion}" >> "$GITHUB_ENV" | ||
| - name: Register gcc problem matcher | ||
| run: echo "::add-matcher::.github/problem-matchers/gcc.json" | ||
| - name: Install dependencies | ||
| run: sudo ./.github/workflows/posix-deps-apt.sh | ||
| - name: Configure OpenSSL env vars | ||
| run: | | ||
| echo "MULTISSL_DIR=${GITHUB_WORKSPACE}/multissl" >> "$GITHUB_ENV" | ||
| echo "OPENSSL_DIR=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}" >> "$GITHUB_ENV" | ||
| echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}/lib" >> "$GITHUB_ENV" | ||
| - name: 'Restore OpenSSL build' | ||
| id: cache-openssl | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: ./multissl/openssl/${{ env.OPENSSL_VER }} | ||
| key: ${{ matrix.os }}-multissl-openssl-${{ env.OPENSSL_VER }} | ||
| - name: Install OpenSSL | ||
| if: steps.cache-openssl.outputs.cache-hit != 'true' | ||
| run: python3 Tools/ssl/multissltests.py --steps=library --base-directory "$MULTISSL_DIR" --openssl "$OPENSSL_VER" --system Linux | ||
| - name: Add ccache to PATH | ||
| run: | | ||
| echo "PATH=/usr/lib/ccache:$PATH" >> "$GITHUB_ENV" | ||
| - name: Configure CPython | ||
| run: ./configure CFLAGS="-fdiagnostics-format=json" --config-cache --enable-slower-safety --with-pydebug --with-openssl="$OPENSSL_DIR" | ||
| - name: Build CPython | ||
| run: make -j4 | ||
| - name: Display build info | ||
| run: make pythoninfo | ||
| - name: SSL tests | ||
| run: ./python Lib/test/ssltests.py | ||
|
|
||
| build-ubuntu-ssltests-awslc: | ||
| name: 'Ubuntu SSL tests with AWS-LC' | ||
| runs-on: ${{ matrix.os }} | ||
| timeout-minutes: 60 | ||
| needs: build-context | ||
| if: needs.build-context.outputs.run-ubuntu == 'true' | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| os: [ubuntu-24.04] | ||
| awslc_ver: [1.55.0] | ||
| env: | ||
| AWSLC_VER: ${{ matrix.awslc_ver}} | ||
| SSL_VER: ${{ matrix.ssl_ver }} | ||
| MULTISSL_DIR: ${{ github.workspace }}/multissl | ||
| OPENSSL_DIR: ${{ github.workspace }}/multissl/aws-lc/${{ matrix.awslc_ver }} | ||
| LD_LIBRARY_PATH: ${{ github.workspace }}/multissl/aws-lc/${{ matrix.awslc_ver }}/lib | ||
| SSL_DIR: ${{ github.workspace }}/multissl/${{ matrix.ssl }}/${{ matrix.ssl_ver }} | ||
| LD_LIBRARY_PATH: ${{ github.workspace }}/multissl/${{ matrix.ssl }}/${{ matrix.ssl_ver }}/lib | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
|
|
@@ -328,39 +282,35 @@ jobs: | |
| - name: Configure SSL lib env vars | ||
| run: | | ||
| echo "MULTISSL_DIR=${GITHUB_WORKSPACE}/multissl" >> "$GITHUB_ENV" | ||
| echo "OPENSSL_DIR=${GITHUB_WORKSPACE}/multissl/aws-lc/${AWSLC_VER}" >> "$GITHUB_ENV" | ||
| echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/aws-lc/${AWSLC_VER}/lib" >> "$GITHUB_ENV" | ||
| - name: 'Restore AWS-LC build' | ||
| id: cache-aws-lc | ||
| echo "SSL_DIR=${GITHUB_WORKSPACE}/multissl/${{ matrix.ssl }}/${SSL_VER}" >> "$GITHUB_ENV" | ||
| echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/${{ matrix.ssl }}/${SSL_VER}/lib" >> "$GITHUB_ENV" | ||
| - name: 'Restore SSL build' | ||
| id: cache-ssl | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: ./multissl/aws-lc/${{ matrix.awslc_ver }} | ||
| key: ${{ matrix.os }}-multissl-aws-lc-${{ matrix.awslc_ver }} | ||
| - name: Install AWS-LC | ||
| if: steps.cache-aws-lc.outputs.cache-hit != 'true' | ||
| path: ./multissl/${{ env.SSL }}/${{ env.SSL_VER }} | ||
| key: ${{ env.IMAGE_OS_VERSION }}-multissl-${{ env.SSL }}-${{ env.SSL_VER }} | ||
| - name: Install SSL | ||
| if: steps.cache-ssl.outputs.cache-hit != 'true' | ||
| run: | | ||
| python3 Tools/ssl/multissltests.py \ | ||
| --steps=library \ | ||
| --base-directory "$MULTISSL_DIR" \ | ||
| --awslc ${{ matrix.awslc_ver }} \ | ||
| --system Linux | ||
| python3 Tools/ssl/multissltests.py --steps=library --base-directory "$MULTISSL_DIR" --system Linux --ssl ${{ matrix.ssl }} --ssl-versions ${{ matrix.ssl_ver }} | ||
hugovk marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - name: Add ccache to PATH | ||
| run: | | ||
| echo "PATH=/usr/lib/ccache:$PATH" >> "$GITHUB_ENV" | ||
| - name: Configure CPython | ||
| run: | | ||
| ./configure CFLAGS="-fdiagnostics-format=json" \ | ||
| --config-cache \ | ||
| --enable-slower-safety \ | ||
| --with-pydebug \ | ||
| --with-openssl="$OPENSSL_DIR" \ | ||
| --with-builtin-hashlib-hashes=blake2 \ | ||
| --with-ssl-default-suites=openssl | ||
| CMD=(./configure CFLAGS="-fdiagnostics-format=json" --config-cache --enable-slower-safety --with-pydebug --with-openssl="$SSL_DIR") | ||
| if [ "${{ matrix.ssl }}" = "openssl" ]; then | ||
| "${CMD[@]}" | ||
| else | ||
| "${CMD[@]}" --with-builtin-hashlib-hashes=blake2 --with-ssl-default-suites=openssl | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we populate a variable called "CONFIGURE_FLAGS" instead and use a switch/case so that we exactly match the SSL libname. It will become easier in the future if we add BoringSSL or LibreSSL tests.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. At this point I'd almost suggest using a small Python script to calculate flags. We should avoid complex shell logic in workflow files as much as possible, I don't want to read a bash switch-case in YAML! An if-statement is simpler to understand here for me, though the
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh yeah, a small Python script is also fine; here what matters to me is to be able to easily add a new library without having 3km long of options. As for |
||
| fi | ||
| - name: Build CPython | ||
| run: make -j | ||
| run: make -j4 | ||
picnixz marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - name: Display build info | ||
| run: make pythoninfo | ||
| - name: Verify python is linked to AWS-LC | ||
| if: matrix.ssl == 'aws-lc' | ||
|
Comment on lines
312
to
+313
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We should, if possible, do it for OpenSSL as well. |
||
| run: ./python -c 'import ssl; print(ssl.OPENSSL_VERSION)' | grep AWS-LC | ||
| - name: SSL tests | ||
| run: ./python Lib/test/ssltests.py | ||
|
|
@@ -446,7 +396,7 @@ jobs: | |
| key: ${{ runner.os }}-multissl-openssl-${{ env.OPENSSL_VER }} | ||
| - name: Install OpenSSL | ||
| if: steps.cache-openssl.outputs.cache-hit != 'true' | ||
| run: python3 Tools/ssl/multissltests.py --steps=library --base-directory "$MULTISSL_DIR" --openssl "$OPENSSL_VER" --system Linux | ||
| run: python3 Tools/ssl/multissltests.py --steps=library --base-directory "$MULTISSL_DIR" --ssl 'openssl' --ssl-versions "$OPENSSL_VER" --system Linux | ||
| - name: Add ccache to PATH | ||
| run: | | ||
| echo "PATH=/usr/lib/ccache:$PATH" >> "$GITHUB_ENV" | ||
|
|
@@ -564,7 +514,7 @@ jobs: | |
| key: ${{ matrix.os }}-multissl-openssl-${{ env.OPENSSL_VER }} | ||
| - name: Install OpenSSL | ||
| if: steps.cache-openssl.outputs.cache-hit != 'true' | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. With |
||
| run: python3 Tools/ssl/multissltests.py --steps=library --base-directory "$MULTISSL_DIR" --openssl "$OPENSSL_VER" --system Linux | ||
| run: python3 Tools/ssl/multissltests.py --steps=library --base-directory "$MULTISSL_DIR" --ssl 'openssl' --ssl-versions "$OPENSSL_VER" --system Linux | ||
| - name: Add ccache to PATH | ||
| run: | | ||
| echo "PATH=/usr/lib/ccache:$PATH" >> "$GITHUB_ENV" | ||
|
|
@@ -689,9 +639,7 @@ jobs: | |
| - build-windows-msi | ||
| - build-macos | ||
| - build-ubuntu | ||
| - build-ubuntu-ssltests-awslc | ||
| - build-ubuntu-ssltests-openssl | ||
| - build-android | ||
| - build-ubuntu-ssltests | ||
| - build-ios | ||
| - build-wasi | ||
| - test-hypothesis | ||
|
|
@@ -706,9 +654,9 @@ jobs: | |
| uses: re-actors/alls-green@05ac9388f0aebcb5727afa17fcccfecd6f8ec5fe | ||
| with: | ||
| allowed-failures: >- | ||
| build-android, | ||
| build-windows-msi, | ||
| build-ubuntu-ssltests-awslc, | ||
| build-ubuntu-ssltests-openssl, | ||
| build-ubuntu-ssltests, | ||
| test-hypothesis, | ||
| cifuzz, | ||
| allowed-skips: >- | ||
|
|
@@ -728,8 +676,8 @@ jobs: | |
| !fromJSON(needs.build-context.outputs.run-ubuntu) | ||
| && ' | ||
| build-ubuntu, | ||
| build-ubuntu-ssltests-awslc, | ||
| build-ubuntu-ssltests-openssl, | ||
| build-ubuntu-ssltests, | ||
| build-wasi, | ||
| test-hypothesis, | ||
| build-asan, | ||
| build-san, | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.