🛠️ Fixing GitLab CI/CD Hangs: Building Docker Images for Lambda Runtime with MSSQL and ODBC

⚠️ Problem Definition

📌 Precise description of the issue

The GitLab CI/CD pipeline was consistently hanging during the Docker image build process. Specifically, the build would stall while attempting to retrieve the Microsoft GPG key required for verifying the msodbcsql17 package from the Microsoft repository.

🚨 Error messages and symptoms

The pipeline would reach a point where it displayed:

#6 56.30 Loaded plugins: ovl
#6 56.37 Retrieving key from https://packages.microsoft.com/keys/microsoft.asc
#6 56.38 Importing GPG key 0xBE1229CF:
#6 56.38  Userid     : "Microsoft (Release signing) <gpgsecurity@microsoft.com>"
#6 56.38  Fingerprint: bc52 8686 b50d 79e3 39d3 721c eb3e 94ad be12 29cf
#6 56.38  From       : https://packages.microsoft.com/keys/microsoft.asc
#6 CANCELED
ERROR: failed to solve: Canceled: context canceled

The build process would then timeout, leading to pipeline failures.

📉 Impact of the issue

This issue prevented the successful deployment of AWS Lambda functions that required MSSQL and ODBC dependencies. It disrupted the CI/CD workflow, causing delays and requiring manual intervention.

🔍 Root Cause Analysis (Optional)

💡 Explanation of why the issue occurred

The direct curl method used to retrieve the Microsoft GPG key was unreliable in the GitLab CI/CD environment. Network inconsistencies, certificate verification issues, or temporary repository problems led to the key retrieval process hanging.

🔧 Solution

➡️ Step-by-step instructions to fix the issue

Install wget and gnupg: Add these packages to your Dockerfile’s yum install command.
Download and Import the GPG Key: Replace the direct curl command with wget and gnupg to download and import the key more reliably.
Install the Key to the Correct Location: Use the install command to place the GPG key in /etc/pki/rpm-gpg/ with the appropriate permissions.
Configure the Microsoft Repository: Maintain the curl command to configure the Microsoft repository.
Install msodbcsql17: Continue with the installation of msodbcsql17.

📜 Code snippets, commands, or configuration changes

RUN yum update -y && \
    yum install -y \
        gcc gcc-c++ make unixODBC-devel unzip python3-devel curl wget gnupg && \
    wget -qO - https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg && \
    install -o root -g root -m 644 microsoft.gpg /etc/pki/rpm-gpg/ && \
    curl -sSL https://packages.microsoft.com/config/rhel/7/prod.repo -o /etc/yum.repos.d/msprod.repo && \
    ACCEPT_EULA=Y yum install -y msodbcsql17 && \
    yum clean all && \
    rm -rf /var/cache/yum /var/tmp/* /tmp/*

🖼️ Screenshots or visual aids

gitlab build failure due to microsoft packages issue for msodbcsql17

✅ Verification

👍 Steps to confirm the issue is resolved

Trigger a new build in your GitLab CI/CD pipeline.
Monitor the build logs to ensure the GPG key is successfully retrieved and imported.
Verify that the msodbcsql17 package is installed without any hangs or timeouts.
Confirm that the Docker image is built successfully.

Successfully installed, which look like:

🛡️ Preventative Measures

🚫 How to prevent the issue from recurring

Use wget and gnupg for GPG Key Retrieval: Avoid relying on direct curl calls for GPG key retrieval in CI/CD environments.
Explicit Key Placement: Always place GPG keys in the correct directory with the required permissions.
Test in CI/CD Environment: Thoroughly test Docker builds in the CI/CD environment to identify potential issues early.
Monitor Network Connectivity: Ensure that your CI/CD runners have stable network connectivity.

🎉 Conclusion

🏁 Summary of the solution and its benefits

By replacing the direct curl method with wget and gnupg for GPG key retrieval, the GitLab CI/CD pipeline now successfully builds Docker images for AWS Lambda with MSSQL and ODBC support. This solution ensures reliable package installation, eliminates build timeouts, and improves the overall deployment process. The robust approach to key management also serves as a valuable lesson for building reliable CI/CD pipelines.