Breaking Barriers: Jenkins to CircleCI for seamless E2E test execution

In software development, the efficiency of your CI/CD pipeline can make or break your productivity. Previously, our team was maintaining both Jenkins (On-premises) and CircleCI (Cloud) for separate parts of the pipeline. This was not only time-consuming but also brought multiple challenges; if a new change was required, it needed to be implemented in both systems, leading to unnecessary duplication of efforts. We were straddling two different worlds — with our Dev team on CircleCI for building, unit testing, and deploying the app and the QA team using Jenkins to run automation End-to-End (E2E) tests. More than just creating an additional layer of complexity, this divided setup was holding back our potential to truly streamline and optimize our workflows.

Driven by this problem, we embarked on a journey to unify our CI/CD pipelines solely under CircleCI. This transition represented more than just a shift in tools; it was a quest to establish a consolidated, efficient, and upheld environment that catered to our end-to-end development process — from building and testing to deployment. This blog post is a recount of our exciting migration journey from Jenkins to CircleCI, the roadmap we followed, the challenges we encountered, the solutions we crafted, and what we learned along the way.

The Migration Objective

Goals for Migration

Our primary intent behind the migration was to improve our CI/CD process by migrating our E2E test execution jobs from Jenkins (On-premises) to CircleCI (Cloud). This task included migrating both the iOS and Android executions. Our final goal was to have a CircleCI setup that could run tests, build the app, and handle environment setup in a way that avoided complexities we faced with Jenkins.

Challenges with the Jenkins Setup

Our decision to migrate was largely propelled by numerous significant challenges our team persistently encountered with the Jenkins setup:

• Frequent Memory Issues: One recurring impediment was that Jenkins was constantly running out of memory. This inadequacy of memory would obstruct the triggering of our tests and on multiple occasions, we found ourselves reaching out to the platform team for assistance.
• Deleting Old Reports: Tied to the memory problem, we often had to resort to deleting old reports to free up some space. This workaround was not only a drain on resources but also led to the loss of potentially valuable historical data.
• Dependency Version Mismatch: Another regular disruption we faced was when the Ruby version in Jenkins didn’t align with what was required, causing app builds to fail. These inconsistencies generated unnecessary delays in our workflow and added a layer of frustration.
• Physical Server Limitations: Being physically linked to an on premises machine, we found ourselves grappling with issues such as network or electricity outages. These setbacks needed manual intervention to fix, which took up valuable time and delayed testing operations.
• Duplication of Effort: When a developer revised the build process in CircleCI, the QA Team had to duplicate the modifications in Jenkins. This magnified the workload and reduced the overall efficiency.
• Hardware and Software Limitations: The physical machine, had limitations in terms of hardware resources, data retention, and susceptibility to physical damage. Furthermore, the variances between the operating systems and libraries used in CircleCI and Jenkins sporadically sparked unpredictable issues in the app build process.

These formidable challenges prompted us to explore the potential of CircleCI, leading us on the path to migrate our testing processes into a more reliable, efficient, and robust environment.

Assumptions for CircleCI Setup

With the goal to migrate to CircleCI, we made a few assumptions:

• We decided to use a specific Python version installed through pyenv for consistency.
• The default resource class provided by CircleCI would be adequate for our needs.
• The test execution command would be the same for both iOS and Android runs.

The Migration Journey

The migration from Jenkins to CircleCI encompassed a series of sequential steps. Here is an overview of the steps we executed:

Setting Up Dependencies Before Tests

Before we could run any tests on CircleCI, we had to set up the executor with the correct dependencies. This task involved installing a specific Python version through pyenv, configuring the timezone, and setting up necessary environment variables and test data. Considering this task upfront ensured that necessary adjustments were made, and we could focus on running the tests subsequently.

Running Individual and Whole Suite Tests

Once the setup was completed and the environment was correctly configured, we started with executing a single test file on both iOS and Android. This was done to ensure that the test mechanisms functioned correctly. After successful execution of individual tests, we then moved to execute the whole suite of tests and confirmed successful execution.

The migration journey was not without stumbling blocks, and we received our fair share of challenges. However, working systematically and incrementally allowed us to handle these effectively and complete the migration process successfully.

Challenges Faced and Solutions Found

The process of migration was not without its challenges. Each roadblock was a learning opportunity that helped us refine our migration process. Let’s discuss some of the challenges we encountered and the solutions we implemented:

• Incompatibility with Python Version: Our automation framework relied on a specific Python version, but the iOS and Android executors came with different versions, causing compatibility issues. We overcame this by installing and configuring pyenv as a Python version manager, ensuring consistent use of the correct Python version.
• Pyenv Installation and Configuration Difficulty: Configuring pyenv needed certain initialization variables to be included in the bash profile. Missing these caused errors. To resolve this, we determined the type of shell used, updated the bash profile and used a separate step to run pyenv, which allowed for the proper initialization of pyenv.
• Default Python Version Issues: There were difficulties replacing the default python version, even after running the pyenv global command. We solved this by running the commands on separate steps, which allowed the new shell to load the Python version defined by the pyenv global command.
• Timezone Configuration: Test execution failed due to errors in timezone configuration. We resolved this by recreating the symbolic link in the executor with a valid timezone. This allowed us to correctly calculate dates based on the local timezone of the host machine.
• Issues with Allure Report Orb: We decided to use a CircleCI’s third-party orb for generating our allure test result reports, but it wasn’t flexible enough and was causing us problems. So instead of using this orb, we opted to directly code our own allure commands which not only solved the issues but also allowed us to customize as we desired.
• Session Grouping: Test executions weren’t getting grouped and labeled as expected. After some debugging we found out that this was happening because we missed migrating one of the necessary configurations from Jenkins. Once we migrated the missing configuration, test results got properly labeled and grouped as expected.

By successfully surmounting these challenges, we not only completed the migration process as planned, but also gained invaluable insights into both Jenkins and CircleCI, which further enriched our experience.

Showcasing the Final CircleCI Config File

Our intention was to create a robust, flexible, and maintainable CircleCI configuration that supported our testing setup needs. The final config.yml file has several features to be proud of:

• Organized Structure: The file is neatly organized with clean demarcations for version, parameters, commands, jobs, and workflows. This structure makes it easy to follow the flow and understand the purpose of each section. Here’s part of it for reference:

parameters:
  test-automation-trigger:
    type: boolean
    default: false
  test-automation-platform:
    type: enum
    default: both
    enum:
      - ios
      - android
      - both
      .....
commands:
  run-test-automation-command:
    parameters:
      platform:
        type: enum
        default: "ios"
        enum: ["android", "ios"]
      device:
        type: string
        .....
jobs:
  android-applicant-app-automation-test-job-trigger:
    executor: android
    resource_class: medium
    parameters:
      ignore-command-failure:
        type: boolean
        default: false
        .....
workflows:
  trigger-applicant-app-nightly-automation-android:
    jobs:
      - build-mobile-test-app:
          flavor: automation
          platform: android
          .....
  trigger-adhoc-applicant-app-automation-run-android:
    when:
      and:
        - equal: [<< pipeline.parameters.test-automation-trigger >>, true]
        .....

• Parameter Flexibility: The parameters section contains multiple user-configurable values. They determine various aspects of the test execution setup, such as the test tag to run, the platform to run the test on, or even the ability use a custom backend environment for the execution. This level of customization gives us the flexibility needed in a dynamic development environment.
• Commands Clarity: The commands section details the different operations the CI/CD pipeline would perform. It outlines how the environment is setup, the dependencies installed, how the tests get executed, etc., providing transparency and control over the execution process working in our favor.
• Comprehensive Jobs and Workflow Configuration: The jobs and workflows sections further optimize the CI/CD process by defining test execution jobs based on parameters and conditions, organizing tasks in efficient workflows, and correctly managing dependencies between tasks. The workflows are easily adjustable, making this a truly flexible and dynamic CI/CD setup.

This was undoubtedly an upgrade that has successfully given us a more efficient and streamlined CI/CD process flowing on CircleCI.

Comparing the Final Result with the Original Plan

When we embarked on this migration journey, the primary goal was to transition our test execution jobs from Jenkins to CircleCI. The resulting CircleCI config file, however, did much more than that. It transformed our original plan into a dynamic, flexible, and robust CI/CD pipeline.

The original plan catered to the immediate need of executing test jobs while alleviating the major pain points that compelled us to migrate. It envisioned a safer and more controlled testing environment, flexible enough to cater to the diverse testing requirements of Android and iOS applications.

In essence, the final result not only aligns precisely with the planned vision but also enhances the setup with CircleCI’s native capabilities, optimizing every aspect of the process.

Conclusion

Our shift from Jenkins to CircleCI was more than a tool swap; it was a transformation. We streamlined our processes, boosted our testing capabilities, and future-proofed our development cycles. Along the way, we tackled unexpected hurdles and innovated solutions that led to robust enhancements.

Now, we have an efficient, robust environment that simplifies complex workflows and refines control over builds and tests. But we’re not stopping here. We aim to broaden CI/CD use by equipping other stakeholders like the development and product teams at Instawork with our pipeline. We want to democratize testing, receive quicker feedback, and improve product quality through collaborative development.

Thanks for following our journey to the very end. Your interest is invaluable, and we hope you’ve found insights to apply in your context. We’re looking forward to sharing more of our learnings in the future.