Parallel Execution with Selenium Grid and Docker

In my opinion Parallel Execution is one of the crucial features we could have in an Automation Framework, why?… because it enables the way we are going to execute our tests, and the velocity on getting results on the Test Suite.

Selenium Grid

Why using selenium grid?, well first of all it is because the team that is developing and maintaining it, is the same team developing the Selenium automation library. Another important point is that, we can use Docker to spin up the Selenium Grid and be ready to send commands, so lets see how we can do this.

Docker Compose

Docker compose now ships with Docker engine and you no longer need to install it separately, if you don’t know how to install docker in your Operative System, you can visit this web site Docker Install.

Docker Compose Files

Compose uses YAML files to define micro services applications. The default name is docker-compose.yml or you can use the -f flag to specify custom file names.

For our specific case, we are going to use the docker compose file provided by the Selenium Grid team, we will modify it a bit to meet our requirements but overall you can visit the Selenium Grid GH Repository. The file we are going to use setups a Full Selenium Grid, that contains a Selenium Distributor and Selenium Router, these will help us on having a load balancing process for our tests.

Create a file named docker-compose.yml at the root of your project and add the next code lines.

version: "3"
services:
  selenium-event-bus:
    image: selenium/event-bus:4.28.1-20250202
    container_name: selenium-event-bus
    ports:
      - "4442:4442"
      - "4443:4443"
      - "5557:5557"

  selenium-sessions:
    image: selenium/sessions:4.28.1-20250202
    container_name: selenium-sessions
    ports:
      - "5556:5556"
    depends_on:
      - selenium-event-bus
    environment:
      - SE_NODE_OVERRIDE_MAX_SESSIONS=true
      - SE_NODE_MAX_SESSIONS=5
      - SE_EVENT_BUS_HOST=selenium-event-bus
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443

  selenium-session-queue:
    image: selenium/session-queue:4.28.1-20250202
    container_name: selenium-session-queue
    ports:
      - "5559:5559"

  selenium-distributor:
    image: selenium/distributor:4.28.1-20250202
    container_name: selenium-distributor
    ports:
      - "5553:5553"
    depends_on:
      - selenium-event-bus
      - selenium-sessions
      - selenium-session-queue
    environment:
      - SE_EVENT_BUS_HOST=selenium-event-bus
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
      - SE_SESSIONS_MAP_HOST=selenium-sessions
      - SE_SESSIONS_MAP_PORT=5556
      - SE_SESSION_QUEUE_HOST=selenium-session-queue
      - SE_SESSION_QUEUE_PORT=5559

  selenium-router:
    image: selenium/router:4.28.1-20250202
    container_name: selenium-router
    ports:
      - "4444:4444"
    depends_on:
      - selenium-distributor
      - selenium-sessions
      - selenium-session-queue
    environment:
      - SE_DISTRIBUTOR_HOST=selenium-distributor
      - SE_DISTRIBUTOR_PORT=5553
      - SE_SESSIONS_MAP_HOST=selenium-sessions
      - SE_SESSIONS_MAP_PORT=5556
      - SE_SESSION_QUEUE_HOST=selenium-session-queue
      - SE_SESSION_QUEUE_PORT=5559

  chrome:
    image: selenium/node-chrome:4.28.1-20250202
    shm_size: 2gb
    depends_on:
      - selenium-event-bus
    environment:
      - SE_NODE_OVERRIDE_MAX_SESSIONS=true
      - SE_NODE_MAX_SESSIONS=5
      - SE_EVENT_BUS_HOST=selenium-event-bus
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443

  edge:
    image: selenium/node-edge:4.28.1-20250202
    shm_size: 2gb
    depends_on:
      - selenium-event-bus
    environment:
      - SE_EVENT_BUS_HOST=selenium-event-bus
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443

  firefox:
    image: selenium/node-firefox:4.28.1-20250202
    shm_size: 2gb
    depends_on:
      - selenium-event-bus
    environment:
      - SE_EVENT_BUS_HOST=selenium-event-bus
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443

We will use 3 different browsers: chrome, firefox and ms edge.

We would be adding for now 2 options to the chrome setup, these options are related to the Max number of sessions we can concurrently execute on chrome browser. You do not need to worry about modifying the above code, those instructions are already set.

- SE_NODE_OVERRIDE_MAX_SESSIONS=true
- SE_NODE_MAX_SESSIONS=5

Those two lines of code will tell the Grid that we are going to enable the MAX SESSION option, and that we need that 5 is the number.

Deploying the Selenium Grid with Compose

Now that we have our setup in place it’s time to deploy the grid, to do so, it’s super simple, we will use Docker Compose commands to deploy it.

docker compose up is the most common way to bring up a Compose app. It pulls all the required images, creates all the networks and volumes, and starts the containers.

docker compose down will bring the Compose app down, all the containers and networks are deleted by default.

By using the docker compose upcommand we can now have our Selenium Grid deployed, we can access it by navigating to Selenium Grid .

This uses the endpoint http:\\localhost:4444

We are ready to run our tests! 😎

Adding Executor Command line option

In the previous blog posts we added a command line option to select the browser on the go (Chrome or Firefox), Selecting a web browser from command line, We will add another option to select the type of Test Execution we want, like local or grid.

We are going to start by adding a new class in the config.py file, this CLI option will request you to pass 2 possible options: local or grid.

class Config_execution:
    def __init__(self, execution):

        SUPPORTED_EXECUTORS = ['local', 'grid']

        if execution == None or execution.lower() not in SUPPORTED_EXECUTORS:
            raise Exception(f'{execution} please select an option from the Supported Executors with the option --execution (supported browsers: {SUPPORTED_EXECUTORS})')


        self.execution_type = {
                'grid': 'grid',
                'local': 'local'
        }[execution]

Now that we have the new CLI option added we can now work on adding the fixture to execute our test locally or on the Selenium Grid.

Execution Fixture

We will add 2 new fixtures to our conftest.py file, those are going to help us to read the option from the --executor CLI option, and store it.

@fixture(scope="function")
def execution_type(request):
    return request.config.getoption("--execution")

Now the one that is going to setup our webdriver as to be running locally or remotelly:

@fixture(scope='function')
def execution(execution_type, browser_type):
    cfg = Config_browser(browser_type).browser_type
    efg = Config_execution(execution_type).execution_type
    try:
        match efg:
            case "local":
                if cfg == "chrome_browser":
                    driver = webdriver.Chrome()
                elif cfg == "firefox_browser":
                    driver = webdriver.Firefox()
            case "grid":
                selenium_grid_config = {
                    'browserName': browser_type,
                }
                options = {
                    "chrome_browser": webdriver.ChromeOptions(),
                    "firefox_browser": webdriver.FirefoxOptions()
                }.get(cfg)
                driver = webdriver.Remote(
                    command_executor='http://localhost:4444',
                    options=options
                )
                print(options)
        yield driver
    finally:
        if driver:
            driver.quit()

This fixture will read the two options --browser and --execution, why we are ready the --browser too? , because we need to setup the browser type, in the Selenium Grid we have several options, so we need to indicate to the webdriver on which browser we will perform our tests.

We will setup the two cases we have, the local one will set up the driver as Chrome or as Firefox to be executed locally. The grid one will then setup the webdriver option, depending on the selected browser, and then it will setup the driver as webdriver.Remote this will indicate that we are going to use a different endpoint and port, and because we are running our Selenium Grid on our computers, the default endpoint is http:\\localhost:4444, if we have our Selenium Grid deployed in an external server, that endpoint will change.

Now the final step, we now need to pass our driver to our tests, because we are now setting up the driver with a different fixture, the tests cases need to be updated to get the execution one, instead of the browser, we update the tests as:

def test_home_page_loaded(execution):

    training_page = AB_TESTING(driver=execution)
    training_page.go()
    training_page.ab_testing_link.click()
    header_text = training_page.ab_testing_header.text
    assert "A/B Test Control" in header_text

We already accomplished a lot!!, but… how are we going to run our test in parallel?

Setting up Pytest Parallel Execution

To accomplish Parellel execution either locally or on the grid, we need another python dependency pytest-xdist.

pytest-xdist is a plugin that extends pytest with new test execution modes, the most used being distributing tests accross multiple CPUs to speed up test execution.

To install the dependency:

pip install pytest-xdist

It is time to set up pytest-xcdist because we need to setup the maximum number of workers that we can process. To do this we will add the next code lines into our pytest.ini file:

[xdist-parallel]
num_workers = 4  # Set the number of worker processes to use

I chose 4 because that is the number of processes that I want to use, and it will depend on the CPU you have. To know more about xdist, you can visit their web site pytest-xdist

Executing Tests

Now the moment we’ve all be waiting for, time to execute our tests.

We will have three new CLI options to run --browser, execution, and --numprocesses this can be abbreviated to -n.

Lets first test our local setup:

pytest --numprocesses 2 --browser chrome --execution local

Pytest will trigger the execution and you should see different chrome browsers poping up, this means we have local parallel execution! 😱

Now Selenium Grid:

pytest -n 2 --browser chrome --execution grid

Pytest is going to send the commands to the selenium grid, you can now see Selenium Grid orchestrating the tests automatically! 🚀

This is awesome we now have different ways to execute our tests and the framework is capable of receiving different CLI options to meet our needs.

Thank you for getting this far in this blog post, please check the code in the link: Pytest Selenium GH Project

• ← Previous Post