How to Run IdP Browser/Integration Tests

Overview

The purpose of the java-idp-integration-tests project is to test the IdP using Selenium. Only a handful of profile flows are currently tested.

The IdP is run in a java.lang.Process to separate it from the tests themselves. Consequently, the testbed is used to provide a mock SP as well as storage interface to validate tests.

Maven is used to :

  • retrieve and unpack the Jetty (jetty-distribution) and IdP (idp-distribution) artifacts

  • retrieve and deploy the testbed webapp

  • add test credentials, flows, metadata, and views from idp-conf to the IdP

This results in a test-distributions directory containing Jetty and the IdP + testbed.

Each test class instance (of the abstract test class BaseIntegrationTest) creates an idp.home directory. This per-test-class idp.home directory is usually customized by test classes or instances using Java, and is deleted if all tests are successful. Helper methods are provided in the abstract test class to customize idp.home by copying or modifying configuration files, etc. Each test class instance runs the IdP + testbed on separate ports so that tests may be run in parallel, this may be disabled during development.

The testbed web app includes an in-memory directory server to test LDAP connectivity.

Each test method should start the container (Jetty) and a Selenium browser (web driver). This is so that the server and client may be configured appropriately for each test method.

Test methods may test SAML profiles by validating assertions displayed by the browser via the testbed's mock SP. The same SAML validators are used in browser tests (java-idp-integration-tests) as well as flow tests (idp-conf).

Setup 

Source Code

Checkout the java-idp-integration-tests project. See Git Repository Access for details.

git clone git@git.shibboleth.net:java-idp-integration-tests

cd java-idp-integration-tests

Initial Maven Build

Build using Maven from the command line, not Eclipse.

From the java-idp-integration-tests directory, build the test-distributions directory via Maven :

mvn clean package -DskipTests

The Maven plugin for Eclipse (m2e) does not support executing the copy or unpack goals of the maven-dependency-plugin before the package phase (MDEP-187 MDEP-98). Build using Maven from the command line, and then refresh (File -> Refresh) in Eclipse.

Next Steps

At this point the java-idp-integration-tests directory should contain a test-distributions directory containing the IdP and Jetty distributions. You should be able to run tests via Maven from the command line or import the project into Eclipse and run them using the TestNG plugin.

Sauce Labs Authentication

To run remote tests on Sauce Labs, an account is required.

Populate the ~/.sauce-ondemand file with your username and access key. For example

~/.sauce-ondemand
username = myname-shibboleth key = 789z... 

Run Tests

Tests may be run from Maven on the command line or via Eclipse and the TestNG plugin.

Test via Eclipse

Import the java-idp-integration-tests into Eclipse and use TestNG to run tests.

Test via Maven

Change to the java-idp-integration-tests directory and run tests using Maven.

To run all tests :

mvn test

To run a specific test :

mvn test -Dtest=StatusTest

Per-Test-Class idp.home Directories

The Maven POM of the integration tests project unpacks the IdP (idp-distribution) and Jetty (jetty-distribution) to the test-distributions directory, deploys the testbed (idp-testbed), and installs test flows from idp-conf. Each instance of the abstract test class (BaseIntegrationTest) copies the default idp.home directory to a new per-test-class idp.home directory named by a timestamp.

test-distributions/

Description

test-distributions/

Description

yyyyMMdd-HHmmssSS

Per-test-class idp.home directory.

jetty-distribution-<version>

Jetty.

shibboleth-identity-provider-<version>

Default idp.home directory : idp-distribution, idp-testbed, and test flows from idp-conf.

The per-test-class idp.home directory is deleted only if all tests pass.

Helper methods are available to configure logging by modifying idp.home/conf/logback.xml.

Per-Test-Class Ports

Each instance of the abstract test class (BaseIntegrationTest) selects available ports for the web server and LDAP server to listen on.

To use the default ports (8080, 8443, and 9443 for backchannel), disable the setUpAvailablePorts() method :

BaseIntegrationTest.java
@BeforeClass(enabled = false) public void setUpAvailablePorts() { ...

Selecting a Browser

The default browser is HtmlUnit for local tests and Firefox for remote tests.

Instances of the BrowserData.java bean supply the desired browser name, platform/OS, and version to test methods as a TestNG DataProvider. For example :

package net.shibboleth.idp.test; import javax.annotation.Nullable; import org.testng.Assert; import org.testng.annotations.Test;   /** Test the IdP's status page. */ public class StatusTest extends BaseIntegrationTest {   @Test(dataProvider = "sauceOnDemandBrowserDataProvider") public void testStatus(@Nullable final BrowserData browserData) throws Exception { startSeleniumClient(browserData); startJettyServer(); driver.get("https://localhost:8443/idp/status"); Assert.assertTrue(getPageSource().startsWith("### Operating Environment Information")); } }

Possible values for browser, OS, and version :

Key

Values Documentation

Key

Values Documentation

browser

BrowserType.java

os

Platorm.java

version

 

Some example values  :

browser

os

browser

os

"internet explorer"

"Windows Server 2012"

"iPhone"

"MAC"

 

 

Selecting a Browser via System Property

Instances of BrowserData.java are populated from the SAUCE_ONDEMAND_BROWSERS system property. For example :

-DSAUCE_ONDEMAND_BROWSERS='[{"browser": "internet explorer", "os": "Windows Server 2012"}]'

Selecting a Browser via Java

In Java tests, browsers may be selected by defining the overrideCapabilities before starting the Selenium client. For example :

Test a Remote Browser

To test a remote browser set the following system properties :

Property Name

Property Value

Description

Property Name

Property Value

Description

SELENIUM_IS_REMOTE

true

Remote browser if true, local browser otherwise.

server.address

IP address

IP address of server.

server.address.private

IP address

Optional private IP address of server.

For example :

-DSELENIUM_IS_REMOTE=true -Dserver.address=108.163.128.190

The private IP address may be useful when running behind a firewall.

Viewing Remote Browser Tests

Remote browser tests may be viewed at https://saucelabs.com/tests.

When run via Jenkins, the "Build" column is populated with the name of the test method and the build number from Jenkins.

Tests run by Jenkins are available at https://saucelabs.com/users/shibboleth-jenkins/tests.

When running tests locally in Eclipse and a tests fails, the SauceOnDemandTestListener prints a public link to the console. 

Running Tests in Parallel

TODO

Because each test starts and stops both the IdP and a browser, it may take a long time to run tests, especially when remote browsers take several seconds to start up.

Development Cycle

When developing a dependency of the IdP, you will need to install the artifact via Maven and then re-build the java-idp-integration-tests project before running tests.

For example :

cd <path>/idp-<module>

mvn clean install

cd <path>/java-idp-integration-tests

mvn clean package -DskipTests

Or, you can install the dependency manually into the test-distributions directory of the java-idp-integration-tests project. Be aware that the manually-installed dependency will need to re-installed after building the java-idp-integration-tests project.

 

 

Â