Overview

The purpose of the java-idp-integration-tests project is to test the IdP using Selenium. Only a handful of profile flows are currently tested.

The IdP is run in a java.lang.Process to separate it from the tests themselves. Consequently, the testbed is used to provide a mock SP as well as storage interface to validate tests.

Maven is used to :

retrieve and unpack the Jetty (jetty-distribution) and IdP (idp-distribution) artifacts
retrieve and deploy the testbed webapp
add test credentials, flows, metadata, and views from idp-conf to the IdP

resulting in a test-distributions directory containing Jetty and the IdP + testbed.

Each test class instance (of the abstract test class BaseIntegrationTest) creates an idp.home directory. This per-test-class idp.home directory is usually customized by test classes or instances using Java, and is deleted if all tests are successful. Helper methods are provided in the abstract test class to customize idp.home by copying or modifying configuration files, etc. Each test class instance runs the IdP + testbed on separate ports so that tests may be run in parallel, this may be disabled during development.

The testbed web app includes an in-memory directory server to test LDAP connectivity.

Each test method should start the container (Jetty) and a Selenium browser (web driver). This is so that the server and client may be configured appropriately for each test method.

Test methods may test SAML profiles by validating assertions displayed by the browser via the testbed's mock SP. The same SAML validators are used in browser tests (java-idp-integration-tests) as well as flow tests (idp-conf).

Setup

Source Code

Checkout the java-idp-integration-tests project. See Git Repository Access for details.

git clone git@git.shibboleth.net:java-idp-integration-tests

cd java-idp-integration-tests

Initial Maven Build

Build using Maven from the command line, not Eclipse.

From the java-idp-integration-tests directory, build the test-distributions directory via Maven :

mvn clean package -DskipTests

The Maven plugin for Eclipse (m2e) does not support executing the copy or unpack goals of the maven-dependency-plugin before the package phase (MDEP-187 MDEP-98). Build using Maven from the command line, and then refresh (File -> Refresh) in Eclipse.

Next Steps

At this point the java-idp-integration-tests directory should contain a test-distributions directory containing the IdP and Jetty distributions. You should be able to run tests via Maven from the command line or import the project into Eclipse and run them using the TestNG plugin.

Sauce Labs Authentication

To run remote tests on Sauce Labs, an account is required.

Populate the ~/.sauce-ondemand file with your username and access key. For example

~/.sauce-ondemand

username = myname-shibboleth
key = 789z...

Run Tests

Tests may be run from Maven on the command line or via Eclipse and the TestNG plugin.

Test via Eclipse

Import the java-idp-integration-tests into Eclipse and use TestNG to run tests.

Test via Maven

Change to the java-idp-integration-tests directory and run tests using Maven.

To run all tests :

mvn test

To run a specific test :

mvn test -Dtest=StatusTest

Per-Test-Class `idp.home` Directories

The Maven POM of the integration tests project :

unpacks the IdP (idp-distribution) to the test-distributions directory
unpacks Jetty (jetty-distribution) to the test-distributions directory
deploys the testbed (idp-testbed)
installs test flows from idp-conf

Each instance of the abstract test class (BaseIntegrationTest) copies the default idp.home directory to a new per-test-class idp.home directory named by a timestamp.

test-distributions/	Description
`yyyyMMdd-HHmmssSS`	Per-test-class `idp.home` directory.
`jetty-distribution-<version>`	Jetty.
`shibboleth-identity-provider-<version>`	Default `idp.home` directory : `idp-distribution`, `idp-testbed`, and test flows from `idp-conf`.

The per-test-class idp.home directory is deleted only if all tests pass.

To not delete per-test-class idp.home directories even when tests pass, set the keepTests system property : -DkeepTests=true

Helper methods are available to configure logging by modifying idp.home/conf/logback.xml.

Per-Test-Class Ports

Each instance of the abstract test class (BaseIntegrationTest) selects available ports for the web server and LDAP server to listen on.

To use the default ports (8080, 8443, and 9443 for backchannel), disable the setUpAvailablePorts() method :

BaseIntegrationTest.java

@BeforeClass(enabled = false)
public void setUpAvailablePorts() {
 ...

Selecting a Browser

The default browser is HtmlUnit for local tests and Firefox for remote tests.

Instances of the BrowserData.java bean supply the desired browser name, platform/OS, and version to test methods as a TestNG DataProvider. For example :

package net.shibboleth.idp.test;

import javax.annotation.Nullable;
import org.testng.Assert;
import org.testng.annotations.Test;
 
/** Test the IdP's status page. */
public class StatusTest extends BaseIntegrationTest {
 
	@Test(dataProvider = "sauceOnDemandBrowserDataProvider")
	public void testStatus(@Nullable final BrowserData browserData) throws Exception {
        startSeleniumClient(browserData);
        startJettyServer();
        driver.get("https://localhost:8443/idp/status");
        Assert.assertTrue(getPageSource().startsWith("### Operating Environment Information"));
	}
}

Possible values for browser, OS, and version :

Key	Values Documentation
browser	BrowserType.java
os	Platorm.java
version

Some example values :

browser	os
`"internet explorer"`	`"Windows Server 2012"`
`"iPhone"`	`"MAC"`

Selecting a Browser via System Property

Instances of BrowserData.java are populated from the SAUCE_ONDEMAND_BROWSERS system property. For example :

-DSAUCE_ONDEMAND_BROWSERS='[{"browser": "internet explorer", "os": "Windows Server 2012"}]'

Selecting a Browser via Java

In Java tests, browsers may be selected by defining the overrideCapabilities before starting the Selenium client. For example :

public void testStatus(@Nullable final BrowserData browserData) throws Exception {
	// Force Internet Explorer.
    overrideCapabilities = DesiredCapabilities.internetExplorer();
	startSeleniumClient(browserData);
	...

Test a Remote Browser

To test a remote browser set the following system properties :

Property Name	Property Value	Description
SELENIUM_IS_REMOTE	true	Remote browser if true, local browser otherwise.
server.address	IP address	IP address of server.
server.address.private	IP address	Optional private IP address of server.

For example :

-DSELENIUM_IS_REMOTE=true -Dserver.address=108.163.128.190

The private IP address may be useful when running behind a firewall.

Viewing Remote Browser Tests

Remote browser tests may be viewed at https://saucelabs.com/tests.

When run via Jenkins, the "Build" column is populated with the name of the test method and the build number from Jenkins.

Tests run by Jenkins are available at https://saucelabs.com/users/shibboleth-jenkins/tests.

When running tests locally in Eclipse and a tests fails, the SauceOnDemandTestListener prints a public link to the console.

Running Tests in Parallel

TODO

Because each test starts and stops both the IdP and a browser, it may take a long time to run tests, especially when remote browsers take several seconds to start up.

Development Cycle

When developing a dependency of the IdP, you will need to install the artifact via Maven and then re-build the java-idp-integration-tests project before running tests.

For example :

cd <path>/idp-<module>

mvn clean install

cd <path>/java-idp-integration-tests

mvn clean package -DskipTests

Or, you can install the dependency manually into the test-distributions directory of the java-idp-integration-tests project. Be aware that the manually-installed dependency will need to re-installed after building the java-idp-integration-tests project.

To Not Run Tests Headless

By default, tests are run using a headless browser provided by HtmlUnit.

To run tests using a "headful" browser, set the boolean firefox system property to true. For example, "-Dfirefox=true" in the Eclipse Launcher program arguments.

Firefox 52 ESR is know to work, although the exact version may change over time.

Currently, the integration tests only support launching the Firefox browser. For Chrome, adding the setupChromeDriver() method and boolean system property should be straightforward.

When using a "headful" browser, it may be helpful to manually manipulate the browser either before or after a test has run. One way to do this is to make the test sleep indefinitely by adding something like

Thread.sleep()

for(;;){
 Thread.sleep(10000);
}

to the test method. To cleanup, exit the browser and kill the process running the tests either via Eclipse or from the command line. You may need to kill the Jetty process running the IdP as well.

Command Line Options

-D8080=true
-DkeepTests=true