The Shibboleth IdP V4 software will leave support on September 1, 2024.
How to Run IdP Browser/Integration Tests
Overview
The purpose of the java-idp-integration-tests project is to test the IdP using Selenium. Only a handful of profile flows are currently tested.
The IdP is run in a java.lang.Process to separate it from the tests themselves. Consequently, the testbed is used to provide a mock SP as well as storage interface to validate tests.
Maven is used to :
retrieve and unpack the Jetty (
jetty-distribution)and IdP (idp-distribution)artifactsretrieve and deploy the testbed webapp
add test credentials, flows, metadata, and views from idp-conf to the IdP
This results in a test-distributions directory containing Jetty and the IdP + testbed.
Each test class instance (of the abstract test class BaseIntegrationTest) creates an idp.home directory. This per-test-class idp.home directory is usually customized by test classes or instances using Java, and is deleted if all tests are successful. Helper methods are provided in the abstract test class to customize idp.home by copying or modifying configuration files, etc. Each test class instance runs the IdP + testbed on separate ports so that tests may be run in parallel, this may be disabled during development.
The testbed web app includes an in-memory directory server to test LDAP connectivity.
Each test method should start the container (Jetty) and a Selenium browser (web driver). This is so that the server and client may be configured appropriately for each test method.
Test methods may test SAML profiles by validating assertions displayed by the browser via the testbed's mock SP. The same SAML validators are used in browser tests (java-idp-integration-tests) as well as flow tests (idp-conf).
Setup
Source Code
Checkout the java-idp-integration-tests project. See Git Repository Access for details.
git clone git@git.shibboleth.net:java-idp-integration-tests
cd java-idp-integration-tests
Initial Maven Build
Build using Maven from the command line, not Eclipse.
From the java-idp-integration-tests directory, build the test-distributions directory via Maven :
mvn clean package -DskipTests
The Maven plugin for Eclipse (m2e) does not support executing the copy or unpack goals of the maven-dependency-plugin before the package phase (MDEP-187 MDEP-98). Build using Maven from the command line, and then refresh (File -> Refresh) in Eclipse.
Next Steps
At this point the java-idp-integration-tests directory should contain a test-distributions directory containing the IdP and Jetty distributions. You should be able to run tests via Maven from the command line or import the project into Eclipse and run them using the TestNG plugin.
Sauce Labs Authentication
To run remote tests on Sauce Labs, an account is required.
Populate the ~/.sauce-ondemand file with your username and access key. For example
~/.sauce-ondemand
username = myname-shibboleth
key = 789z... Run Tests
Tests may be run from Maven on the command line or via Eclipse and the TestNG plugin.
Test via Eclipse
Import the java-idp-integration-tests into Eclipse and use TestNG to run tests.
Test via Maven
Change to the java-idp-integration-tests directory and run tests using Maven.
To run all tests :
mvn test
To run a specific test :
mvn test -Dtest=StatusTest
Per-Test-Class idp.home Directories
The Maven POM of the integration tests project unpacks the IdP (idp-distribution) and Jetty (jetty-distribution) to the test-distributions directory, deploys the testbed (idp-testbed), and installs test flows from idp-conf. Each instance of the abstract test class (BaseIntegrationTest) copies the default idp.home directory to a new per-test-class idp.home directory named by a timestamp.
test-distributions/ | Description |
|---|---|
yyyyMMdd-HHmmssSS | Per-test-class |
jetty-distribution-<version> | Jetty. |
shibboleth-identity-provider-<version> | Default |
The per-test-class idp.home directory is deleted only if all tests pass.
Helper methods are available to configure logging by modifying idp.home/conf/logback.xml.
Per-Test-Class Ports
Each instance of the abstract test class (BaseIntegrationTest) selects available ports for the web server and LDAP server to listen on.
To use the default ports (8080, 8443, and 9443 for backchannel), disable the setUpAvailablePorts() method :
BaseIntegrationTest.java
@BeforeClass(enabled = false)
public void setUpAvailablePorts() {
...Selecting a Browser
The default browser is HtmlUnit for local tests and Firefox for remote tests.
Instances of the BrowserData.java bean supply the desired browser name, platform/OS, and version to test methods as a TestNG DataProvider. For example :
package net.shibboleth.idp.test;
import javax.annotation.Nullable;
import org.testng.Assert;
import org.testng.annotations.Test;
/** Test the IdP's status page. */
public class StatusTest extends BaseIntegrationTest {
@Test(dataProvider = "sauceOnDemandBrowserDataProvider")
public void testStatus(@Nullable final BrowserData browserData) throws Exception {
startSeleniumClient(browserData);
startJettyServer();
driver.get("https://localhost:8443/idp/status");
Assert.assertTrue(getPageSource().startsWith("### Operating Environment Information"));
}
}Possible values for browser, OS, and version :
Key | Values Documentation |
|---|---|
browser | |
os | |
version |
|
Some example values :
browser | os |
|---|---|
"internet explorer" |
|
"iPhone" |
|
|
|
Selecting a Browser via System Property
Instances of BrowserData.java are populated from the SAUCE_ONDEMAND_BROWSERS system property. For example :
-DSAUCE_ONDEMAND_BROWSERS='[{"browser": "internet explorer", "os": "Windows Server 2012"}]'
Selecting a Browser via Java
In Java tests, browsers may be selected by defining the overrideCapabilities before starting the Selenium client. For example :
Test a Remote Browser
To test a remote browser set the following system properties :
Property Name | Property Value | Description |
|---|---|---|
SELENIUM_IS_REMOTE | true | Remote browser if true, local browser otherwise. |
server.address | IP address | IP address of server. |
server.address.private | IP address | Optional private IP address of server. |
For example :
-DSELENIUM_IS_REMOTE=true -Dserver.address=108.163.128.190
The private IP address may be useful when running behind a firewall.
Viewing Remote Browser Tests
Remote browser tests may be viewed at https://saucelabs.com/tests.
When run via Jenkins, the "Build" column is populated with the name of the test method and the build number from Jenkins.
Tests run by Jenkins are available at https://saucelabs.com/users/shibboleth-jenkins/tests.
When running tests locally in Eclipse and a tests fails, the SauceOnDemandTestListener prints a public link to the console.
Running Tests in Parallel
TODO
Because each test starts and stops both the IdP and a browser, it may take a long time to run tests, especially when remote browsers take several seconds to start up.
Development Cycle
When developing a dependency of the IdP, you will need to install the artifact via Maven and then re-build the java-idp-integration-tests project before running tests.
For example :
cd <path>/idp-<module>
mvn clean install
cd <path>/java-idp-integration-tests
mvn clean package -DskipTests
Or, you can install the dependency manually into the test-distributions directory of the java-idp-integration-tests project. Be aware that the manually-installed dependency will need to re-installed after building the java-idp-integration-tests project.