The purpose of the java-idp-integration-tests
project is to test the IdP using Selenium. Only a handful of profile flows are currently tested.
The IdP is run in a java.lang.Process to separate it from the tests themselves. Consequently, the testbed is used to provide a mock SP as well as storage interface to validate tests.
Maven is used to :
jetty-distribution)
and IdP (idp-distribution)
artifactsresulting in a test-distributions
directory containing Jetty and the IdP + testbed.
Each test class instance (of the abstract test class BaseIntegrationTest
) creates an idp.home
directory. This per-test-class idp.home
directory is usually customized by test classes or instances using Java, and is deleted if all tests are successful. Helper methods are provided in the abstract test class to customize idp.home
by copying or modifying configuration files, etc. Each test class instance runs the IdP + testbed on separate ports so that tests may be run in parallel, this may be disabled during development.
The testbed web app includes an in-memory directory server to test LDAP connectivity.
Each test method should start the container (Jetty) and a Selenium browser (web driver). This is so that the server and client may be configured appropriately for each test method.
Test methods may test SAML profiles by validating assertions displayed by the browser via the testbed's mock SP. The same SAML validators are used in browser tests (java-idp-integration-tests
) as well as flow tests (idp-conf)
.
Checkout the java-idp-integration-tests
project. See Git Repository Access for details.
git clone git@git.shibboleth.net:java-idp-integration-tests
cd java-idp-integration-tests
Build using Maven from the command line, not Eclipse. |
From the java-idp-integration-tests
directory, build the test-distributions
directory via Maven :
mvn clean package -DskipTests
The Maven plugin for Eclipse (m2e) does not support executing the copy or unpack goals of the maven-dependency-plugin before the package phase (MDEP-187 MDEP-98). Build using Maven from the command line, and then refresh (File -> Refresh) in Eclipse.
At this point the java-idp-integration-tests
directory should contain a test-distributions
directory containing the IdP and Jetty distributions. You should be able to run tests via Maven from the command line or import the project into Eclipse and run them using the TestNG plugin.
To run remote tests on Sauce Labs, an account is required.
Populate the ~/.sauce-ondemand
file with your username
and access key
. For example
username = myname-shibboleth key = 789z... |
Tests may be run from Maven on the command line or via Eclipse and the TestNG plugin.
Import the java-idp-integration-tests
into Eclipse and use TestNG to run tests.
Change to the java-idp-integration-tests
directory and run tests using Maven.
To run all tests :
mvn test
To run a specific test :
mvn test -Dtest=StatusTest
idp.home
DirectoriesThe Maven POM of the integration tests project :
idp-distribution)
to the test-distributions
directoryjetty-distribution
) to the test-distributions
directoryidp-testbed)
idp-conf
Each instance of the abstract test class (BaseIntegrationTest
) copies the default idp.home
directory to a new per-test-class idp.home
directory named by a timestamp.
test-distributions/ | Description |
---|---|
yyyyMMdd-HHmmssSS | Per-test-class idp.home directory. |
| Jetty. |
shibboleth-identity-provider-<version> | Default idp.home directory : idp-distribution , idp-testbed , and test flows from idp-conf . |
The per-test-class idp.home
directory is deleted only if all tests pass.
To not delete per-test-class idp.home
directories even when tests pass, set the keepTests
system property : -DkeepTests=true
Helper methods are available to configure logging by modifying idp.home/conf/logback.xml
.
Each instance of the abstract test class (BaseIntegrationTest) selects available ports for the web server and LDAP server to listen on.
To use the default ports (8080, 8443, and 9443 for backchannel), disable the setUpAvailablePorts()
method :
@BeforeClass(enabled = false) public void setUpAvailablePorts() { ... |
The default browser is HtmlUnit for local tests and Firefox for remote tests.
Instances of the BrowserData.java bean supply the desired browser name, platform/OS, and version to test methods as a TestNG DataProvider. For example :
package net.shibboleth.idp.test; import javax.annotation.Nullable; import org.testng.Assert; import org.testng.annotations.Test; /** Test the IdP's status page. */ public class StatusTest extends BaseIntegrationTest { @Test(dataProvider = "sauceOnDemandBrowserDataProvider") public void testStatus(@Nullable final BrowserData browserData) throws Exception { startSeleniumClient(browserData); startJettyServer(); driver.get("https://localhost:8443/idp/status"); Assert.assertTrue(getPageSource().startsWith("### Operating Environment Information")); } } |
Possible values for browser, OS, and version :
Key | Values Documentation |
---|---|
browser | BrowserType.java |
os | Platorm.java |
version |
Some example values :
browser | os |
---|---|
"internet explorer" | "Windows Server 2012" |
"iPhone" | "MAC" |
Instances of BrowserData.java are populated from the SAUCE_ONDEMAND_BROWSERS
system property. For example :
-DSAUCE_ONDEMAND_BROWSERS='[{"browser": "internet explorer", "os": "Windows Server 2012"}]'
In Java tests, browsers may be selected by defining the overrideCapabilities
before starting the Selenium client. For example :
public void testStatus(@Nullable final BrowserData browserData) throws Exception { // Force Internet Explorer. overrideCapabilities = DesiredCapabilities.internetExplorer(); startSeleniumClient(browserData); ... |
To test a remote browser set the following system properties :
Property Name | Property Value | Description |
---|---|---|
SELENIUM_IS_REMOTE | true | Remote browser if true, local browser otherwise. |
server.address | IP address | IP address of server. |
server.address.private | IP address | Optional private IP address of server. |
For example :
-DSELENIUM_IS_REMOTE=true -Dserver.address=108.163.128.190
The private IP address may be useful when running behind a firewall.
Remote browser tests may be viewed at https://saucelabs.com/tests.
When run via Jenkins, the "Build" column is populated with the name of the test method and the build number from Jenkins.
Tests run by Jenkins are available at https://saucelabs.com/users/shibboleth-jenkins/tests.
When running tests locally in Eclipse and a tests fails, the SauceOnDemandTestListener
prints a public link to the console.
TODO
Because each test starts and stops both the IdP and a browser, it may take a long time to run tests, especially when remote browsers take several seconds to start up.
When developing a dependency of the IdP, you will need to install the artifact via Maven and then re-build the java-idp-integration-tests
project before running tests.
For example :
cd <path>/idp-<module>
mvn clean install
cd <path>/java-idp-integration-tests
mvn clean package -DskipTests
Or, you can install the dependency manually into the test-distributions
directory of the java-idp-integration-tests
project. Be aware that the manually-installed dependency will need to re-installed after building the java-idp-integration-tests
project.
By default, tests are run using a headless browser provided by HtmlUnit.
To run tests using a "headful" browser, set the boolean firefox
system property to true. For example, "-Dfirefox=true
" in the Eclipse Launcher program arguments.
Firefox 52 ESR is know to work, although the exact version may change over time.
Currently, the integration tests only support launching the Firefox browser. For Chrome, adding the setupChromeDriver() method and boolean system property should be straightforward.
When using a "headful" browser, it may be helpful to manually manipulate the browser either before or after a test has run. One way to do this is to make the test sleep indefinitely by adding something like
for(;;){ Thread.sleep(10000); } |
to the test method. To cleanup, exit the browser and kill the process running the tests either via Eclipse or from the command line. You may need to kill the Jetty process running the IdP as well.
-D8080=true
-DkeepTests=true