Software Test Engineer's Blog

Monday, March 16, 2015

JUnit 4 Vs TestNG – Comparison

TestNG Introduction

TestNG is a testing framework inspired from JUnit and NUnit but introducing some new functionality that make it more powerful and easier to use.
TestNG is an open source automated testing framework; where NG of TestNG means Next Generation. TestNG is similar to JUnit but it is much more powerful than JUnit but still it’s inspired by JUnit. It is designed to be better than JUnit, especially when testing integrated classes. Pay special thanks to Cedric Beust who is the creator of TestNG.
TestNG eliminates most of the limitations of the older framework and gives the developer the ability to write more flexible and powerful tests with help of easy annotations, grouping, sequencing & parameterizing.

Benefits of TestNG

There are number of benefits of TestNG but from Selenium perspective, major advantages of TestNG are :
1) It gives the ability to produce HTML Reports of execution
2) Annotations made testers life easy
3) Test cases can be Grouped & Prioritized more easily
4) Parallel testing is possible
5) Generates Logs
6) Data Parameteriz ation is possible

Test Case Writing

Writing a test in TestNG is quite simple and basically involves following steps:
Step 1 – Write the business logic of the test
Step 2 – Insert TestNG annotations in the code
Step 3 – Add the information about your test (e.g. the class names, methods names, groups names etc…) in a testng.xml file
Step 4 – Run TestNG

Annotations in TestNG

@BeforeSuite: The annotated method will be run before all tests in this suite have run.
@AfterSuite: The annotated method will be run after all tests in this suite have run.
@BeforeTest: The annotated method will be run before any test method belonging to the classes inside the tag is run.
@AfterTest: The annotated method will be run after all the test methods belonging to the classes inside the tag have run.
@BeforeGroups: The list of groups that this configuration method will run before. This method is guaranteed to run shortly before the first test method that belongs to any of these groups is invoked.
@AfterGroups: The list of groups that this configuration method will run after. This method is guaranteed to run shortly after the last test method that belongs to any of these groups is invoked.
@BeforeClass: The annotated method will be run before the first test method in the current class is invoked.
@AfterClass: The annotated method will be run after all the test methods in the current class have been run.
@BeforeMethod: The annotated method will be run before each test method.
@AfterMethod: The annotated method will be run after each test method.
@Test: The annotated method is a part of a test case.

Benefits of using annotations

1) TestNG identifies the methods it is interested in by looking up annotations. Hence method names are not restricted to any pattern or format.
2) We can pass additional parameters to annotations.
3) Annotations are strongly typed, so the compiler will flag any mistakes right away.
4) Test classes no longer need to extend anything (such as Test Case, for JUnit 3).

List of Topics

Below is the list of TestNg topics which we are going to study in the next chapters of TestNg Tutorial:
1) TestNg Installation
2) First Test Case with TestNG
3) Grouping, Prioritization & Sequencing of Test case
4) Running Test Cases as a Test Suite
5) TestNg Reporter Logs and Assertion
6) Data driven testing with TestNG Reporter
7) TestNG Listeners

TestNG Test Suite
In any project, you will end up to a place where you need to execute so many test cases on a run. Running a set of test cases together is call executing a Test Suite. Those test cases can be dependent to each other or may have to be executed in a certain order. TestNg gives us the capability to manage our test execution.
In TestNG framework, we need to create testng.xml file to create and handle multiple test classes. This is the xml file where you will configure your test run, set test dependency, include or exclude any test, method, class or package and set priority etc.
How to do it…
Step 1 : Create a TestNG XML
1) Right click on Project folder, go to New and select ‘File‘ as shown in below image.
2) In New file wizard, add file name = ‘testng.xml‘ as shown in below given image and click on Finish button.

3) It will add testng.xml file under your project folder.

Step 2 : Write xml code ?
1) Now add below given code in your testng.xml file.
1
2
3
4
5
6
7
8
9
10
11
<suite name="Any Name" >
<test name="Any Name" >
<classes> <class name="PackageName.TestCaseName" />
</classes>
</test>
</suite>
Note: You can choose any name for your Test Suite & Test Name as per your need.
2) After giving appropriate names, now your testng.xml file will looks like this:

Hope you have understood the xml code, as it is quite simple hierarchy:
Very first tag is the Suite tag<suite>, under that it is the Test tag<test> and then the Class tag<classes>. You can give any name to the suite and the test but you need to provide the correct name to the <classes> tag which is a combination of your Package name and Test Case name.

Step 3 : Execute a testng.xml
Now it’s time to run the xml. Run the test by right click on the testng.xml file and select Run As > TestNG Suite.
It will take few seconds to start the testng execution engine and soon you will notice that your test will run and complete. Once the execution is complete, you can view test execution result under the TestNg console.
This is the one simple example of creating and running testng.xml file in eclipse.

Building a Test Suite

Now when you have learned how to build the xml, now it’s time to learn how to build a Test Suite using testng.xml. It is again not a complex task, all you need to do is to add your test cases to your xml file in <classes> tag.

The above test will execute only those tests, which are mentioned in the testng.xml. The rest of the test cases under ‘automationFramework’ package will remain untouched.

TestNG Annotations

In the TestNG Introduction chapter we have came across different annotations used in TestNG Framework but so far we have used just three(Before, After & Test). All though these are the most frequently used annotations but who know how far you will go with your framework and may like to use other useful TestNG annotations.
Before that I would like you to give a small idea on Annotations hierarchy or Annotations levels in TestNG.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
<suite>
   <test>
<classes>
<method>
       <test>
    </method>
</classes>
   </test>
</suite>
It says that @Test is the smallest annotation here. @Method will be executed first, before and after the execution of @Test. The same way @Class will be executed first, before and after the execution of @Method and so on.
Now with the below example it will be clear to you easily.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
package automationFramework;
import org.testng.annotations.AfterClass;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.AfterSuite;
import org.testng.annotations.AfterTest;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.BeforeTest;
import org.testng.annotations.Test;
public class Sequencing {
@Test
public void testCase1() {
System.out.println("This is the Test Case 1");
}
@Test
public void testCase2() {
System.out.println("This is the Test Case 2");
}
@BeforeMethod
public void beforeMethod() {
System.out.println("This will execute before every Method");
}
@AfterMethod
public void afterMethod() {
System.out.println("This will execute after every Method");
}
@BeforeClass
public void beforeClass() {
System.out.println("This will execute before the Class");
}
@AfterClass
public void afterClass() {
System.out.println("This will execute after the Class");
}
@BeforeTest
public void beforeTest() {
System.out.println("This will execute before the Test");
}
@AfterTest
public void afterTest() {
System.out.println("This will execute after the Test");
}
@BeforeSuite
public void beforeSuite() {
System.out.println("This will execute before the Test Suite");
}
@AfterSuite
public void afterSuite() {
System.out.println("This will execute after the Test Suite");
}
}

Output of the above code will be like this:

It is clearly visible that the @Suite annotation is the very first and the very lastly executed. Then @Test followed by @Class. Now if you notice, the @Method has executed twice. As @Test is a method in the class, hence @Method will always executed for each @Test method.

Test Case Grouping

‘Groups‘ is one more annotation of TestNG which can be used in the execution of multiple tests. Let’s say you have hundred tests of class vehicle and in it ten method of car, ten method of scooter and so on. You probably like to run all the scooter tests together in a batch. And you want all to be in a single test suite. With the help of grouping you can easily overcome this situation.
How to do it…
1) Create two methods for Car, two methods for Scooter and one method in conjunction with Car & Sedan Car.
2) Group them separately with using  (groups = { ” Group Name” })
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
package automationFramework;
import org.testng.annotations.Test;
public class Grouping {
  @Test (groups = { "Car" })
  public void Car1() {
  System.out.println("Batch Car - Test car 1");
  }
  @Test (groups = { "Car" })
  public void Car2() {
  System.out.println("Batch Car - Test car 2");
  }
  @Test (groups = { "Scooter" })
  public void Scooter1() {
  System.out.println("Batch Scooter - Test scooter 1");
  }
  @Test (groups = { "Scooter" })
  public void Scooter2() {
  System.out.println("Batch Scooter - Test scooter 2");
  }
  @Test (groups = { "Car", "Sedan Car" })
  public void Sedan1() {
  System.out.println("Batch Sedan Car - Test sedan 1");
  }
}

3) Create a testng xml like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<suite name="Suite">
    <test name="Practice Grouping">
        <groups>
    <run>
<include name="Car" />
    </run>
</groups>
<classes>
    <class name="automationFramework.Grouping" />
</classes>
    </test>
</suite>

4) Run the test by right click on the testng.xml file and select Run As > TestNG Suite. Output will be like this in TestNg console:

 Note: We have just call the group ‘Car’ from the xml and it also executed the test for Sedan Car, as we have mentioned the ‘Car’ as well while declaring the group of Sedan Car.
Clubbing of groups is also possible, take a look at the below xml:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
<suite name="Suite">
   <test name="Practice Grouping">
      <groups>
         <define name="All">
   <include name="Car"/>
   <include name="Scooter"/>
</define>
<run>
   <include name="All"/>
</run>
   </groups>
<classes>
      <class name="automationFramework.Grouping" />
</classes>
   </test>
</suite>
You can see that we have created a new Group with the name ‘All’ and include other groups in it. Then simply called the newly created group for execution. The output will be like this:


Dependent Test

Sometimes, you may need to invoke methods in a Test case in a particular order or you want to share some data and state between methods. This kind of dependency is supported by TestNG as it supports the declaration of explicit dependencies between test methods.
TestNG allows you to specify dependencies either with:
  • Using attributes dependsOnMethods in @Test annotations OR
  • Using attributes dependsOnGroups in @Test annotations.
Take a look over the below example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
package automationFramework;
import org.testng.annotations.Test;
public class Dependent {
  @Test (dependsOnMethods = { "OpenBrowser" })
  public void SignIn() {
  System.out.println("This will execute second (SignIn)");
  }
  @Test
  public void OpenBrowser() {
  System.out.println("This will execute first (Open Browser)");
  }
  @Test (dependsOnMethods = { "SignIn" })
  public void LogOut() {
  System.out.println("This will execute third (Log Out)");
  }
The output will be like this:

TestNG-Dependent1

Multiple Tests

There will be situations when you want to put number of tests under a single test class and like to run all in single shot.  With the help of TestNG ‘@Test‘ annotations we can execute multiple tests in single TestNG file.
Take an example of four different tests under one testng class and print the test sequence on the console.
How to do it…
1) Press Ctrl+N , select “TestNG Class” under TestNG category and click Next.
Or
Right click on Test Case folder, go to TestNG and select “TestNG Class“.

2) If your project is set up and you have selected the Test Case folder before creating TestNG class then the source folder and the package name will be pre-populated on the form. Set class name as ‘TestNG‘.
Leave rest of the settings untouched, do not check for “@BeforeMethod”, “@AfterMethod” for now and click Finish. That’s it.

3) By default a new class will have only one @Test method. Add two more methods by yourself and put your code accordingly in methods. Code will look like:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
package automationFramework;
import org.openqa.selenium.WebDriver;
import org.testng.annotations.Test;
public class MultipleTest {
public WebDriver driver;
  @Test
  public void One() {
      System.out.println("This is the Test Case number One");
  }
  @Test
  public void Two() {
  System.out.println("This is the Test Case number Two");
  }
  @Test
  public void Three() {
  System.out.println("This is the Test Case number Three");
  }
  @Test
  public void Four() {
  System.out.println("This is the Test Case number Four");
  }
}
This will enable you to execute all four tests with just one testng class. Take a look on the output.

Attention: By default, methods annotated by @Test are executed alphabetically. Take a look over the next topic to see how to prioritize @Test.

Sequencing & Prioritizing

You need to use the ‘priority‘ parameter, if you want the methods to be executed in your order. Parameters are keywords that modify the annotation’s function.
Let’s take the same above example and execute all @Test methods in right order. Simply assign priority to all @Test methods starting from 0(Zero).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
package automationFramework;
import org.openqa.selenium.WebDriver;
import org.testng.annotations.Test;
public class MultipleTest {
public WebDriver driver;
  @Test(priority = 0)
  public void One() {
      System.out.println("This is the Test Case number One");
  }
  @Test(priority = 1)
  public void Two() {
  System.out.println("This is the Test Case number Two");
  }
  @Test(priority = 2)
  public void Three() {
  System.out.println("This is the Test Case number Three");
  }
  @Test(priority = 3)
  public void Four() {
  System.out.println("This is the Test Case number Four");
  }
}
Note: TestNG will execute the @Test annotation with the lowest priority value up to the largest.
Output of the above:

Skipping a Test Case

Think of a situation where you are required to skip one or more @Test from your testng class. In testng, you can easily able to handle this situation by setting the ‘enabled’ parameter to ‘false’ for e.g.:
@Test(enabled = false)
To use two or more parameters in a single annotation, separate them with a comma:
@Test(priority = 3, enabled = false)
Again take the same example and set the value false for the third test.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
package automationFramework;
import org.openqa.selenium.WebDriver;
import org.testng.annotations.Test;
public class MultipleTest {
public WebDriver driver;
  @Test(priority = 0)
  public void One() {
      System.out.println("This is the Test Case number One");
  }
  @Test(priority = 1)
  public void Two() {
  System.out.println("This is the Test Case number Two");
  }
  @Test(priority = 2, enabled = false)
  public void Three() {
  System.out.println("This is the Test Case number Three");
  }
  @Test(priority = 3)
  public void Four() {
  System.out.println("This is the Test Case number Four");
  }
}
Output of the above example:

TestNG Reporters & Asserts

TestNG Reporters

TestNG is a Framework and so far we have already seen the many different powerful features of TestNG. It almost gives you all the important things you are required to complete the Framework.

TestNG Reporter Logs

TestNG also gives us the logging facility for the test. For example during the running of test case user wants some information to be logged in the console. Information could be any detail depends upon the purpose. Keeping this in mind that we are using Selenium for testing, we need the information which helps the User to understand the test steps or any failure during the test case execution. With the help of TestNG Logs it is possible to enable logging during the Selenium test case execution.
In selenium there are two types of logging. High level logging and Low level logging. In low level logging you try to produce logs for the every step you take or every action you make in your automation script. In high  level logging you just try to capture main events of your test.
Everybody has their own style of logging and I have mine too. I am also a big fan of Log4j logging and that’s why I do not mix log4j logging with testng logging but on the same side I make to use of both of its. I perform low level logging with log4j and high level logging with testng reporter logs.
How to do it…
1) Write a test case for Sign In application and implement Log4j logging on every step.
2) Insert Reporter logs on the main events of the test.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
package automationFramework;
import java.util.concurrent.TimeUnit;
import org.apache.log4j.Logger;
import org.apache.log4j.xml.DOMConfigurator;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.testng.Reporter;
import org.testng.annotations.Test;
import utility.Log;
public class ReporterLogs {
private static WebDriver driver;
private static Logger Log = Logger.getLogger(Log.class.getName());
    @Test
public static void test() {
DOMConfigurator.configure("log4j.xml");
        driver = new FirefoxDriver();
        Log.info("New driver instantiated");
        driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
        Log.info("Implicit wait applied on the driver for 10 seconds");
        driver.get("http://www.store.demoqa.com");
        Log.info("Web application launched");
        // Our first step is complete, so we produce a main event log here for our reports.
        Reporter.log("Application Lauched successfully | ");
        driver.findElement(By.xpath(".//*[@id='account']/a")).click();
        Log.info("Click action performed on My Account link");
        driver.findElement(By.id("log")).sendKeys("testuser_1");
        Log.info("Username entered in the Username text box");
        driver.findElement(By.id("pwd")).sendKeys("Test@123");
        Log.info("Password entered in the Password text box");
        driver.findElement(By.id("login")).click();
        Log.info("Click action performed on Submit button");
        // Here we are done with our Second main event
        Reporter.log("Sign In Successful | " );
        driver.findElement(By.id("account_logout"));
        Log.info("Click action performed on Log out link");
        driver.quit();
        Log.info("Browser closed");
        // This is the third main event
        Reporter.log("User is Logged out and Application is closed | ");
}
}
3) Run the test by right click on the test case script and select Run As > TestNG Test.
Your Log4j logging output will look like this:

But your Reporters log will look like this:

Log4j logging will help you to report a bug or steps taken during the test, on the other hand reporters log will help you to share the test status with leadership. As leadership is just interested in the test results, not the test steps.
I also use reporter’s logs on the verification during the test. For example
1
2
3
4
5
6
7
8
9
if(Text1.equals(Text2)){
Reporter.log("Verification Passed forText");
}else{
Reporter.log("Verification Failed for Text");
}

TestNG Asserts

TestNG also gives us the power to take decisions in the middle of the test run with the help of Asserts. With this we can put various checkpoints in the test. Asserts are the most popular and frequently used methods while creating Selenium Scripts. In selenium there will be many situations in the test where you just like to check the presence of an element. All you need to do is to put an assert statement on to it to verify its existence.
Different Asserts Statements
1) Assert.assertTrue() & Assert.assertFalse() 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
package automationFramework;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.testng.Assert;
import org.testng.annotations.Test;
public class Asserts {
private static WebDriver driver;
  @Test
  public void f() {
  driver = new FirefoxDriver();
      driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
      driver.get("http://www.store.demoqa.com");
      // Here driver will try to find out My Account link on the application
      WebElement myAccount = driver.findElement(By.xpath(".//*[@id='account']/a"));
      //Test will only continue, if the below statement is true
      //This is to check whether the link is displayed or not
      Assert.assertTrue(myAccount.isDisplayed());
      //My Account will be clicked only if the above condition is true
      myAccount.click();
  }
}
Note: Assert true statement fails the test and stop the execution of the test, if the actual output is false.  Assert.assertFalse() works opposite of Assert.assertTrue(). It means that if you want your test to continue only if when some certain element is not present on the page. You will use Assert false, so it will fail the test in case of the element present on the page.
2) Assert.assertEquals()

1
2
3
4
5
6
7
8
9
10
11
12
13
  @Test
  public void test() {
  String sValue = "Lakshay Sharma";
  System.out.println(" What is your full name");
  Assert.assertEquals("Lakshay Sharma", sValue);
  System.out.println(sValue);
  }
It also works the same way like assert true and assert fail. It will also stop the execution, if the value is not equal and carry on the execution, if the value is equal.

TestNG Parameters & Data Provider

TestNG Parameters

Everybody knows the importance of Parameterization in testing and in automation testing. It allows us to automatically run a test case multiple times with different input and validation values. As Selenium Webdriver is more an automated testing framework than a ready-to-use tool, you will have to put in some effort to support data driven testing in your automated tests. I usually prefer to use Microsoft Excel as the format for storing my parameters but so many of my followers have requested to write an article on TestNG Data Provider.
TestNG again gives us another interesting feature called TestNG Parameters. TestNG lets you pass parameters directly to your test methods with your testng.xml.
How to do it…
Let me take a very simple example of LogIn application, where the username and password is required to clear the authentication.
1) Create a test on my demo OnlineStore application to perform LogIn which takes the two string argument as username & password.
2) Provide Username & Password as parameter using TestNG Annotation.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
package automationFramework;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.testng.annotations.Test;
import org.testng.annotations.Parameters;
public class TestngParameters {
private static WebDriver driver;
  @Test
  @Parameters({ "sUsername", "sPassword" })
  public void test(String sUsername, String sPassword) {
  driver = new FirefoxDriver();
      driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
      driver.get("http://www.store.demoqa.com");
      driver.findElement(By.xpath(".//*[@id='account']/a")).click();
      driver.findElement(By.id("log")).sendKeys(sUsername);
      driver.findElement(By.id("pwd")).sendKeys(sPassword);
      driver.findElement(By.id("login")).click();
      driver.findElement(By.xpath(".//*[@id='account_logout']/a")).click();
      driver.quit();
  }
}
3) The parameter would be passed values from testng.xml which we will see in the next step.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
<suite name="Suite">
    <test name="ToolsQA">
<parameter name="sUsername" value="testuser_1"/>
<parameter name="sPassword" value="Test@123"/>
<classes>
    <class name="automationFramework.TestngParameters" />
</classes>
    </test>
</suite>
Now, run the testng.xml, which will run the parameterTest method. TestNG will try to find a parameter named sUsername & sPassword.

TestNG DataProviders

When you need to pass complex parameters or parameters that need to be created from Java (complex objects, objects read from a property file or a database, etc…), in such cases parameters can be passed using Dataproviders. A Data Provider is a method annotated with @DataProvider. A Data Provider returns an array of objects.
Let us check out the same Sign In examples using Dataproviders.
How to do it…
1)  Define the method credentials() which is defined as a Dataprovider using the annotation. This method returns array of object array.
2) Add a method test() to your DataProviderTest class. This method takes two strings as input parameters.
3) Add the annotation @Test(dataProvider = “Authentication”) to this method. The attribute dataProvider is mapped to “Authentication”.
Test will look like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
package automationFramework;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;
public class DataProviderTest {
private static WebDriver driver;
  @DataProvider(name = "Authentication")
  public static Object[][] credentials() {
        return new Object[][] { { "testuser_1", "Test@123" }, { "testuser_1", "Test@123" }};
  }
  // Here we are calling the Data Provider object with its Name
  @Test(dataProvider = "Authentication")
  public void test(String sUsername, String sPassword) {
  driver = new FirefoxDriver();
      driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
      driver.get("http://www.store.demoqa.com");
      driver.findElement(By.xpath(".//*[@id='account']/a")).click();
      driver.findElement(By.id("log")).sendKeys(sUsername);
      driver.findElement(By.id("pwd")).sendKeys(sPassword);
      driver.findElement(By.id("login")).click();
      driver.findElement(By.xpath(".//*[@id='account_logout']/a")).click();
      driver.quit();
  }
}
Run the test by right click on the test case script and select Run As > TestNG Test. Give it few minutes to complete the execution, once it is finished the results will look like this in the TestNg Result window.

Note: As the test data is provided two times, the above test executed two times completely.

TestNG DataProvider with Excel

Data Driven Testing

A key benefit of automating functional testing is the ability to test large volumes of data on the system quickly. But you must be able to manipulate the data sets, perform calculations, and quickly create hundreds of test iterations and permutations with minimal effort. Test Automation Frameworks must have capability to integrate with spreadsheets and provide powerful calculation features.

Apache POI (Excel)

Most commercial automated software tools on the market support some sort of data driven testing, which allows you to automatically run a test case multiple times with different input and validation values. As Selenium Webdriver is more an automated testing framework than a ready-to-use tool, you will have to put in some effort to support data driven testing in your automated tests. I usually prefer to use Microsoft Excel as the format for storing my parameters. An additional advantage of using Excel is that you can easily outsource the test data administration to someone other than yourself, someone who might have better knowledge of the test cases that need to be run and the parameters required to execute them.

TestNG Data Providers

When you need to pass complex parameters or parameters that need to be created from Java (complex objects, objects read from a property file or a database, etc…), in such cases parameters can be passed using Dataproviders. A Data Provider is a method annotated with @DataProvider. A Data Provider returns an array of objects.
Let us check out the same Sign In examples using Data Providers with Excel data sheet.

How to do it…

Here we will follow a simple step by step process to Implement Excel with TestNg Data Provider.
Step 1: Create a test case of Login Application with TestNG Data Provider.
Step 2:  Create a Test Data sheet.
Step 3: Create functions to Open & Read data from Excel
Step 4: Create a TestNg test case for accepting data from Excel using Data Provider.
Step 5: Run the test against the Test Case name in the Test Data file.

Step 1: Create a test case of LogIn Application with TestNG Data Provider

1) Create a TestNG class ‘DataProviderTest’ by Pressing Ctrl+N , select ‘Create TestNG Class’ under TestNG category and Under Annotations, check ‘DataProvider’ and click Finish.
2) By default, the DataProvider name is ‘dp’, change it to ‘Authentication’. This method returns array of object array.
3) Add a method Registration_data() to your Test class. This method takes two strings as input parameters.
4) Write script for LogIn Application under method @Test.
Test Case will look like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
package automationFramework;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;
public class DataProviderTest {
    private static WebDriver driver;
  @DataProvider(name = "Authentication")
  public static Object[][] credentials() {
        // The number of times data is repeated, test will be executed the same no. of times
        // Here it will execute two times
        return new Object[][] { { "testuser_1", "Test@123" }, { "testuser_1", "Test@123" }};
  }
  // Here we are calling the Data Provider object with its Name
  @Test(dataProvider = "Authentication")
  public void test(String sUsername, String sPassword) {
      driver = new FirefoxDriver();
      driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
      driver.get("http://www.store.demoqa.com");
      driver.findElement(By.xpath(".//*[@id='account']/a")).click();
      // Argument passed will be used here as String Variable
      driver.findElement(By.id("log")).sendKeys(sUsername);
      driver.findElement(By.id("pwd")).sendKeys(sPassword);
      driver.findElement(By.id("login")).click();
      driver.findElement(By.xpath(".//*[@id='account_logout']/a")).click();
      driver.quit();
  }
}

Step 2:  Create a Test Data sheet

1) Create a ‘New Package‘ file and name it as ‘testData’, by right click on the Project and select New > Package. I always place my Test Data file under separate test data folder.

2) Place a Excel file in the above created package location and save it as TestData.xlsx. Fill the data in the excel like below image:


Step 3: Create functions to Open & Read data from Excel

We need a way to open this Excel sheet and read data from it within our Selenium test script. For this purpose, I use the Apache POI library, which allows you to read, create and edit Microsoft Office-documents using Java. The classes and methods we are going to use to read data from Excel sheet are located in the org.apache.poi.hssf.usermodel package.
To see step by step process to set up Apache POI Excel, please visit Data Driven Framework.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
package utility;
        import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import org.apache.poi.xssf.usermodel.XSSFCell;
import org.apache.poi.xssf.usermodel.XSSFRow;
import org.apache.poi.xssf.usermodel.XSSFSheet;
import org.apache.poi.xssf.usermodel.XSSFWorkbook;
    public class ExcelUtils {
private static XSSFSheet ExcelWSheet;
private static XSSFWorkbook ExcelWBook;
private static XSSFCell Cell;
private static XSSFRow Row;
public static Object[][] getTableArray(String FilePath, String SheetName) throws Exception {  
   String[][] tabArray = null;
   try {
   FileInputStream ExcelFile = new FileInputStream(FilePath);
   // Access the required test data sheet
   ExcelWBook = new XSSFWorkbook(ExcelFile);
   ExcelWSheet = ExcelWBook.getSheet(SheetName);
   int startRow = 1;
   int startCol = 1;
   int ci,cj;
   int totalRows = ExcelWSheet.getLastRowNum();
   // you can write a function as well to get Column count
   int totalCols = 2;
   tabArray=new String[totalRows][totalCols];
   ci=0;
   for (int i=startRow;i<=totalRows;i++, ci++) {             
  cj=0;
   for (int j=startCol;j<=totalCols;j++, cj++){
   tabArray[ci][cj]=getCellData(i,j);
   System.out.println(tabArray[ci][cj]);  
}
}
}
catch (FileNotFoundException e){
System.out.println("Could not read the Excel sheet");
e.printStackTrace();
}
catch (IOException e){
System.out.println("Could not read the Excel sheet");
e.printStackTrace();
}
return(tabArray);
}
public static String getCellData(int RowNum, int ColNum) throws Exception {
try{
Cell = ExcelWSheet.getRow(RowNum).getCell(ColNum);
int dataType = Cell.getCellType();
if  (dataType == 3) {
return "";
}else{
String CellData = Cell.getStringCellValue();
return CellData;
}catch (Exception e){
System.out.println(e.getMessage());
throw (e);
}
}
}

Step 4: Create a TestNg test case for accepting data from Excel using Data Provider

1) Create a TestNG class ‘DataProviderWithExcel’ by Pressing Ctrl+N , select ‘Create TestNG Class‘ under TestNG category and Under Annotations, check ‘@BeforeMethod‘, ‘@AfterMethod‘ & ‘DataProvider‘ and click Finish.
3) Add a method Registration_data() to your Test class. This method takes two strings as input parameters.
4) Now divide the test case in to three parts :
@BeforeMethod : Launch Firefox and direct it to the Base URL
@Test : Enter Username & Password to Login, Print console message and Log out
@AfterMethod : Close Firefox browser
Test Case will look like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
package practiceTestCases;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
import org.testng.annotations.DataProvider;
import utility.ExcelUtils;
public class DataProviderWithExcel_001 {
WebDriver driver;
    @BeforeMethod
    public void beforeMethod() throws Exception {
    driver = new FirefoxDriver();
        driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
        driver.get("http://www.store.demoqa.com");
}
@Test(dataProvider="Authentication")
    public void Registration_data(String sUserName,String sPassword)throws  Exception{
        driver.findElement(By.xpath(".//*[@id='account']/a")).click();
        driver.findElement(By.id("log")).sendKeys(sUserName);
System.out.println(sUserName);
        driver.findElement(By.id("pwd")).sendKeys(sPassword);
System.out.println(sPassword);
        driver.findElement(By.id("login")).click();
        System.out.println(" Login Successfully, now it is the time to Log Off buddy.");
        driver.findElement(By.xpath(".//*[@id='account_logout']/a")).click();
}
    @DataProvider
    public Object[][] Authentication() throws Exception{
         Object[][] testObjArray = ExcelUtils.getTableArray("D://ToolsQA//OnlineStore//src//testData//TestData.xlsx","Sheet1");
         return (testObjArray);
}
    @AfterMethod
    public void afterMethod() {
       driver.close();
     }
}

Note: This LogIn test will execute two times as there are two users credentials in data provider Array.

Step 5: Run the test against the Test Case name in the Test Data file

1) This means that your test should be run once only with the data which is mentioned against the Test Case name. To do this we need to tweak the Excel utility class, plus need to add few more function to fetch out the current Test Case name & the row number which contain the Test Case name.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
package utility;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import org.apache.poi.xssf.usermodel.XSSFCell;
import org.apache.poi.xssf.usermodel.XSSFRow;
import org.apache.poi.xssf.usermodel.XSSFSheet;
import org.apache.poi.xssf.usermodel.XSSFWorkbook;
    public class ExcelUtils {
private static XSSFSheet ExcelWSheet;
private static XSSFWorkbook ExcelWBook;
private static XSSFCell Cell;
private static XSSFRow Row;
//This method is to set the File path and to open the Excel file, Pass Excel Path and Sheetname as Arguments to this method
public static void setExcelFile(String Path,String SheetName) throws Exception {
   try {
// Open the Excel file
FileInputStream ExcelFile = new FileInputStream(Path);
// Access the required test data sheet
ExcelWBook = new XSSFWorkbook(ExcelFile);
ExcelWSheet = ExcelWBook.getSheet(SheetName);
} catch (Exception e){
throw (e);
}
}
public static Object[][] getTableArray(String FilePath, String SheetName, int iTestCaseRow)    throws Exception
{  
   String[][] tabArray = null;
   try{
   FileInputStream ExcelFile = new FileInputStream(FilePath);
   // Access the required test data sheet
   ExcelWBook = new XSSFWorkbook(ExcelFile);
   ExcelWSheet = ExcelWBook.getSheet(SheetName);
   int startCol = 1;
   int ci=0,cj=0;
   int totalRows = 1;
   int totalCols = 2;
   tabArray=new String[totalRows][totalCols];
   for (int j=startCol;j<=totalCols;j++, cj++)
   {
   tabArray[ci][cj]=getCellData(iTestCaseRow,j);
   System.out.println(tabArray[ci][cj]);
   }
}
catch (FileNotFoundException e)
{
System.out.println("Could not read the Excel sheet");
e.printStackTrace();
}
catch (IOException e)
{
System.out.println("Could not read the Excel sheet");
e.printStackTrace();
}
return(tabArray);
}
//This method is to read the test data from the Excel cell, in this we are passing parameters as Row num and Col num
public static String getCellData(int RowNum, int ColNum) throws Exception{
   try{
  Cell = ExcelWSheet.getRow(RowNum).getCell(ColNum);
  String CellData = Cell.getStringCellValue();
  return CellData;
  }catch (Exception e){
return"";
}
}
public static String getTestCaseName(String sTestCase)throws Exception{
String value = sTestCase;
try{
int posi = value.indexOf("@");
value = value.substring(0, posi);
posi = value.lastIndexOf(".");
value = value.substring(posi + 1);
return value;
}catch (Exception e){
throw (e);
}
}
public static int getRowContains(String sTestCaseName, int colNum) throws Exception{
int i;
try {
int rowCount = ExcelUtils.getRowUsed();
for ( i=0 ; i<rowCount; i++){
if  (ExcelUtils.getCellData(i,colNum).equalsIgnoreCase(sTestCaseName)){
break;
}
}
return i;
}catch (Exception e){
throw(e);
}
}
public static int getRowUsed() throws Exception {
try{
int RowCount = ExcelWSheet.getLastRowNum();
return RowCount;
}catch (Exception e){
System.out.println(e.getMessage());
throw (e);
}
}
}

Final Test Case
1) Get the Test Case name.
2) With the Test Case name, get the row number of the Excel sheet for the test.
3) Get the data from the excel sheet for the fetched test row.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
package practiceTestCases;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.testng.annotations.Test;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.DataProvider;
import utility.ExcelUtils;
public class DataProviderWithExcel_002 {
private String sTestCaseName;
private int iTestCaseRow;
WebDriver driver;
  @BeforeMethod
  public void beforeMethod() throws Exception {
  driver = new FirefoxDriver();
      driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
      driver.get("http://www.store.demoqa.com");
  }
  @Test(dataProvider = "Authentication")
  public void f(String sUserName, String sPassword) {
    driver.findElement(By.xpath(".//*[@id='account']/a")).click();
    driver.findElement(By.id("log")).sendKeys(sUserName);
System.out.println(sUserName);
    driver.findElement(By.id("pwd")).sendKeys(sPassword);
System.out.println(sPassword);
    driver.findElement(By.id("login")).click();
    System.out.println(" Login Successfully, now it is the time to Log Off buddy.");
    driver.findElement(By.xpath(".//*[@id='account_logout']/a")).click();
  }
  @AfterMethod
  public void afterMethod() {
   driver.close();
  }
  @DataProvider
  public Object[][] Authentication() throws Exception{
    // Setting up the Test Data Excel file
ExcelUtils.setExcelFile("D://ToolsQA//OnlineStore//src//testData//TestData.xlsx","Sheet1");
sTestCaseName = this.toString();
   // From above method we get long test case name including package and class name etc.
   // The below method will refine your test case name, exactly the name use have used
   sTestCaseName = ExcelUtils.getTestCaseName(this.toString());
    // Fetching the Test Case row number from the Test Data Sheet
    // Getting the Test Case name to get the TestCase row from the Test Data Excel sheet
iTestCaseRow = ExcelUtils.getRowContains(sTestCaseName,0);
    Object[][] testObjArray = ExcelUtils.getTableArray("D://ToolsQA//OnlineStore//src//testData//TestData.xlsx","Sheet1",iTestCaseRow);
     return (testObjArray);
}
}

Note: This will just execute your test once against the current est case data.

Multi Browser, Cross Browser & Parallel Testing using TestNG


When the time comes to turn your site from mock-up to something fully functional, you’ll want to make sure that it works great for everyone visiting your site whether they’re using Internet Explorer, Firefox, or any other browser. Testing your website with multiple combinations of browsers is known as Cross Browser testing.
Your site will look different in different browsers. That’s because browsers understand some code slightly differently. Your designer should be testing to make sure that your site works well in all modern browsers. But as a tester we need to make sure that functionality should at least tested on Internet Explorer, Firefox, Safari & Google Chrome browser.

Multi Browser Testing using Selenium TestNG

In every project it is required to perform multi-browser testing to make sure that the functionality is working as expected with every browser to give equal user experience to all of the wide range of audience. It takes a considerable time to test everything on every browser and when we have used automation to reduce the testing efforts then why don’t we perform the multi-browser testing using automation. TestNG gives us functionality to perform same test on different browsers in a simple and easy way.
How to do it…
1)Create your Script to test a LogIn application using TestNG class.
2) Pass ‘Browser Type’ as parameters using TestNG annotations to the before method of the TestNG class. This method will launch only the browser, which will be provided as parameter.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
package automationFramework;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.ie.InternetExplorerDriver;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Parameters;
import org.testng.annotations.Test;
public class MultiBrowser {
public WebDriver driver;
  @Parameters("browser")
  @BeforeClass
  // Passing Browser parameter from TestNG xml
  public void beforeTest(String browser) {
  // If the browser is Firefox, then do this
  if(browser.equalsIgnoreCase("firefox")) {
  driver = new FirefoxDriver();
  // If browser is IE, then do this   
  }else if (browser.equalsIgnoreCase("ie")) {
  // Here I am setting up the path for my IEDriver
  System.setProperty("webdriver.ie.driver", "D:\ToolsQA\OnlineStore\drivers\IEDriverServer.exe");
  driver = new InternetExplorerDriver();
  }
  // Doesn't the browser type, lauch the Website
  driver.get("http://www.store.demoqa.com");
  }
  // Once Before method is completed, Test method will start
  @Test public void login() throws InterruptedException {
driver.findElement(By.xpath(".//*[@id='account']/a")).click();
    driver.findElement(By.id("log")).sendKeys("testuser_1");
    driver.findElement(By.id("pwd")).sendKeys("Test@123");
    driver.findElement(By.id("login")).click();
}  
  @AfterClass public void afterTest() {
driver.quit();
}
}

3) Create a TestNG XML for running your test. Configure the TestNG XML for passing parameters i.e. to tell which browser should be used for Running the Test.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite" parallel="none">
<test name="FirefoxTest">
<parameter name="browser" value="firefox" />
<classes>
<class name="automationFramework.MultiBrowser" />
</classes>
</test>
<test name="IETest">
<parameter name="browser" value="ie" />
<classes>
<class name="automationFramework.MultiBrowser" />
</classes>
</test>
</suite>
 Note: You can set any number of Browsers here and just for the example purpose I have set up only two main browsers.
4) Now it’s time to run the xml. Run the test by right click on the testng.xml file and select Run As > TestNG Suite.

Note: TestNg will execute the test one by one. You may like to perform parallel tests, next topic will cover that.

Parallel Tests using TestNG

Using the feature provided by TestNG for Parallel Executions. just take the above example for Sign In application with two different browsers. This time all we want is to execute test in both browsers simultaneously.
Now just set the ‘parallel‘ attribute to ‘tests‘ in the above used xml and give a run again. This time you will notice that your both browsers will open almost simultaneously and your test will run in parallel.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite" parallel="tests">
<test name="FirefoxTest">
<parameter name="browser" value="firefox" />
<classes>
<class name="automationFramework.MultiBrowser" />
</classes>
</test>
<test name="IETest">
<parameter name="browser" value="ie" />
<classes>
<class name="automationFramework.MultiBrowser" />
</classes>
</test>
</suite>

Note: You may see some intermittent issues using parallel testing. I will not recommend you this rather run one by one only.

   

It is quite an important and the last chapter of my tutorial on TestNG. It took me a long time to figure out the perfect simple example for my viewers, as I am from a non-technical background and I hooped around so many sites to get the perfect example of listener but I was unable to find one. Everybody has pasted almost same example all over the internet. Enough talking, let’s start now.
In very non-technical term, TestNG manages everything through Suite, Test and Methods and the Listeners gives us the ability to act before and after of every Suite, Test and Methods.

Types of Listeners

There are many types of listeners available in TestNG for example IAnnotationTransformer, IAnnotationTransformer2, IConfigurable, IConfigurationListener, IConfigurationListener2, IExecutionListener, IHookable, IInvokedMethodListener, IInvokedMethodListener2, IMethodInterceptor, IReporter, ISuiteListener, ITestListener.
As far as testing concern only few can be used effectively such as :
ISuiteListener: It has two method in it onStart() & onFinish(). Whenever a class implements this listener, TestNG guarantees the end-user that it will invoke the methods onStart() and onFinish() before and after running a TestNG Suite. So before TestNG picks up your suite for execution, it first makes a call to onStart() method and runs whatever has been scripted in this method. In a similar way, it again makes a call to onFinish() method after a suite has been run.
ITestListener: The working of this listener is also exactly the same as ISuiteListerner but the only difference is that it makes the call before and after the Test not the Suite. It has seven methods in it.
onFinish(): Invoked after all the tests have run and all their Configuration methods have been called.
onStart(): Invoked after the test class is instantiated and before any configuration method is called.
onTestFailedButWithinSuccessPercentage(ITestResult result): Invoked each time a method fails but has been annotated with successPercentage and this failure still keeps it within the success percentage requested.
onTestFailure(ITestResult result): Invoked each time a test fails.
onTestSkipped(ITestResult result): Invoked each time a test is skipped
onTestStart(ITestResult result): Invoked each time before a test will be invoked.
onTestSuccess(ITestResult result): Invoked each time a test succeeds.

IInvokedMethodListener: The working of this listener is also exactly the same as ISuiteListerner & ITestListerner and the only difference is that it makes the call before and after every Method. It has only two methods in it.
afterInvocattion(): Invoke after each method
beforeInvocation(): Invoke before each method
Ok, so now am guessing that you must be pretty aware of what listeners are all about and we also saw how to write a listener. Now comes the big question.
How to do it…
1) Create a ‘New Class’ file and give it a name ‘Listener‘, by right click on the Package and select New > Class.
2) Now Implements ISuiteListener, ITestListener and IInvokedMethodListener to this newly created class. For implementing a listener class, the class has to implement the org.testng.ITestListener interface. These classes are notified at runtime by TestNG when the test starts, finishes, fails, skips, or passes.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
package utility;
import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ISuite;
import org.testng.ISuiteListener;
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestNGMethod;
import org.testng.ITestResult;
import org.testng.Reporter;
public class Listener implements ITestListener, ISuiteListener, IInvokedMethodListener {
// This belongs to ISuiteListener and will execute before the Suite start
@Override
public void onStart(ISuite arg0) {
Reporter.log("About to begin executing Suite " + arg0.getName(), true);
}
// This belongs to ISuiteListener and will execute, once the Suite is finished
@Override
public void onFinish(ISuite arg0) {
Reporter.log("About to end executing Suite " + arg0.getName(), true);
}
// This belongs to ITestListener and will execute before starting of Test set/batch
public void onStart(ITestContext arg0) {
Reporter.log("About to begin executing Test " + arg0.getName(), true);
}
// This belongs to ITestListener and will execute, once the Test set/batch is finished
public void onFinish(ITestContext arg0) {
Reporter.log("Completed executing test " + arg0.getName(), true);
}
// This belongs to ITestListener and will execute only when the test is pass
public void onTestSuccess(ITestResult arg0) {
// This is calling the printTestResults method
printTestResults(arg0);
}
// This belongs to ITestListener and will execute only on the event of fail test
public void onTestFailure(ITestResult arg0) {
// This is calling the printTestResults method
printTestResults(arg0);
}
// This belongs to ITestListener and will execute before the main test start (@Test)
public void onTestStart(ITestResult arg0) {
System.out.println("The execution of the main test starts now");
}
// This belongs to ITestListener and will execute only if any of the main test(@Test) get skipped
public void onTestSkipped(ITestResult arg0) {
printTestResults(arg0);
}
// This is just a piece of shit, ignore this
public void onTestFailedButWithinSuccessPercentage(ITestResult arg0) {
}
// This is the method which will be executed in case of test pass or fail
// This will provide the information on the test
private void printTestResults(ITestResult result) {
Reporter.log("Test Method resides in " + result.getTestClass().getName(), true);
if (result.getParameters().length != 0) {
String params = null;
for (Object parameter : result.getParameters()) {
params += parameter.toString() + ",";
}
Reporter.log("Test Method had the following parameters : " + params, true);
}
String status = null;
switch (result.getStatus()) {
case ITestResult.SUCCESS:
status = "Pass";
break;
case ITestResult.FAILURE:
status = "Failed";
break;
case ITestResult.SKIP:
status = "Skipped";
}
Reporter.log("Test Status: " + status, true);
}
// This belongs to IInvokedMethodListener and will execute before every method including @Before @After @Test
public void beforeInvocation(IInvokedMethod arg0, ITestResult arg1) {
String textMsg = "About to begin executing following method : " + returnMethodName(arg0.getTestMethod());
Reporter.log(textMsg, true);
}
// This belongs to IInvokedMethodListener and will execute after every method including @Before @After @Test
public void afterInvocation(IInvokedMethod arg0, ITestResult arg1) {
String textMsg = "Completed executing following method : " + returnMethodName(arg0.getTestMethod());
Reporter.log(textMsg, true);
}
// This will return method names to the calling function
private String returnMethodName(ITestNGMethod method) {
return method.getRealClass().getSimpleName() + "." + method.getMethodName();
}
}
3) Create a ‘New Class’ file and give it a name ‘TestListener‘, by right click on the Package and select New > Class.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
package automationFramework;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
public class TestListener {
  @Test
  public void main() {
  System.out.println("Execution of Main test is carring on");
  }
  @BeforeMethod
  public void beforeMethod() {
  System.out.println("Execution of Before method is carring on");
  }
  @AfterMethod
  public void afterMethod() {
  System.out.println("Execution of After method is carring on");
  }
}

How do I let TestNG know that I have such a listener which it should invoke when it is executing my tests ?
There are essentially two ways of adding up a listener to a particular class.
1) Implement TestNG Listener to your test class
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
package automationFramework;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
// This code will implement TestNG listeners
@Listeners(PackageName.ListenerClassName)
// For e.g. @Listeners(utility.Listener.class)
public class TestListener {
  @Test
  public void main() {
  }
}

2) Listener tag in TestNG xml: Although approach 1 is more than enough to get you started, it’s not an “elegant” way of using Listeners, because you are forced to add this @Listeners section to each of your classes, which you perhaps won’t want. So what you do is, you create a TestNG Suite xml and then add up the listeners section to this suite xml file. That way, all of your tests would essentially leverage the listener that you wrote.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
<suite name="Suite-Listeners" parallel="none">
<listeners>
<listener class-name="utility.Listener"></listener>
</listeners>
<test name="Batch-Listeners">
<classes>
<class name="automationFramework.TestListener" />
</classes>
</test>
</suite>

Output of the test case will look like this:


TestNg-Listener-0

 What is TestNG? And whats the difference between Junit and TestNG


TestNG is a testing framework inspired from JUnit and NUnit but introducing some new functionalities that make it more powerful and easier to use, such as:
  • Annotations.
  • Flexible test configuration.
  • Support for data-driven testing (with @DataProvider).
  • Support for parameters.
  • Allows distribution of tests on slave machines.
  • Powerful execution model (no more TestSuite).
  • Supported by a variety of tools and plug-ins (Eclipse, IDEA, Maven, etc...).
  • Dependent methods for application server testing.
If you want to learn TestNG you can download it and see its documentation at “http://testng.org/doc/index.html”.
TestNG is extensively used in Selenium automation and Code testing instead of Junit due to its advantages over it.

Using TestNG Selenium automation becomes simplified and also the Parallel execution using Selenium Grid becomes easy.

TestNG have a lot of features in it and hence I wont be covering every thing in my blog. But will only talk about advantages of TestNG over Junit. I will be discussing about different ways of using TestNg functions in my coming blogs.

Advantages of TestNG over Junit
  1. In Junit we have to declare @BeforeClass and @AfterClass which is a constraint where as in TestNG there is no constraint like this.
  2. Additional Levels of setUp/tearDown level are available in TestNG like @Before/AfterSuite,@Before/AfterTest and @Before/AfterGroup
  3. No Need to extend any class in TestNG.
  4. There is no method name constraint in TestNG as in Junit. You can give any name to the test methods in TestNG
  5. In TestNG we can tell the test that one method is dependent on another method where as in Junit this is not possible. In Junit each test is independent of another test.
  6. Grouping of testcases is available in TestNG where as the same is not available in Junit.
  7. Execution can be done based on Groups. For ex. If you have defined many cases and segregated them by defining 2 groups as Sanity and Regression. Then if you only want to execute the “Sanity” cases then just tell TestNG to execute the “Sanity” and TestNG will automatically execute the cases belonging to the “Sanity” group.
  8. Also using TestNG your selenium testcase execution can be done in parallel.
--------------------------------------------------------




JUnit 4 and TestNG are both very popular unit test framework in Java. Both frameworks look very similar in functionality. Which one is better? Which unit test framework should I use in Java project?
Here I did a feature comparison between JUnit 4 and TestNG.
junit-vs-testngjpg

1. Annotation Support

The annotation supports are implemented in both JUnit 4 and TestNG look similar.
FeatureJUnit 4TestNG
test annotation@Test@Test
run before all tests in this suite have run–@BeforeSuite
run after all tests in this suite have run–@AfterSuite
run before the test–@BeforeTest
run after the test–@AfterTest
run before the first test method that belongs to any of these groups is invoked–@BeforeGroups
run after the last test method that belongs to any of these groups is invoked–@AfterGroups
run before the first test method in the current class is invoked@BeforeClass@BeforeClass
run after all the test methods in the current class have been run@AfterClass@AfterClass
run before each test method@Before@BeforeMethod
run after each test method@After@AfterMethod
ignore test@ignore@Test(enbale=false)
expected exception@Test(expected = ArithmeticException.class)@Test(expectedExceptions = ArithmeticException.class)
timeout@Test(timeout = 1000)@Test(timeout = 1000)
The main annotation differences between JUnit4 and TestNG are
1. In JUnit 4, we have to declare “@BeforeClass” and “@AfterClass” method as static method. TestNG is more flexible in method declaration, it does not have this constraints.
2. 3 additional setUp/tearDown level: suite and group (@Before/AfterSuite, @Before/AfterTest, @Before/AfterGroup). See more detail here.
JUnit 4 
    @BeforeClass
    public static void oneTimeSetUp() {
        // one-time initialization code   
     System.out.println("@BeforeClass - oneTimeSetUp");
    }
TestNG
    @BeforeClass
    public void oneTimeSetUp() {
        // one-time initialization code   
     System.out.println("@BeforeClass - oneTimeSetUp");
}
In JUnit 4, the annotation naming convention is a bit confusing, e.g “Before”, “After” and “Expected”, we do not really understand what is “Before” and “After” do, and what we “Expected” from test method? TestiNG is easier to understand, it uses “BeforeMethod”, “AfterMethod” and “ExpectedException” instead.

2. Exception Test

The “exception testing” means what exception throws from the unit test, this feature is implemented in both JUnit 4 and TestNG.
JUnit 4
      @Test(expected = ArithmeticException.class)  
 public void divisionWithException() {  
   int i = 1/0;
 }
TestNG
      @Test(expectedExceptions = ArithmeticException.class)  
 public void divisionWithException() {  
   int i = 1/0;
 }



3. Ignore Test

The “Ignored” means whether it should ignore the unit test, this feature is implemented in both JUnit 4 and TestNG .

JUnit 4






        @Ignore("Not Ready to Run")  
 @Test
 public void divisionWithException() {  
   System.out.println("Method is not ready yet");
 }





TestNG






 @Test(enabled=false)
 public void divisionWithException() {  
   System.out.println("Method is not ready yet");
 }







4. Time Test

The
 “Time Test” means if an unit test takes longer than the specified 
number of milliseconds to run, the test will terminated and mark as 
fails, this feature is implemented in both JUnit 4 and TestNG .

JUnit 4






        @Test(timeout = 1000)  
 public void infinity() {  
  while (true);  
 }





TestNG






 @Test(timeOut = 1000)  
 public void infinity() {  
  while (true);  
 }







5. Suite Test

The
 “Suite Test” means bundle a few unit test and run it together.  This 
feature is implemented in both JUnit 4 and TestNG. However both are 
using very different method to implement it.

JUnit 4 

The
 “@RunWith” and “@Suite” are use to run the suite test. The below class 
means both unit test “JunitTest1” and “JunitTest2” run together after 
JunitTest5 executed. All the declaration is define inside the class.






@RunWith(Suite.class)
@Suite.SuiteClasses({
        JunitTest1.class,
        JunitTest2.class
})
public class JunitTest5 {
}





TestNG

XML
 file is use to run the suite test. The below XML file means both unit 
test “TestNGTest1” and “TestNGTest2” will run it together.






<!DOCTYPE suite SYSTEM "http://beust.com/testng/testng-1.0.dtd" >
<suite name="My test suite">
  <test name="testing">
    <classes>
       <class name="com.fsecure.demo.testng.TestNGTest1" />
       <class name="com.fsecure.demo.testng.TestNGTest2" />
    </classes>
  </test>
</suite>





TestNG
 can do more than bundle class testing, it can bundle method testing as 
well. With TestNG unique “Grouping” concept, every method is tie to a 
group, it can categorize tests according to features.  For example,

Here is a class with four methods, three groups (method1, method2 and method3)






        @Test(groups="method1")
 public void testingMethod1() {  
   System.out.println("Method - testingMethod1()");
 }  
 
 @Test(groups="method2")
 public void testingMethod2() {  
  System.out.println("Method - testingMethod2()");
 }  
 
 @Test(groups="method1")
 public void testingMethod1_1() {  
  System.out.println("Method - testingMethod1_1()");
 }  
 
 @Test(groups="method4")
 public void testingMethod4() {  
  System.out.println("Method - testingMethod4()");
 }





With the following XML file, we can execute the unit test with group “method1” only.






<!DOCTYPE suite SYSTEM "http://beust.com/testng/testng-1.0.dtd" >
<suite name="My test suite">
  <test name="testing">
   <groups>
      <run>
        <include name="method1"/>
      </run>
    </groups>
    <classes>
       <class name="com.fsecure.demo.testng.TestNGTest5_2_0" />
    </classes>
  </test>
</suite>





With
 “Grouping” test concept, the integration test possibility is unlimited.
 For example, we can only test the “DatabaseFuntion” group from all of 
the unit test classes.



6. Parameterized Test

The 
“Parameterized Test” means vary parameter value for unit test.  This 
feature is implemented in both JUnit 4 and TestNG. However both are 
using very different method to implement it.

JUnit 4

The
 “@RunWith” and “@Parameter” is use to provide parameter value for unit 
test, @Parameters have to return List[], and the parameter will pass 
into class constructor as argument.






@RunWith(value = Parameterized.class)
public class JunitTest6 {
 
  private int number;
 
  public JunitTest6(int number) {
     this.number = number;
  }
 
  @Parameters
  public static Collection<Object[]> data() {
    Object[][] data = new Object[][] { { 1 }, { 2 }, { 3 }, { 4 } };
    return Arrays.asList(data);
  }
 
  @Test
  public void pushTest() {
    System.out.println("Parameterized Number is : " + number);
  }
}





It has many 
limitations here; we have to follow the “JUnit” way to declare the 
parameter, and the parameter has to pass into constructor in order to 
initialize the class member as parameter value for testing. The return 
type of parameter class is “List []”, data has been limited to String or
 a primitive value for testing.

TestNG

XML file or “@DataProvider” is use to provide vary parameter for testing.

XML file for parameterized test.

 Only “@Parameters” declares in method which needs parameter for 
testing, the parametric data will provide in TestNG’s XML configuration 
files. By doing this, we can reuse a single test case with different 
data sets and even get different results. In addition, even end user, QA
 or QE can provide their own data in XML file for testing.

Unit Test






      public class TestNGTest6_1_0 {
 
    @Test
    @Parameters(value="number")
    public void parameterIntTest(int number) {
       System.out.println("Parameterized Number is : " + number);
    }
 
      }





XML File






<!DOCTYPE suite SYSTEM "http://beust.com/testng/testng-1.0.dtd" >
<suite name="My test suite">
  <test name="testing">
 
    <parameter name="number" value="2"/>  
 
    <classes>
       <class name="com.fsecure.demo.testng.TestNGTest6_0" />
    </classes>
  </test>
</suite>





@DataProvider for parameterized test.

While
 pulling data values into an XML file can be quite handy, tests 
occasionally require complex types, which can’t be represented as a 
String or a primitive value. TestNG handles this scenario with its 
@DataProvider annotation, which facilitates the mapping of complex 
parameter types to a test method.

@DataProvider for Vector, String or Integer as parameter






        @Test(dataProvider = "Data-Provider-Function")
 public void parameterIntTest(Class clzz, String[] number) {
    System.out.println("Parameterized Number is : " + number[0]);
    System.out.println("Parameterized Number is : " + number[1]);
 }
 
 //This function will provide the patameter data
 @DataProvider(name = "Data-Provider-Function")
 public Object[][] parameterIntTestProvider() {
  return new Object[][]{
       {Vector.class, new String[] {"java.util.AbstractList", 
"java.util.AbstractCollection"}},
       {String.class, new String[] {"1", "2"}},
       {Integer.class, new String[] {"1", "2"}}
      };
 }





@DataProvider for object as parameter
 P.S  “TestNGTest6_3_0” is an simple object with just get set method for demo.






        @Test(dataProvider = "Data-Provider-Function")
 public void parameterIntTest(TestNGTest6_3_0 clzz) {
    System.out.println("Parameterized Number is : " + clzz.getMsg());
    System.out.println("Parameterized Number is : " + clzz.getNumber());
 }
 
 //This function will provide the patameter data
 @DataProvider(name = "Data-Provider-Function")
 public Object[][] parameterIntTestProvider() {
 
  TestNGTest6_3_0 obj = new TestNGTest6_3_0();
  obj.setMsg("Hello");
  obj.setNumber(123);
 
  return new Object[][]{
       {obj}
  };
 }





TestNG’s 
parameterized test is very user friendly and flexible (either in XML 
file or inside the class). It can support many complex data type as 
parameter value and the possibility is unlimited. As example above, we 
even can pass in our own object (TestNGTest6_3_0) for parameterized test



7. Dependency Test

The
 “Parameterized Test” means methods are test base on dependency, which 
will execute before a desired method. If the dependent method fails, 
then all subsequent tests will be skipped, not marked as failed.

JUnit 4

JUnit framework is focus on test isolation; it did not support this feature at the moment.

TestNG

It use “dependOnMethods “ to implement the dependency testing  as following






        @Test
 public void method1() {
    System.out.println("This is method 1");
 }
 
 @Test(dependsOnMethods={"method1"})
 public void method2() {
  System.out.println("This is method 2");
 }





The “method2()” will execute only if “method1()” is run successfully, else “method2()” will skip the test.



Conclusion

After
 go thought all the features comparison, i suggest to use TestNG as core
 unit test framework for Java project, because TestNG is more advance in
 parameterize testing, dependency testing and suite testing (Grouping 
concept). TestNG is meant for high-level testing and complex integration
 test. Its flexibility is especially useful with large test suites. In 
addition, TestNG also cover the entire core JUnit4 functionality. It’s 
just no reason for me to use JUnit anymore.



References

TestNG
 ————
 http://en.wikipedia.org/wiki/TestNG
 http://www.ibm.com/developerworks/java/library/j-testng/
 http://testng.org/doc/index.html
 http://beust.com/weblog/

JUnit
 ———–
 http://en.wikipedia.org/wiki/JUnit
 http://www.ibm.com/developerworks/java/library/j-junit4.html
 http://junit.sourceforge.net/doc/faq/faq.htm
 http://www.devx.com/Java/Article/31983/0/page/3
 http://ourcraft.wordpress.com/2008/08/27/writing-a-parameterized-junit-test/

TestNG VS JUnit
 ——————
 http://docs.codehaus.org/display/XPR/Migration+to+JUnit4+or+TestNG
 http://www.ibm.com/developerworks/java/library/j-cq08296/index.html
 http://www.cavdar.net/2008/07/21/junit-4-in-60-seconds/
Posted by Engr. Mehedi Zaman at 11:48 PM
Email This BlogThis! Share to X Share to Facebook Share to Pinterest
Newer Post Older Post Home

About Me

Engr. Mehedi Zaman
View my complete profile

Blog Archive

  • ▼  2015 ( 23 )
    • ▼  March ( 6 )
      • Writing Tests for Appium
      • A Windows GUI for Appium
      • JUnit 4 Vs TestNG – Comparison
      • Selenium WebDriver Advanced
      • Selenium WebDriver Intro
      • Internet Explorer Driver for Selenium WebDriver
    • ►  February ( 6 )
    • ►  January ( 11 )
Powered by Blogger.