One Simple Test Automation Scenario Interview Question that Most Candidates Failed

Simple, yet every candidate I interviewed failed.

This article is one of the “Be aware of Fake Test Automation/DevOps Engineers” and the “IT Job Interview” series.

After the four screening questions, I usually ask one scenario question during the interview (otherwise, too short, not respecting candidates). Personally, I don’t like long, brain-teasers or tricky questions. So, this only test automation knowledge question must be related to real work.

The Question

Describe the steps you would write for a common E2E web automated test case via UI: “User Change Password”.

Everyone understands this simple scenario, the explanation for business logic is unnecessary. It is so common that this would be among the first batch of automated tests written for a web app. To assure the candidate, I will tell the candidate: “There is no trick here, and no functional limitations (ignore any password change restrictions). This is simply about test design. Just say or write down the steps, such as a user logs in, click profile and ‘change password’ link, …, etc.”

Before the candidate started, I said: “A reminder, this is an automated End-to-End test via UI (functional testing). Don’t overthink, just focus on designing a good automated test that verifies that the user can change password.”

The Answers: Most Got it Wrong!

It is a quick and easy interview question, right? However, not a single interview candidate answered it correctly (regardless of the candidates’ current job titles, such as Senior Test Automation Engineer or Test Automation Architect). Also, I asked this question in my training session and in some conference talks. I’ve only gotten the correct answer once (yelled from the audience at STARWEST 2013 in California; of course, there might be some real test automation engineers, but many seem to prefer to remain silent).

Some readers wonder, how could there be such a high percentage of wrong answers. Is there a trick? NO, definitely not.

Let’s see a typical answer from the interview candidates:

User logs in
Click the ‘Change Password’ link
Enter the current password
Enter a new password and confirmation
Click the ‘Change Password’ button
Log out
Login in with the new password
Confirm the logged-in successfully

Often, candidates expressed the step in detail (fear there were some tricks) or validations such as confirmed-password-matching. Then I would assure them, don’t worry about validations, alternative paths or functional limits. Just focus on a completed automated test design for the main flow. The more concise (being correct), the better.

Some even stopped at Step 5. But most could get Step 8.

Then I would ask: “Is that all?”

Usually, the candidate would show signs of fear of missing something. Then I will assure them again: “There is no trick, just write down or speak out how you normally test this common scenario?”

Intentionally left blank.

So motivated readers can work out the answer.

However, no one candidate could work out the missing step:

9. Change the password back!

This is just one of many possible solutions (four of them by my daughter). For readers who don’t yet understand why, please stay with this solution for a moment.

# revert the password
driver.find_element(:link, "Change Password").click
driver.find_element(:id, "current_pass").send_keys("def456") # new
driver.find_element(:id, "new_pass").send_keys("abc123")     # old
driver.find_element(:id, "new_pass_confirm").send_keys("abc123")
driver.find_element(:id, "change-btn").click

I used the raw Selenium WebDriver (Ruby) in the above test script. Normally, I would use page objects as in the Maintainable Automated Test Design (easier to read and maintain).

Some might think of alternative ways rather than ‘changing the password back’. Check out the answers (4 of them) by my daughter. But the purpose of this question is NOT to solve this individual test case, but rather a quick check whether the candidate has a good test automation mindset.

I explained: “We are talking about automated testing. After the password of a test user (e.g. manager01) was changed from abc123 to def456 and your automated test passed. What will happen if someone runs your test again?”

It would fail, of course: unable to log in (step 1). Once I revealed it, most candidates understood it immediately.

From my memory, no one (interview candidates or audience at various talks) debated me on this, as it is so obvious.

This scenario question exposed that none of the candidates (I interviewed) had done real test automation. There are many reasons, the №1 is that they don’t understand:

Automated functional tests must be executed many times to be useful. (there is little value in doing a throw-away automated test script, right?)

If you got it right, Congrats! You are better than 95+% automated testers based on my observations (for over 15 years). To self-assess your test automation levels objectively, check out Definition of End-to-End Test Automation Success, or AgileWay Continuous Testing Grading.

Then I would ask casually: “How would you do this in manual testing?”

After a short pause, I continued: “As a matter of fact, even in manual testing, we need that step as well. But often, manual testers will do it as a preparation step beforehand (Step 0, instead of Step 9) intentionally or unintentionally. However, in automated testing, this step is clearly a mandatory part of the test script.”

This is a minor complication with Step 9

Making an automated test highly reliable requires effort, quite often, a big effort. Check out the article “Working Automated Test ≠ Good Reliable Test”; software teams need guidance from a real test automation coach. Consider this, Step 9 consists of a few minor steps (as shown above) to revert the password, what if one of the minor steps failed in execution? As we know, this could happen in UI Testing.

This would mean, manager01 ’s password could be stuck with the new password, intermittently. Any other tests with the use of manager01 would fail, i.e. your next run of regression testing in a CI/CT server could fail.

See, real test automation is not easy if do it seriously. I will write another article to address test-data-related issues in test automation.

Lessons learned from this exercise

First of all, I declare that I am not belittling those candidates or many automated testers who were struggling. This is the failure of our IT education on test automation: Universities do not teach; Companies do not provide training. Motivated engineers can learn, though. In fact, I hired some (they had all failed on this question), and they learned well under my coaching: can work on real tests in a matter of days.

I was only aware of Automated End-to-End Test Automation in 2005, after working as a senior (contract) Java programmer for eight years. I was extremely lucky to receive mentoring from a world-renowned software engineer, for just a few weeks. I learned test automation and CI from him, which changed my professional career.

There is no shame for software engineers in admitting “unable to do automated UI testing”. Again, it is a failure of education. Just Learn It. However, I despise fake automated testers who sabotage real test automation because of fear or trying different test automation frameworks for their resumes, like B in this real story: An IT Graduate’s frustration with a Fake ‘Senior Test Automation Engineer’.

Here are what I learned from candidates’ answering this scenario question:

Real Test Automation Engineers are extremely rare. Most IT people have never witnessed one successful test automation in their careers, now you should know why. Check out this Wired article: “The Software Revolution Behind LinkedIn’s Gushing Profits”. LinkedIn needed to lure one real automation engineer (Kevin Scott) from Google to implement automated testing that enables daily production releases. Many automated testers mistakenly thought, “create an automated test and run once OK”, after that, the test automation task is done. Far from that! Running all test scripts frequently (as regression testing) and keeping them valid is where the real test automation work is. That’s another reason to have a practical test automation strategy, such as the 35-Word Functional Test Automation Strategy. With commonly little knowledge among test automation engineers, as presented by this simple interview question, planning big test automation is often a total waste of time. Train your automated testers first.
Talking about test automation tools/languages is meaningless if you don’t understand the nature of automated testing. There are many ‘automated testers’ like talking about test automation, such as which framework is better, which tool is cool, …, etc. As you can see, if (fake) automated testers don’t really understand test automation, failure is inevitable, regardless of which tool/framework/scripting language.
Don’t worry about hiring medium/senior test automation engineers. Hire new graduates and get a real test automation coach to train them. Even having low expectations, I was still surprised by the 100% failure rate in many interviews (in my city) I conducted. That’s also the reason why, if the management seeks my opinion, I don’t suggest hiring so-called experienced automated testers. Instead, I prefer training fresh new graduates. WIN-WIN. The company saves money (lower salary) and gets right-mind-set automated testers quickly. After my one-day training, new automated testers can start work on real tests the next day, under my mentoring.
Most so-called CI/CD or CT processes are wrong or incomplete. CI/CD is not very useful if there are no running automated End-to-End tests (unable to achieve daily production delivery if just unit testing). With proper CI/CD, defects for new code and regression errors would be discovered quickly, and frequently. For more, check out Testing Pyramid Clarified.

Facebook’s Testing Pyramid, pyramid image credit: “Continuous Integration at Facebook” presentation

Related reading:

My eBooks: - “Practical Web Test Automation with Selenium WebDriver” - “Practical Continuous Testing: make Agile/DevOps real”
Web Test Automation with Selenium WebDriver Training workbook series. suitable for 10+ years old without any software knowledge. Motivated learners can achieve >90% automated testers after 50 short (~20 mins) sessions.
Working Automated Test ≠ Good Reliable Test a recommended reading after this. A case study of my daughter’s working automated test (prepared for an interview), but not reliable.
Case Study: User Change Password Automated Test
Fake End-to-End Test Automation Clarified
An IT Graduate’s frustration with a Fake ‘Senior Test Automation Engineer’
A Story of a Veteran Manual Tester’s Learning Test Automation from a Coach: “One Question per Day”
35-Word Functional Test Automation Strategy
How to Implement Real Automated Regression Testing?
Testing Pyramid Clarified
Why Do Most UI Test Automation Fail? Part 1: Wrong Automation Framework, Part 2: Wrong Choice of Test Syntax Framework, Part 3: Wrong Scripting Language (more to come)