Changing context for generating better testing idea’s

As a tester one part of your job is to come up with testing idea’s.  A way that is very often used is take the template used in a previous testcase. The test cases can have the format like excel or something like As a user I want so that. It is very comfortable to do, it gives you guidance. The result is often good.

I tend to use a different approach. The tool I use to execute the test, this can be manual or automated, I will not use for brainstorming / test case generation. In a project where I use cucumber-js for checking acceptance criteria. I use a mind map for generating the testing idea’s. The tool that I use is Xmind.
I tend to start with a blank document, this is done so that I am not biased by the previous example. After this test example generation, I have a nice set of tests. Then I open a previous test to see if I have something forgotten or I need to add test from the previous test. This follows the same process (look, see, image, show) as described in on the The back of the napkin from Dan Roam.
Other great sources of tests idea’s are the domain testing workbook from Cem Kamer, testing tours, the heuristic test strategy model, You Are Not Done Yet list.

I also use the mind map in Xmind to mark al test that (in this case) are automated, see example image.
When these idea’s do not suites you an other option is to use a little technique to play with words. Let me give an example: As an site admin I want to see if my site is operational so that …

As an site admin I want to see if my site is operational.
As an site owner I want to see if the site is operational.
As an site user I want to see if the site is operational.
As an site developer I want to see if the site is operational.
As an site … I want to see if the site is operational.

As an site admin I want to get a mail if the site is operational.
As an site admin I want to get a mail if the site is not operational.
As an site admin I want to get text message if the site becomes not operational.
As an site admin I want to get text message if the site becomes operational again.
As an site admin I want to get mail if the site is degraded.
As an site admin I want to get mail if the site is slow.

You got the idea (I think), so you can create many more testing idea’s.

Tip: Try each word to changed with an inverse one, an alternative word like in this example change see into / call / text message /mail /… .

Chancing the format / the view helps to get new idea’s. Try to print and use a pen and paper. Try to go where your product is used or go outside in a park, at least go to a different room.

In my case grabbing a guitar will also help. Be surprised about what happens.

Shorten your Feedback cycle while testing.

During the last months I confronted my selve with the fact that I have material to share but where? Was it in a presentation a document or a piece of code? somewhere I have it?

I will start a serie with small post on practical things / techniques that help teams to continue. This is probably just as helpful for you then it is for me.

If your testing and you observe the status of the GUI but do you also observe the status of the backend or hardware? An example:

You have an import function. It will probably have successful and unsuccessful messages after importing. In your backend there are probably also messages of succes or not. I recommend to view live (like tail message.log) on the server if it really works. If an import seems to work on UI level it will not say that is dit not give acceptions in the backend. If you want to enhance it, you change the color of the terminal or logging window or add a sound. This will shorten the feedback cycle. The problem is that after 10 different import it is hard to sync which error occurred on which step. If an error occurs you direct see the color change. The investigation can start.

You make it visible

Below you see an example which uses this code (tailored for a mac). In the upper terminal I edit a file (message.log). I save the file and add more text this time with the word error. The lower terminal changes the color when the text error is present in the message.log file.

tail -F message.log | grep -m 1 "error";tput setab 13; clear; tail message.log

My approach on getting grip on exploratory testing

How to do a decent job on Exploratory testing. That is the question I gave my self at the beginning of this year.

What did I do.

My colleague Huib Schoot explained me his view on exploratory testing, as a context driving tester, he pointed out that there are more views on exploratory testing. Like the overview on exploratory testing by  Gitte Ottosen  See slideshare

Huib also pointed me to the course Getting a grip on exploratory testing by James Lyndsay.

It is a training that triggered me in an excellent way. I was already excited about exploratory testing but it gave me a boost. The relaxed way of teaching by James and the extreme amount of experience made it great to follow. It was a training with experienced students like me therefore the coffee breaks were also worthwhile.
James creates testmachine for understanding exploratory testing. They are great tool to let you experience that testing something this simplistic like a one button application is not always easy to test. I found out that it can be challenging to test something that simple. All the participants explained their test idea’s and together with James ideas we got the idea we have a good start.  James explained  the framework In / Out which uses inputs and outputs to discover the behavior of the program.

If you want to know more about exploratory testing it is a great course. They only downside I noticed that the available time for the course could be more. Two days for such a great training is short.

After the course it was time to practice at my daily job. After this, it was time to give training together with Huib Schoots and later give another training and a couple workshops. Giving training is a great way of learning.  At TestNet Huib and I gave a workshop and we gave a tutorial at Agile Testing Days where exploratory testing was a part of the tutorial.

The great things about giving training it lets you practice and tests your knowledge. Especially when having a colleague like Huib Schoots, he is a walking wikipedia on Exploratory testing. It is great to explore exploratory testing with a person like him.

All this gave me an experience boost on exploratory testing this year.  

User Story Risk Quadrants

On my way back from Agile Testing Days 2013 I had an conversation with Gitte Ottosen of course it was about testing. We talked a bout risk and how to address that in agile. Therefore I have a very simple but effect method to get the team more involved.

I call it User Story Risk Quadrants it is designed to let the team discus how much risk is involved with the user story.

In the past I gave presentation on risk in agile so I search for an old presentation and found out that I did not have anything online to point out to. I send her an email with an example. Her response to my email  was “I just wanted to let you know that people are really loving the risk quadrant you showed me.”

This is why I wrote this blog.

 

What is it?

It is a detailed version of a standard Product Risk Analysis or a graph to show the combination of likelihood and impact.

This can be used in a project if they want to use a risk based approach. The idea is focus on the more riskier part of the software. You can find these graphs for example in a master testplan. As in Agile there is no focus on the master testplan so I moved the addressing risk to a lower level, the user stories part. 

Why should I use it? 

I see two problems and if you like to solve them

1. The high and low risk areas are tested the same way.
2. There is no conversation about risk in the sprint

1. Isn’t it strange that risky areas are tested the same way?

There is a golden rule I use. Diversify in technique and in data. The problem sometime is that you as an tester should at least know 15 techniques to diversify. If you know one you need to learn more first.
The use of User Story Risk Quadrants is a good way to show the rest of the team that this user story will be tested more in dept and the other only by a simple error guessing. It is hard for some team member to diverse the test. Therefor this helps them to remind to diversify. Example PPT  Example key

2. As an tester you are the test mind in the team. When the risk is high the level of testing is chaining accordingly and you can have a discussion why this user story is needs more testing.

How to use it?

I start by asking the Product owner and the development team about the impact and likelihood of that user story. Add the result to the user story.  As a big fan of visualization I like put it on the wall

When to use it?

Most likely I use it during the preparation of the user story with the Product Owner. User stories are not done when they do not have a risk attribute.

And during the planning I tell the team that this user story is a quadrant 4 user story and that means these kind of tests.

Getting a plan and start building

Third iteration Getting a plan.

We known that we are allowed to build and we have an assumption that we can afford it.

The next questions that should be solved are: What is the time frame? And when should we start building.
The plan was to do the most by our selves. The plan is simple

• Remove the old shed 2 weeks.
• Brick layering 1 month
• Carpenters work  1 month
• Garden reconstruction 2 weeks.

This means a total of 3 month work.  The quality is fixed for us and we also wanted to fixed the time. The time is more or less fixed because at an certain moment you want to finnish. I hate projects of a year or longer. So with these assumption we found out we want needed help if we want to meet the deadline. We need specialist to help us. We asked a carpenter and a bricklayer to help us. Both we asked what do you think of our assumptions and are you able to help us.  The bricklayer has 40 years of experience and said it could be done in less then 2 weeks. The carpenter with 25 years of experience said between 6 and 8 weeks. Both said that we had to help them to keep the amount of hours low. (Both get paid by the hour).  Because I know both persons for a long time we made a deal with a handshake. TRUST (value people)  We had a shared understanding of what my vision is (the prototype and the drawing) With an handshake and an estimation in weeks I went home with a smile. With my neighbor we made the agreement that he should do all the non technical stuff, like carrying all the stuff to the garden  and I will do all the technical stuff. Again with a handshake.

Iteration successful with an agreement (my plan).

Iteration number four: starting to build

Everything was perfect. We had an agreement, we were  allowed, we had the money. Nothing can go wrong. But it can go wrong. We planned it to start early in 2010. During Christmas (2009 ) my neighbor fell while playing soccer with his grand child. He broke his hip. So If everything was postponed 3 months  The facts is that  surgeons can makes mistakes so he was operated 3 times and it took until the end of summer before he was recovered. After this negative delay we had positive delay. I was becoming father for the second time. That means from experience, It impacts the available time for a job like this. So it got postponed again.

Iteration four failed.

Iteration number five : Starting to build, again. 

In the summer 2012 we planned to start again. My daughter is then almost a year  and she is doing great.  My neighbor is healthy so we started.

In the first two weeks we removed the shed from me and my neighbor. We put the new concrete floor on top of the old concrete floor to make it stronger to hold the mansion. The bricklayer came and build all the walls in two weeks. My neighbor and I could help a lot and everything went fine and within the estimation of hours. We requested to build the brick wall instead of the wooden wall so we changed the “plan” a little back to the original brick wall idea. It took 2 additional days but that was not a problem. All the bricklaying was done in 2 and half week. Perfect.

everything still ok

It was time to get started with all the wood. Unlucky I had injured my shoulder badly. It slowed me down. The carpenter worked hard and there was a very interesting thing that was going on. He used the prototype and the drawing just for imagination.  He was very experienced and told me drawings are nice but this is the real world and that world changes.

It became a way of living: wakeup, go to work, come back home and go to work in the garden until late in the evening and go to sleep. Next day same patern. The idea was to finish in 8 weeks. At the end it was 8 weeks but with an huge amount of hours for my but it was done. I was committed to get it done.

Iteration 5 : done

Iteration number six : finishing . 

The mansion is finished now with even electricity and water.  I call it done but the garden is not that pretty anymore. In the next month we will see how nature survived it.

When I look back there are so many similarities with building the mansion with building software.

1. You need good people.
2. You need a clear vision of the future
3. fun
4. commitment.
5. Things will change

I was writing this while sitting on the porch.  For me is physical work great to get my brain relaxed.

Speaking at STAREAST 2013

The first time speaking on STAREAST. My talk was about thinking visual ad visualisation tools for testers. I’ve arrived there well prepared, was it good enough? I’ve ask James Bach and Michael Bolton to “test” my talk. Not only the content but also the way how I present. James was not able to make it due to an interview but he did send his brother John.  Which I thought was very nice.

I talked to Derk jan de Grood and Ruud Teunissen in the morning and both said we will go through our talk. Then I talk to James Lindsay and he said prepare your talk.  After 3 experienced speakers I’ve got the message, ill go through it one more time. I did discoverd that when I use my phone as a timer you need to set the auto lock to off.

Being well prepared I checked the room an hour before to check if I need additional extension material or connectors / converters. I also went to the presentation of Paul Holland prior to me in the room that I speak in. Paul Holland did the session about PCO, he had some audio problems. Therefore I took extra time to check that. Everything was setup the way I wanted it to be. The room was full of people the presentation could start.

It was an honor to talk to a fully pack room. The presentation went very well, the feedback should tell if it is true.  There were quite some people telling me they liked it and I’m pleased with that. Like here on twitter http://twitter.com/#!/hattsoff_inc/status/329772370531336193. Got feedback from Jon Bach and from Michael Bolton on my talk and they where positive about my presentation. A few improvements that is alway good, I am not perfect. Like forgetting to take a picture of the audience to show you.

If your interessted in the slides check them Think Different- Visualization Tools for Testers StarEast 2013 PascalDufour.

What did STAREAST bring me?

It confirmed that it is good to rehearsal and take the time to get your presentation setup.

Presentation was my main point to go to stareast, there was more that made it a great conference. I was very pleased with the invitation from James Lindsay for the test lab. Later I’ve got invited for an interview as well for the virtual conference. And I’ll already signed up for meet the speakers.  There were also a lot of very interesting people over there.  And off course there some great presentations. (In the picture  you see Derkjan Jan de Grood speaking, same room ).

James Lindsay ask me to do a session on the testlab. It was great to do a handson session. The software under test was the aloha editor. The approach I took how visualization can help you to, to understand the system.

The first step was draw a tiny PCO. This to make clear what we test. In an normal situation I would take the next steps.

1. Collect as  much information as I can
2. Try to understand relation between the gatherd information
3. Try to visualize what I want to test
4. Create a test report / create dashboard

In the testlab I did focus on one part of my approach.

After the tiny PCO the next step was pick one function of the PCO, in this case it was enabling and disabling bold text. Then we zoomed in on the feature how it works. We used a flip over to draw an overview.  We explored the design / architecture of the application by sketching together. This sketching together is an nessecary step to understanding the situation. You as a tester needs to know where it can go wrong therefor the design / architecture is important .  We must understand how it works see slide visual test strategy in my presentation. The last part of my approach is get all information I need for the test report. It is always good to know what your story is that you like to tell. It is also a check if I am missing information during testing. For example what are the average response time during testing. If you do not measure it during testing you can not put it in your report.

The testlab was for me a lot of fun.

Next time I lke to do it again!!

The interview

It was my first interview, hope I did we’ll . When I know the link I will publish it.

Meet the speaker at the table.  This time not with visualization as title but with my other favorite title developers are my agile friends.

This was for me a nice experience there were nice colleague testers that could not make it to the  presentation but had questions. We had some nice discussions on visualization. There were also two testers by acident were sitting at the table. With them I got great discussion on documenting the test. How they do it. How I do it.

It where two great days at the conference.  Next year ill like to go again.

Magic Estimation can work!

Today a team asked me on how to do a good grooming session. They were arguing on details and process.

After the team had spend an hour or two on estimation I asked them to join me in the afternoon to do a different kind of estimation session. What I did was suggesting magic estimation I did it some years before but it seemed right for this context , were the product owner wants an first estimation on his product backlog, which was filled with new epics. A couple epic with details some epics pretty vague.

I started with a little game (this to fresh up the brain). And explained how magic estimation works.

All the poker cards are on the table from 1 to 100. I gave everybody a set of stories / epics. In pure silence they lay down the stories on the table with corresponds to the correct number. After this all story have an initial estimation. Everybody had a good look at each estimation and we discussed a couple strange ones. Within 45 minutes we had estimated 30 stories / epics.

The product owner wrote down all the points and prioritized the list. Of course it is not a perfect and detailed estimation but that is not what the scrum team wanted.

DONE and a happy team.

wooden open source shed: second iteration can I afford this mansion

Second iteration: Can I afford it and am I allowed to build this mansion?

The first iteration was validating my vision.  Now the vision was successfully validated. The next step was to validate the most important question: can I afford this mansion? How do you go from a prototype to a calculation? The step I took is to ask a family friend who is an architect to draw Technical Drawing. I used this drawing to make a list of all the material needed to build this mansion. With this list I checked all the prices and calculated the total amount. This is an estimation of course. The total price was 28.000 Euro, My chopping list and price assumption spreadsheet (in Dutch) can be found here. Because I have never build a mansion like this before I added a 20% margin (risk). Why 20? I used 20% because the carpenter said is it useful. With the technical drawing I asked several companies if they could be built it and how much it would cost. The outcome was between 45.000 and 47.000 Euro. Together with my neighbors we set the budget to 25.000 Euro. Which was the savings we had. If we want to realize our mansion, we had to do a lot of work by ourselves. Maybe we had to change the design? We found out that 2 things can be changed without changing the real design to be consistent with my vision or idea. We changed from fully brick walls to a combination of bricks and wood.. The other change was the type of roof plates. This resulted in an estimation of 23.000 Euro. The first part of this iteration was done. 

Next was the validation of the second important question: am I allowed to build this kind of mansion? I expected some problems in a bureaucratic environment like the government. You make an appointment and then you are send home with a truckload of paperwork to be filled in. The form I had to fill in consisted of several checklists (entry criteria as might call them the testing community) to get a permit to allow us to start building. Within a couple of days we filled in all necessary paperwork. We used the original technical drawing.  After 6 weeks we got the good news. We were allowed to build it and for this type we did not need a permit. All the filled-in forms and the time it took became waste! Good thing that we did not change the technical drawing.

What is the relation to testing, you might think. When I test it is always in perspective to the mission or the project status. If the goal is to see if the software solution is what the costumer wants, I do not test to find minor bugs. The same thing happened here. The technical drawing seemed very accurate but in reality it is not. For example the length of the mansion seems 6 meter but in reality its only 5.5 meters. For the permit it is not a problem if you build it a bit smaller.

Now I knew I could afford the mansion (best guess) and I was allowed to build, took me to the next step. A new question arose: when is a good time to start building it?  Read it in my next blogpost: getting a plan.

Open source wooden shed

There is a great TED talk from Massimo Banzi: How Arduino is open-sourcing imagination. This talk made my self enthusiastic about open source hardware. The last half year I been working on replacing my old shed by a new shed or better said a mansion. I ve spent loads of time to find out how my mansion should be build. As a Scrum Master I work iterative and incremental at the same time. Can I use the same strategies to build my mansion. Can I embrace change during the building of my mansion. All the data I used is available here to use for free.

Here is how it went the first iteration.

The first iteration: Testing my vision.

To go from my vision to something to talk about I did create a 10 minute sketch. The result was not promising. I got to many questions, what do you mean with this and how do you mean that. It was inspired by Getting the correct image of the “complex” roof construction needed a different approach. The second step was to take was building a small paper and tape model (prototype) to get feedback on the idea. The color of is not represent reality the goal but it was to explain the roof construction not which colors to use. The mansion I like to build is together with my neighbors : the left side would be mine and the right side is my neighbors. I had to validate if they like the model? The outcome of the talk was that they liked it a lot.

We came up with more questions:

• How much will it cost to build?
• Should I build it by myself?
• Should I ask a construction company to build it?
• How long would it take to build?
• Am I aloud to build my mansion?
• How much money and am I allowed to build it.
• What are the next questions that should be answered first.

First iteration was successful the project continuous. Testing the vision for software is similar to testing my mansion vision. Prototyping / modeling all are useful.

Next iteration: Validating the most important question: Can I afford this mansion?

How was the coaching session with Anne-Marie Charrett.

When communicate I always tell others to stop using all kinds of chat. The communication could be way better if you use sound or even better talk in real life. What I had With Anne-Marie I had a more than 2 hours chat session. Was it better if I directly could talk to Anne-Marie face to face?

This is an overview on how it was.

After explaining my experience, we had a short discussion about what challenge is for me, which was very nice. I was complaining they are not challenging me. Anne-Marie great responded, o yes they are challenging you but maybe not in a positive way. After this she gave me an assignment to do. Test something. I choose the reconnection of my magic mouse. She used different techniques like polarization and drill down to questioning my task (sounds like testing). She made some great arguments to change my testing. First I thought that the brought up assumptions were not important enough. After drill down in the details I change my approach because my assumption were not good enough.

After the “test” session there was the review on the coaching session. Which gave me inside in drill down and why I made the decision to change my test approach

And there is more to learn.

The good thing is that we had chat! The chat session script was saved and stored in a text file. We both went through the script (22 pages of chat history) and annotate. The annotations where shared and reviewed. This is something that is impossible to do with face to face communication. The coaching session with chat is very helpful and reading through the session after a month you can learn from your failure and be still very happy with the answers you gave.

I can tell if you have the opportunity. Go for an coaching session you will learn valuable things.