ansible/docsite/rst/developing_test_pr.rst

Helping Testing PRs
```````````````````

If you're a developer, one of the most valuable things you can do is look at the github issues list and help fix bugs.  We almost always prioritize bug fixing over
feature development, so clearing bugs out of the way is one of the best things you can do.

Even if you're not a developer, helping test pull requests for bug fixes and features is still immensely valuable.  

This goes for testing new features as well as testing bugfixes.  

In many cases, code should add tests that prove it works but that's not ALWAYS possible and tests are not always comprehensive, especially when a user doesn't have access
to a wide variety of platforms, or that is using an API or web service.  

In these cases, live testing against real equipment can be more valuable than automation that runs against simulated interfaces.
In any case, things should always be tested manually the first time too.

Thankfully helping test ansible is pretty straightforward, assuming you are already used to how ansible works.

Get Started with A Source Checkout
++++++++++++++++++++++++++++++++++

You can do this by checking out ansible, making a test branch off the main one, merging a GitHub issue, testing, 
and then commenting on that particular issue on GitHub. Here's how:

.. note::
   Testing source code from GitHub pull requests sent to us does have some inherent risk, as the source code
   sent may have mistakes or malicious code that could have a negative impact on your system. We recommend
   doing all testing on a virtual machine, whether a cloud instance, or locally.  Some users like Vagrant
   or Docker for this, but they are optional.  It is also useful to have virtual machines of different Linux or 
   other flavors, since some features (apt vs. yum, for example) are specific to those OS versions.

First, you will need to configure your testing environment with the neccessary tools required to run our test
suites. You will need at least::

   git
   python-nosetests

Second, if you haven't already, clone the Ansible source code from GitHub::

   git clone https://github.com/ansible/ansible.git
   cd ansible/

.. note::
   If you have previously forked the repository on GitHub, you could also clone it from there.

Activating The Source Checkout
++++++++++++++++++++++++++++++

The Ansible source includes a script that allows you to use Ansible directly from source without requiring a
full installation, that is frequently used by developers on Ansible. 

Simply source it (to use the Linux/Unix terminology) to begin using it immediately::

   source ./hacking/env-setup

This script modifies the PYTHONPATH enviromnent variables (along with a few other things), which will be temporarily
set as long as your shell session is open.  

If you'd like your testing environment to always use the latest source, you could call the command from startup scripts (for example,
`.bash_profile`).

Finding A Pull Request and Checking It Out On A Branch
++++++++++++++++++++++++++++++++++++++++++++++++++++++

Next, find the pull request you'd like to test and make note of the line at the top which describes the source
and destination repositories. It will look something like this::

   Someuser wants to merge 1 commit into ansible:devel from someuser:feature_branch_name

.. note::
   It is important that the PR request target be ansible:devel, as we do not accept pull requests into any other branch.
   Dot releases are cherry-picked manually by ansible staff.

At the bottom of the GitHub page, there is a link that says "You can also merge this request on the command line".  Click this link
to expand the GitHub interface, and it will show instructions that look somewhat like this::

    Step 1: From your project repository, check out a new branch and test the changes.

    git checkout -b username-branchname branchname
    git pull git://github.com/username/branchname.git branchname

Do not follow step 2, as you don't want to merge the user features back into your branch.

.. note::
   If the GitHub user interface shows that the pull request will not merge cleanly, we do not recommend proceeding if you
   are not somewhat familiar with git and coding, as you will have to resolve a merge conflict.  This is the responsibility of
   the original pull request contributor.

.. note::
   Some users do not create feature branches, which can cause problems when they have multiple, un-related commits in
   their version of `devel`. If the source looks like `someuser:devel`, make sure there is only one commit listed on
   the pull request.

For Those About To Test, We Salute You
++++++++++++++++++++++++++++++++++++++

At this point, you should be ready to begin testing!

If the PR is a bug-fix pull request, the first things to do are to run the suite of unit and integration tests, to ensure
the pull request does not break current functionality::

   # Unit Tests
   make tests

   # Integration Tests
   cd test/integration
   make

.. note::
   Ansible does provide integration tests for cloud-based modules as well, however we do not recommend using them for some users
   due to the associated costs from the cloud providers.  As such, typically it's better to run specific parts of the integration battery
   and skip these tests.

Integration tests aren't the end all beat all - in many cases what is fixed might not *HAVE* a test, so determining if it works means
checking the functionality of the system and making sure it does what it said it would do.

Pull requests for bug-fixes should reference the bug issue number they are fixing. 

We encourage users to provide playbook examples for bugs that show how to reproduce the error, and these playbooks should be used to verify the bugfix does resolve
the issue if available.  You may wish to also do your own review to poke the corners of the change.

Since some reproducers can be quite involved, you might wish to create a testing directory with the issue # as a sub-
directory to keep things organized::

   mkdir -p testing/XXXX # where XXXX is again the issue # for the original issue or PR
   cd testing/XXXX
   <create files or git clone example playbook repo>

While it should go without saying, be sure to read any playbooks before you run them.  VMs help with running untrusted content greatly,
though a playbook could still do something to your computing resources that you'd rather not like.

Once the files are in place, you can run the provided playbook (if there is one) to test the functionality::

   ansible-playbook -vvv playbook_name.yml

If there's not a playbook, you may have to copy and paste playbook snippets or run a ad-hoc command that was pasted in.

Our issue template also included sections for "Expected Output" and "Actual Output", which should be used to gauge the output
from the provided examples.

If the pull request resolves the issue, please leave a comment on the pull request, showing the following information:

    * "Works for me!"
    * The output from `ansible --version`.

In some cases, you may wish to share playbook output from the test run as well.  

Example!::

   Works for me!  Tested on `Ansible 1.7.1`.  I verified this on CentOS 6.5 and also Ubuntu 14.04.

If the PR does not resolve the issue, or if you see any failures from the unit/integration tests, just include that output instead::

   This doesn't work for me.

   When I ran this my toaster started making loud noises!

   Output from the toaster looked like this:

      ```
      BLARG
      StrackTrace
      RRRARRGGG
      ```

We understand some users may be inexperienced with git, or other aspects of the above procedure, so feel free to stop by ansible-devel
list for questions and we'd be happy to help answer them.