* Fix unit test parametrize order on Python 3.5.
(cherry picked from commit 53b230ca74)
* Fix ansible-test unit test execution. (#45772)
* Fix ansible-test units requirements install.
* Run unit tests as unprivileged user under Docker.
(cherry picked from commit 379a7f4f5a)
* Run unit tests in parallel. (#45812)
(cherry picked from commit abe8e4c9e8)
* Minor fixes for unit test delegation.
(cherry picked from commit be199cfe90)
* add support for opening shell on remote Windows host (#43919)
* add support for opening shell on remote Windows host
* added arg completion and fix sanity check
* remove uneeded arg
(cherry picked from commit 6ca4ea0c1f)
* Block network access for unit tests in docker.
(cherry picked from commit 99cac99cbc)
* Make ansible-test available in the bin directory. (#45876)
(cherry picked from commit f3d1f9544b)
* Support comments in ansible-test flat files.
(cherry picked from commit 5a3000af19)
* Fix incorrect use of subprocess.CalledProcessError (#45890)
(cherry picked from commit 24dd87bd0a)
* Improve ansible-test match error handling.
(cherry picked from commit 2056c981ae)
* Improve error handling for docs-build test.
(cherry picked from commit 2148999048)
* Bug fixes and cleanup for ansible-test. (#45991)
* Remove unused imports.
* Clean up ConfigParser usage in ansible-test.
* Fix bare except statements in ansible-test.
* Miscellaneous cleanup from PyCharm inspections.
* Enable pylint no-self-use for ansible-test.
* Remove obsolete pylint ignores for Python 3.7.
* Fix shellcheck issuers under newer shellcheck.
* Use newer path for ansible-test.
* Fix issues in code-smell tests.
(cherry picked from commit ac492476e5)
* Fix integration test library search path.
This prevents tests from loading modules outside the source tree,
which could result in testing the wrong module if a system-wide
install is present, or custom modules exist.
(cherry picked from commit d603cd41fe)
* Update default container to version 1.2.0.
(cherry picked from commit d478a4c3f6)
(cherry picked from commit 21c4eb8db5)
* Fix ansible-test docker python version handling.
This removes the old name based version detection behavior and
uses versions defined in the docker completion file instead, as
the new containers do not follow the old naming scheme.
(cherry picked from commit 54937ba784)
* Reduce noise in docs-build test failures.
(cherry picked from commit 4085d01617)
* Fix ansible-test encoding issues for exceptions.
(cherry picked from commit 0d7a156319)
* Fix ansible-test multi-group smoke test handling. (#46363)
* Fix ansible-test smoke tests across groups.
* Fix ansible-test list arg defaults.
* Fix ansible-test require and exclude delegation.
* Fix detection of Windows specific changes.
* Add minimal Windows testing for Python 3.7.
(cherry picked from commit e53390b3b1)
* Use default-test-container version 1.3.0.
(cherry picked from commit 6d9be66418)
* Add file exists check in integration-aliases test.
(cherry picked from commit 33a8be9109)
* Improve ansible-test environment checking between tests. (#46459)
* Add unified diff output to environment validation.
This makes it easier to see where the environment changed.
* Compare Python interpreters by version to pip shebangs.
This helps expose cases where pip executables use a different
Python interpreter than is expected.
* Query `pip.__version__` instead of using `pip --version`.
This is a much faster way to query the pip version. It also more
closely matches how we invoke pip within ansible-test.
* Remove redundant environment scan between tests.
This reuses the environment scan from the end of the previous test
as the basis for comparison during the next test.
(cherry picked from commit 0dc7f38787)
* Add symlinks sanity test. (#46467)
* Add symlinks sanity test.
* Replace legacy test symlinks with actual content.
* Remove dir symlink from template_jinja2_latest.
* Update import test to use generated library dir.
* Fix copy test symlink setup.
(cherry picked from commit e2b6047514)
* Fix parametrize warning in unit tests.
(cherry picked from commit 1a28898a00)
* Update MANIFEST.in (#46502)
* Update MANIFEST.in:
- Remove unnecessary prune.
- Include files needed by tests.
- Exclude botmeta sanity test.
These changes permit sanity tests to pass on sdist output.
(cherry picked from commit cbb49f66ec)
* Fix unit tests which modify the source tree. (#45763)
* Fix CNOS unit test log usage.
* Use temp dir for Galaxy unit tests.
* Write to temp files in interfaces_file unit test.
* Fix log placement in netapp_e_ldap unit test.
(cherry picked from commit 0686450cae)
* Fix ansible-test custom docker image traceback.
(cherry picked from commit 712ad9ed64)
* ansible-test: Create public key creating Windows targets (#43760)
* ansible-test: Create public key creating Windows targets
* Changed to always set SSH Key for Windows hosts
(cherry picked from commit adc0efe10c)
* Fix and re-enable sts_assume_role integration tests (#46026)
* Fix the STS assume role error message assertion when the role to assume does not exist.
(cherry picked from commit 18dc928e28)
* Fix ACI unit test on Python 3.7.0.
The previous logic was only needed for pre-release versions of 3.7.
(cherry picked from commit c0bf9815c9)
* Remove placeboify from unit tests that are not calling AWS (i.e. creating a recording) (#45754)
(cherry picked from commit 2167ce6cb6)
* Update sanity test ignore entries.
* Do not join flag parameters
This put a comma between every character of the tcp flag parameters, resulting in a bad iptables command.
Fixes#36490
* Use suboptions to ensure tcp_flags options are lists
* Add unit tests for tcp_flags
* Add example of how to use tcp_flags
(cherry picked from commit c9d3bb59a4)
The generated file was completely unusable by the system
therefore the fix which ensures that diffing the file
prior to changes and after only shows diffs
Furthermore the code did not work for Python 3.6
> f.writelines(to_bytes(lines, errors='surrogate_or_strict'))
E TypeError: a bytes-like object is required, not 'int'
The other modifications (lambda variable renaming) is to
comply with default flake8 rules
(cherry picked from commit 612d0d6634)
* When inserting a new rule in `insert_after_rule`, check if the old rule is
the last rule, to avoid a list index out of range error when attempting to
access the next rule.
* Add a test for inserting a new rule after the last rule.
* Added . and / to rule args regexp
Things like pam_echo.so file=/etc/foo.txt weren't being matched and
causing incorrect change counts. Adding / and . fixed that.
Fixes#33351
* pamd: test argument with value
Relates #33351
* move set_module_args to units.modules.utils
* unit tests: reuse set_module_args
* unit tests: mock exit/fail_json in module.utils.ModuleTestCase
* unit tests: use module.utils.ModuleTestCase
* unit tests: fix 'import shadowed by loop variable'
* Add some tests for iptables
* Fix remove bug (calls 2 times check to remove a chain)
* Add me as maintainer
* Fix PEP8
* Doc: Give more information on issue #18988
* Fix#18988 and test it
* Fix doc (thanks Pillou)
* enable PEP8 check for iptables
* Tidy mount module for testing
Fix spelling mistakes in comments. I *think* the example for omitting parents
root has the wrong parent ID.
Make mountinfo file a parameter for testing.
* Don't strip leading slash from mounts
The current code does not follow the example, it produces src=tmp/aaa instead
of src=/tmp/aaa. This causes problems with bind mounts under /rootfs.
* Use dictionary to store mounts by ID
Instead of looping over each one to check if the ID matches. This does not
preserve the order of the output on < Python3.6, but that is not necessary.
* Make linux_mounts a dict
Always accessed by 'dst', so avoid looping by just making it a key.
* Add test case for get_linux_mounts
* Added new module interfaces_file
* interfaces_file: added unit tests
* interfaces_file: added golden files for unit tests
* interfaces_file: moved to system modules
* interfaces_file: fixed code formatting and convention issues
* Add unit tests for parted module
Test the current expected behavior of the module:
- mock parted() and get_device_info()
- Use some of the examples and test the 'script' passed to parted.
- mock check_parted_label() to return false, as if parted version is > 3.1
- assert get_device_info output is correct
Current implementation of the module runs parted several time while going
through all parameters (flags, name, ...). Between calls it uses get_device_info
to update the dictionary. Use check_mode for some of the tests to force module
to go through all the parameters even is dictionary is not updated.
* test_parted.py: add "name" param into expected results
since 78fff751ab, parse_partition_info
fetch the partition name. This commit adds 'name' key and value into
the expected results.
It was in lib/ansible/modules/system/setup.py since it
was the only thing using it, but move it back to module_utils
and add a ansible_collector.get_ansible_collector() to build
a facts collector just like the one used by setup.py
mv test_setup.py -> test_ansible_collector.py
All the code it was testing is now in ansible_collector
rm code to create 'ansible_facts' subkey from namespace
Just leave it up to the caller to do, and just return a
flat dictionary from AnsibleFactCollector.collect()
Facts Refresh (2.4 roadmap)
This commit implements most of the 2.4 roadmap 'Facts Refresh'
- move facts.py to facts/__init__.py
- move facts Distribution() to its own class
- add a facts/utils.py
- move get_file_content and get_uname_version to facts/utils.py
- move Facts() class from facts/__init__ to facts/facts.py
- mv get_file_lines to facts/utils.py
- mv Ohai()/Facter() class to facts/ohai.py and facter.py
- Start moving fact Hardware() classes to facts/hardware/*.py
- mv HPUX() hardware class to facts/hardware/hpux.py
- move SunOSHardware() fact class to facts/hardware/sunos.py
- move OpenBSDHardware() class to facts/hardware/openbsd.py
- mv FreeBsdHardware() and DragonFlyHardware() to facts/hardware/
- mv NetBSDHardware() to facts/hardware/netbsd.py
- mv Darwin() hardware class to facts/hardware/darwin.py
- pep8/etc cleanups on facts/hardware/*.py
- Mv network facts classes to facts/network/*.py
- mv Virtual fact classes to facts/virtual
- mv Hardware.get_sysctl to facts/sysctl.py:get_sysctl
- Also mv get_uname_version from facts/utils.py -> distribution.py
since distribution.py is the only thing using it.
- add collector.py with new BaseFactCollector
- add a subclass for AnsibleFactCollector
- hook up dict key munging FactNamespaces
- add some test cases for testing the names of facts
- mv timeout stuff to facts.timeout
- rm ansible_facts()/get_all_facts() etc
- Instead of calling facts.ansible_facts(), fact collection
api used by setup.py is now to create an AnsibleFactCollector()
and call it's collect method.
- replace Facts.get_user_facts with UserFactCollector
- add a 'systems' facts package, mv UserFactCollector there
- mv get_dns_facts to DnsFactCollector
- mv get_env_facts to EnvFactCollector
- include the timeout length in exception message
- modules and module_utils that use AnsibleFactCollector
can now theoretically set the 'valid_subsets'
May be useful for network facts module that currently have
to reimplement a good chunk of facts.py to get gather_subsets
to work.
- get_local_facts -> system/LocalFactCollector
- get_date_time -> system/date_time.py
- get_fips_facts -> system/fips.py
- get_caps_facts() -> system/caps.py
- get_apparmor_facts -> system/apparmor.py
- get_selinux_facts -> system/selinux.py
- get_lsb_facts -> system/lsb.py
- get_service_mgr_facts -> system/service_mgr.py
- Facts.is_systemd_managed -> system/service_mgr.py
- get_pkg_mgr_facts -> system/pkg_mgr.py
- Facts()._get_mount_size_facts() -> facts.utils.get_mount_size()
- add unit test for EnvFactCollector
- add a test case for minimal gather_subsets
- add test case for collect_ids
- Make gather_subset match existing behavior or '!all'
If 'gather_subset' is provided as '!all', the existing behavior
(in 2.2/2.3) is that means 'dont collect any facts except those
from the Facts() class'. So 'skip everything except
'apparmor', 'caps', 'date_time', 'env', 'fips', 'local', 'lsb',
'pkg_mgr', 'python', 'selinux', 'service_mgr', 'user', 'platform', etc.
The new facts setup was making '!all' mean no facts at all, since
it can add/exclude at a finer granularity. Since that makes more
sense for the ansible collector, and the set of minimal facts to
collect is really more up to setup.py to decide we do just that.
So if setup.py needs to always collect some gather_subset, even
on !all, setup.py needs to have the that subset added to the
list it passes as minimal_gather_subset.
This should fix some intg tests that assume '!all' means that
some facts are still collected (user info and env for example).
If we want to make setup.py collect a more minimal set, we can do that.
- force facts_dicts.keys() to a list so py3 works
- split fact collector tests to test_collectors.py
- convert Facter(Facts) -> other/facter.py:FacterFactCollector
- add FactCollector.collect_with_namespace()
regular .collect() will return a dict with the key names
using the base names ('ip_address', 'service_mgr' etc)
.collect_with_namespace() will return a dict where the key names
have been transformed with the collectors namespace, if there is
one. For most, this means a namespace that adds 'ansible_' to the
start of the key name.
For 'FacterFactCollector', the namespace transforms the key to
'facter_*'.
- add test cases for collect_with_namespace
- move all the concrete 'which facts does setup.py' stuff to setup.py
The caller of AnsibleFactCollector.from_gather_subset() needs to
pass in the list of collector classes now.
- update system/setup.py to import all of the fact classes and pass
in that list.
- split the Distribution fact class up a bit
extracted the 'distro release' file handling (ie, linux
boxes with /etc/release, /etc/os-release etc) into its
own class.
- extract get_cmdline_facts -> cmdline.py
- extract get_public_ssh_host_keys -> system/ssh_pub_keys.py
- extract get_platform_facts -> system/platform.py
platform.py may be a good candidate for further splitting.
- rm test for plain Facts() base class
- let the base class for Collector unit tests provide collected_facts
some Collectors and/or their migrated Facts() subsclasses need
to look at facts collected by other modules ('ansible_architecture'
the main one...).
Collector.collect() has the collected_facts arg for this, so add
a class variable to BaseFactsTest so we can specify it.
- mv Ohai to other/ohai.py and convert to Collector
- update hardware/*.py to return facts (no side effects)
- mv AnsibleFactCollector to setup.py
- extra collector class gathering to module method in
facts/__init__.py (collector_classes_from_gather_subset)
- add a CollectorMetaDataCollector collector used to provide
the 'gather_setup' fact
- add unit test module for 'setup' module
(test/units/modules/system/setup.py)
- Collector init now doesnt need a module, but collect does
An instance of a FactCollector() isnt tied to a AnsibleModule
instance, but the collect() method can be, so optionally pass
in module to FactCollector.collect() (everywhere)
- add a default_collectors for list of default collectors
import and use it from setup.py module
eventually, would like to replace this with a plugin loader
style class finder/loader
- unit tests for module_utils/facts/__init__.py
- add unit tests for ohai facts collector
- remove self.facts side effect on populate() in hardware/sunos.py
- convert OpenBSDHardware() to rm side effects on self.facts
- try to rm some self.facts side effects in Network()
plumb in collected_facts from populate() where it is needed.
stop passing collected_facts into Network() [via cached_facts=,
where it eventually becomes self.facts]
- nothing provides Fact() cached_facts arg now, rm it
Facts() should be internal only implementation so nothing
should be using it.
Of course, now someone will.
- add a Collector.name attr to build a map of name->_fact_ids
To properly exclude a gather_subset spec like '!hardware', we
need to know that 'hardware' also means 'devices', 'dmi', etc.
Before, '!hardware' would remove the 'hardware' collector name
but not 'devices'. Since both would end up in id_collector_map,
we would still end up with the HardwareCollector in the collector
list. End result being that '!hardware' wouldn't stop hardware
from being collected.
So we need to be able to build that map, so add the Collector.name
attribute that is the primary name (like 'hardware') and let
Collector._fact_ids be the other fact ids that a collector is
responsible for.
Construct the aliases_map of Collector.name -> set of _fact_ids
in fact/__init__.py get_collector_names, and use it when we are
populating the exclude set.
- refactor of distribution.py
make the big OS_FAMILY literal a little easier to read
Also keys can now be any string instead of python literals
99% sure the test for 'KDE Neon' was wrong
I don't see how/where it should or could get 'Neon' instead
of 'KDE Neon' as provided in os-release NAME=
Use 'distribution' string for key to OS_MAP
ie, we dont need to make it a valid python label anymore so dont.
move _has_dist_file to module as _file_exists
easier to mock without mucking with os.path
mv platform.system() calls to within get_distribution_facts() instead
of Distribution() init.
- remove _json compat module
The code in here was to support:
-a 'json' python module that was not the standard one included
with python since 2.6.
- potentially fallback to simplejson if 'json' was not available.
'json' is available for all supported python versions now so
no longer needed.
- mv get_collector_names -> facts.collector
- mv collector_classes_from_gather_subset -> facts.collector
- mv collector tests from test_facts -> test_collector
- Use six's reduce() in sunos/netbsd hardware facts
- rm extraneous get_uname_version in utils
only system/distribution.py uses it
- Remove Facts() subclass metaclass usage
- using fact_id and a platform id for matching collectors
gut most of Facts() subclasses
rm Facts() subclasses with weird metaclass
only add collectors that match the fact_ids and the platform_info
to the list of collectors used.
atm, a collectors platform_id will default to 'Generic', and
any platform matches 'Generic'
goal is to select collector classes including matching the
systems platform in collector.py, instead of relying on
metaclasses in hardware/*. To finish this, the various
Facts() subclasses will need to be replaced entirely with
Collector() subclasses.
use collector classmethod platform_match() to match the platform
This lets the particular class decide if it is compatible with
a given platform_info. platform_info is a dict like obj, so it could be
expanded in the future.
Add a default platform_match to BaseFactCollector that matches
platform_info['system'] == cls._platform
They were needed previously to trigger a module
load on all the collector classes when we import
facts/hardare so that the Hardware() and related
classes that used __new__ and find_all_subclasses()
would work.
Now that is done in collectors based on platform matching
at runtime we dont need to do it py module import/parse
time. So the non empty __init__.pys are no longer needed
and their is a more flexible mechanism for selection
platform specific stuff.
facts/facts.py is no longer used, rm'ed
- if we dont find an implement class for gather spec.. just ignore it.
Would be useful to add a warn to warn about this case.
- Fix SD-UX typo (should be HP-UX)
- Port fix for #21893 (0 sockets) to this branch
This readds the change from 8ad182059d
that got lost in merge/rebase
Fixes#21893
- port sunos fact locale fix for #24542 to this branch
based on e558ec19cdFixes#24542
Solaris fact fix (#24793)
ensure locale for solaris fact gathering
fixes issue with locale interfering with proper reading of decimals
- raise exceptions in the air like we just dont care.
Pretty much ignore any not exit exception in facts
collection. And add some test cases.
- added new selinux fact to clarify python lib
the selinux fact is boolean false when the library is not installed,
a dictionary/hash otherwise, but this is ambigous
added new fact so we can eventually remove the type dichtomy and normalize it as a dict
Re-add of devel commit 85c7a7b844 to
the new code layout, since it got removed in merge/rebase
* test/: PEP8 compliancy
- Make PEP8 compliant
* Python3 chokes on casting int to bytes (#24952)
But if we tell the formatter that the var is a number, it works
* known_hosts: support --diff
* known_hosts: support --diff also without --check
* Add unit tests and fix incorrect diff in one corner case
Tests are good!
* Refactor for readability
* Python 3 compat
* More Python 3 compat
* Add an integration test for known_hosts
* Handle ssh-keygen -HF returning non-zero exit code
AFAICT this is a bug in ssh-keygen in some newer OpenSSH versions
(>= 6.4 probably; see commit dd9d5cc670):
when you invoke ssh-keygen with -H and -F <host> options, it always
returns exit code 1. This is because in ssh-keygen.c there's a function
do_known_hosts() which calls
exit (find_host && !ctx.found_key);
at the end, and find_host is 1 (because we passed -F on the command line),
but ctx.found_key is always 0. Why is found_key always 0? Because the
callback passed to hostkeys_foreach(), which is known_hosts_hash(),
never bothers to set found_key to 1.
* This test does not need root
* Avoid ssh-ed25519 keys in sample known_hosts file
Older versions of OpenSSH do not like them and ssh-keygen -HF
aborts with an error when it sees such keys:
line 5 invalid key: example.net...
/root/ansible_testing/known_hosts is not a valid known_hosts file.
* Fix Python 3 errors
Specifically, the default mode of tempfile.NamedTemporaryFile is 'w+b',
which means Python 3 wants us to write bytes objects to it -- but the
keys we have are all unicode strings.
