* s3_bucket: fix policy sorting for python3 so strings are evaluated as less than tuples.
Add tests to ensure this behavior is maintained.
* Fix s3_bucket comparison function to work on both Python 3.5 and 3.6
* s3_bucket: document that cmp_to_key is used for python 2.7.
Add another test for s3_bucket to compare policies of different sizes.
* fix pep8
* Work around code-smell grepping by not using the word 'cmp'.
* New module for managing AWS Datapipelines
* Supports create/activate/deactivate and deletion
* Handles idempotent creation by embeding the version in the
uniqueId field
* Waits for requested state to be reached, as Botocore doesn't
have waiters yet for datapipelines
* rename module, fix imports, add tags option, improve exit_json results, fix a couple bugs, add a TODO so I don't forget
fix pep8
allow timeout to be used for pipeline creation
make .format syntax uniform
fix pep8
fix exception handling
allow pipeline to be modified, refactor, add some comments, remove unnecessary imports
pipeline activation may not be in the activated state long
remove datapipeline version option
change a loop to a list comprehension
create idempotence by hashing the options given to the module minus the objects (which can be modified)
small bugfix
* data_pipeline unittests
make unittests pep8
fix bug in unittests
* remove exception handling that serves no purpose
* Fix python3 incompatibilities in datapipeline tests and add placebo fixture maybe_sleep for faster tests
Fix python3 incompatibilities in data_pipeline build_unique_id()
Don't delete a pipeline in diff_pipeline() because it's unexpected
Don't use time.time() because it causes an issue with placebo testing
re-recorded tests
fix pep8 in data_pipeline
Remove disable_rollback from tests
Make sure unique identifier is a string
re-record tests
* improve documentation and add another example
* use a placebo fixture instead of redundant code in tests
fix tests for PLACEBO_RECORD=false
* Fix data_pipeline docs
use isinstance instead of type()
fix documentation
* fix documentation
* Remove use of undefined variable from data_pipeline module and fix license
* fix copyright header
* Initial commit for integration of HPE OneView resources with Ansible Core. Adding FC Network and FC Network Fact modules and unit tests, and OneView base class for all OV resources.
previously gather_subset=['!all'] would still gather the
min set of facts, and there was no way to collect no facts.
The 'min' specifier in gather_subset is equilivent to
exclude the minimal_gather_subset facts as well.
gather_subset=['!all', '!min'] will collect no facts
This also lets explicitly added gather_subsets override excludes.
gather_subset=['pkg_mgr', '!all', '!min'] will collect only the pkg_mgr
fact.
* Add module cv_server_provision for integration with Arista CloudVision Portal.
* Doc update.
* Remove shebang from test file. Update short description with company and product name.
* Update exception syntax to Python3 style.
* Remove blank line between imports.
* Remove newlines from RETURN documentation.
* Add cvprac to unittest requirements.
* Update unittest format. Add a few additional tests.
* Mock exceptions from cvprac so the library is not needed for unittests.
* Mock cvprac imports.
* Update unit tests to support python 3.5.
* Mock full cvprac library for unittests.
* Update Jinja2 import to pass updated CI checks.
* Update cvprac imports format for new CI tests.
* Add __metaclass__ and __future__.
When parsing a vaulttext blob, use .splitlines()
instead of split(b'\n') to handle \n newlines and
windows style \r\n (CRLF) new lines.
The vaulttext enevelope at this point is just the header line
and a hexlify()'ed blob, so CRLF is a valid newline here.
Fixes#22914
* Added new module interfaces_file
* interfaces_file: added unit tests
* interfaces_file: added golden files for unit tests
* interfaces_file: moved to system modules
* interfaces_file: fixed code formatting and convention issues
* Add 2.0-2.3 facts api compat (ansible_facts(), get_all_facts())
These are intended to provide compatibilty for modules that
use 'ansible.module_utils.facts.ansible_facts' and
'ansible.module_utils.facts.get_all_facts' from 2.0-2.3 facts
API.
Fixes#25686
Some related changes/fixes needed to provide the compat api:
* rm ansible.constants import from module_utils.facts.compat
Just use a hard coded default for gather_subset/gather_timeout
instead of trying to load it from non existent config if the
module params dont include it.
* include 'external' collectors in compat ansible_facts()
* Add facter/ohai back to the valid collector classes
facter/ohai had gotten removed from the default_collectors
class used as the default list for all_collector_classes by
setup.py and compat.py
That made gather_subset['facter'] fail.
* Add aggregate parameter validation
aggregate parameter validation will support checking each individual dict
to resolve conditions for aliases, no_log, mutually_exclusive,
required, type check, values, required_together, required_one_of
and required_if conditions in argspec. It will also set default values.
eg:
tasks:
- name: Configure interface attribute with aggregate
net_interface:
aggregate:
- {name: ge-0/0/1, description: test-interface-1, duplex: full, state: present}
- {name: ge-0/0/2, description: test-interface-2, active: False}
register: response
purge: Yes
Usage:
```
from ansible.module_utils.network_common import AggregateCollection
transform = AggregateCollection(module)
param = transform(module.params.get('aggregate'))
```
Aggregate allows supports for `purge` parameter, it will instruct the module
to remove resources from remote device that hasn’t been explicitly
defined in aggregate. This is not supported by with_* iterators
Also, it improves performace as compared to with_* iterator for network device
that has seperate candidate and running datastore.
For with_* iteration the sequence of operartion is
load-config-1 (candidate db) -> commit (running db) -> load_config-2
(candidate db) -> commit (running db) ...
With aggregate the sequence of operation is
load-config-1 (candidate db) -> load-config-2 (candidate db) -> commit
(running db)
As commit is executed only once per task for aggregate it has
huge perfomance benefit for large configurations.
* Fix CI issues
* Fix review comments
* Add support for options validation for aliases, no_log,
mutually_exclusive, required, type check, value check,
required_together, required_one_of and required_if
conditions in sub-argspec.
* Add unit test for options in argspec.
* Reverted aggregate implementaion.
* Minor change
* Add multi-level argspec support
* Multi-level argspec support with module's top most
conditionals options.
* Fix unit test failure
* Add parent context in errors for sub options
* Resolve merge conflict
* Fix CI issue
* Make camel_to_snake work on capitalized plurals
`TargetGroupARNs` should become `target_group_arns`, not
`target_group_ar_ns`
Promote `camel_to_snake` to top layer function but prefix
it with an underscore.
Add tests for improved `_camel_to_snake` function.
Reduce use of `re.compile` as it makes no sense when the
compilation result is not reused.
* Remove unused LooseVersion check
* Fix PLURALs case for camel_to_snake
Also renamed EXPECTED_CAMELIZATION to EXPECTED_SNAKIFICATION
* ACI module_utils library for ACI modules
This PR includes:
- the ACI argument_spec
- an aci_login function
- an experimental aci_request function
- an aci_response function
- included the ACI team
* New prototype using ACIModule
This PR includes:
- A new ACIModule object with various useful methods
Fixes#13243
** Add --vault-id to name/identify multiple vault passwords
Use --vault-id to indicate id and path/type
--vault-id=prompt # prompt for default vault id password
--vault-id=myorg@prompt # prompt for a vault_id named 'myorg'
--vault-id=a_password_file # load ./a_password_file for default id
--vault-id=myorg@a_password_file # load file for 'myorg' vault id
vault_id's are created implicitly for existing --vault-password-file
and --ask-vault-pass options.
Vault ids are just for UX purposes and bookkeeping. Only the vault
payload and the password bytestring is needed to decrypt a
vault blob.
Replace passing password around everywhere with
a VaultSecrets object.
If we specify a vault_id, mention that in password prompts
Specifying multiple -vault-password-files will
now try each until one works
** Rev vault format in a backwards compatible way
The 1.2 vault format adds the vault_id to the header line
of the vault text. This is backwards compatible with older
versions of ansible. Old versions will just ignore it and
treat it as the default (and only) vault id.
Note: only 2.4+ supports multiple vault passwords, so while
earlier ansible versions can read the vault-1.2 format, it
does not make them magically support multiple vault passwords.
use 1.1 format for 'default' vault_id
Vaulted items that need to include a vault_id will be
written in 1.2 format.
If we set a new DEFAULT_VAULT_IDENTITY, then the default will
use version 1.2
vault will only use a vault_id if one is specified. So if none
is specified and C.DEFAULT_VAULT_IDENTITY is 'default'
we use the old format.
** Changes/refactors needed to implement multiple vault passwords
raise exceptions on decrypt fail, check vault id early
split out parsing the vault plaintext envelope (with the
sha/original plaintext) to _split_plaintext_envelope()
some cli fixups for specifying multiple paths in
the unfrack_paths optparse callback
fix py3 dict.keys() 'dict_keys object is not indexable' error
pluralize cli.options.vault_password_file -> vault_password_files
pluralize cli.options.new_vault_password_file -> new_vault_password_files
pluralize cli.options.vault_id -> cli.options.vault_ids
** Add a config option (vault_id_match) to force vault id matching.
With 'vault_id_match=True' and an ansible
vault that provides a vault_id, then decryption will require
that a matching vault_id is required. (via
--vault-id=my_vault_id@password_file, for ex).
In other words, if the config option is true, then only
the vault secrets with matching vault ids are candidates for
decrypting a vault. If option is false (the default), then
all of the provided vault secrets will be selected.
If a user doesn't want all vault secrets to be tried to
decrypt any vault content, they can enable this option.
Note: The vault id used for the match is not encrypted or
cryptographically signed. It is just a label/id/nickname used
for referencing a specific vault secret.
* Module argument_spec now accepts a callable for the type argument, which is passed through and called with the value when appropriate. On validation/conversion failure, the name of the callable (or its type as a fallback) is used in the error message.
* adds basic smoke tests for custom callable validator functionality
* Enable the pylint no-name-in-module check. Checks that identifiers in
imports actually exist. When we do this, we also have to ignore
_MovedItems used in our bundled six. This means pylint won't check
for bad imports below ansible.module_utils.six.moves but that's
something that pylint punts on with a system copy of six so this is
still an improvement.
* Remove automatic use of system six. The exec in the six code which
tried to use a system library if available destroyed pylint's ability
to check for imports of identifiers which did not exist (the
no-name-in-module check). That test is important enough that we
should sacrifice the bundling detection in favour of the test.
Distributions that want to unbundle six can replace the bundled six in
ansible/module_utils/six/__init__.py to unbundle. however, be aware
that six is tricky to unbundle. They may want to base their efforts
off the code we were using:
2fff690caa/lib/ansible/module_utils/six/__init__.py
* Update tests for new location of bundled six Several code-smell tests
whitelist the bundled six library. Update the path to the library so
that they work.
* Also check for basestring in modules as the enabled pylint tests will
also point out basestring usage for us.
* Nuage module and unit tests with requested changes
* Cleanup of imports
* Adding check on python version
* Adding import try and catch wrappers
* Cleanup of requirements and adding integration tests
* Using pypi package for simulator
* Cleanup of requirements and adding integration tests
* Adding aliases for integration tests
* Adding module to import sanity test skip list
* Revert "Adding module to import sanity test skip list"
This reverts commit eab23af8c5ca7c503af63c05610b5db66d31fae4.
* Adding check for importlib and cleanup of requirements
* Move tests to their own file
* Port to a pytest parametrized test so it's easier to define new tests
* Now that _symbolic_mode_to_octal is a classmethod, we don't have to
instantiate an AnsibleModule to test it.
* Add tests for #14994, having more than one operator per role and umask
chmod
* add unit test: nested dynamic includes
* nested dynamic includes: avoid AnsibleFileNotFound error
Error was:
Unable to retrieve file contents
Could not find or access 'include2.yml'
Before 8f758204cf, at the end of
'path_dwim_relative' method, the 'search' variable contained amongst
others paths:
'/tmp/roles/testrole/tasks/tasks/included.yml' and
'/tmp/roles/testrole/tasks/included.yml'.
The commit mentioned before removed the last one despite the method
docstrings specify 'with or without explicitly named dirname subdirs'.
* add integration test: nested includes
* Adding ciscowlc_command module and unit tests.
* Adding __init__.py for unit test.
* Fixing PEP8 W503.
* Renaming module from ciscowlc_command to aire_command.
* Renaming aire_command to aireos_command.
* Adding aruba_command module along with unit tests.
* Fixing PEP8 E303 too many blank lines.
* Adding default for timeout.
* Removing unused arguments. Moving default for timeout argument. Fixing cliconf to find hostname.
* Fixing PEP8 E302.
* Add unit tests for parted module
Test the current expected behavior of the module:
- mock parted() and get_device_info()
- Use some of the examples and test the 'script' passed to parted.
- mock check_parted_label() to return false, as if parted version is > 3.1
- assert get_device_info output is correct
Current implementation of the module runs parted several time while going
through all parameters (flags, name, ...). Between calls it uses get_device_info
to update the dictionary. Use check_mode for some of the tests to force module
to go through all the parameters even is dictionary is not updated.
* test_parted.py: add "name" param into expected results
since 78fff751ab, parse_partition_info
fetch the partition name. This commit adds 'name' key and value into
the expected results.
Consolidate the module_utils, constants, and config functions that
convert values into booleans into a single function in module_utils.
Port code to use the module_utils.validate.convert_bool.boolean function
isntead of mk_boolean.
* aci_rest: New module to access Cisco ACI
This PR includes:
- Relicense as GPLv3+
- Check-mode support
- Cosmetic changes to documentation
- Examples in YAML format
- Removal of incorrect requirements (for this module)
- Do not log passwords
- Implement native fetch_url instead of requests
- Use standard hostname, username and password parameters
- Add alias src for parameter config_file
- Add mutual exclusive content option for inline data (and show some inline examples)
- Add timeout parameter
- Add validate_certs parameter
- Handling ACI result output (identical for JSON as XML input)
- Parse/expose ACI error output to user
* Lower case method, add use_ssl, Use python dicts
This commit includes:
- Use lowercase method names
- Add `use_ssl` parameter (not the `protocol` parameter)
- Use a python dict for the request data (not a JSON string)
- Documentation improvements
* Ensure one of 'content' or 'src' is provided
* Fix issue with totalCount being a string in JSON
This fixes the problem with JSON output where totalCount is a string and
not an integer.
This fixesjedelman8/aci-ansible#7
* Improve code documentation
* Improve error handling and module response
* Small typo
* Improve documentation and examples
* Keep protocol parameter, but deprecate it
* Extrude aci functions from module_utils
* aci_rest: Add unit tests
* add first, last and next usable
* add usable ip filters
* add size usable, range usable and wildcard
* add ip prefix and netmask filter
* add network formatting and check if ip in subnet
* clean up order, add comments
* fix pep8
* update format by index
* clean up and updates from jmcgill298
* adds more intelligent save logic and diff to network config modules
* adds sha1 property to NetworkConfig
* adds new argument save_when to argument_spec
* adds new argument diff_against to argument_spec
* adds new argument intended_config to argument_spec
* renames config argument to running_config with alias to config
* deprecates the use of the save argument
* before and after now work with src argument
* misc module clean
Modules updated
* nxos_config
* ios_config
* eos_config
Most notably this makes the save mechanism more intelligent for config
modules for devices that need to copy the ephemeral config to
non-volatile storage.
The diff_against argument allows the playbook task to control what the
device's running-config is diff'ed against. By default it will return
the diff of the startup-config.
* removes ios_config from pep8/legacy_files.txt
* extends the ignore lines argument to the module
* clean up CI errors
* add missing list brackets
* fixes typo
* fixes unit test cases
* remove last line break when returning config contents
* encode config string to bytes before hashing
* fix typo
* addresses feedback in PR
* update unit test cases
It was in lib/ansible/modules/system/setup.py since it
was the only thing using it, but move it back to module_utils
and add a ansible_collector.get_ansible_collector() to build
a facts collector just like the one used by setup.py
mv test_setup.py -> test_ansible_collector.py
All the code it was testing is now in ansible_collector
rm code to create 'ansible_facts' subkey from namespace
Just leave it up to the caller to do, and just return a
flat dictionary from AnsibleFactCollector.collect()
* junos_linkagg implementation and junos modules refactor
* junos_linkagg implementation
* junos_linkagg integration test
* net_linkagg integration test for junos
* decouple `load_config` and `commit` operations,
to allow single commit (in case on confirm commit) and
to perform batch commit (multiple `load_config` followed by single
`commit`)
* Other related refactor
* Fix CI issues
* Fix unit test failure
The iApp service module worked fine previously, but this patch
adds enhancements to it to include more fields that can be
specified when creating iapp services.
restored 'rc' inspection but only when failed is not specified
removed redundant changed from basic.py as task_executor already adds
removed redundant filters, they are tests
added aliases to tests removed from filters
fixed test to new rc handling
* Fix ansible-doc traceback when a plugin doesn't parse correctly
* Change extract_metadata ivocation to take either an ast or source
code. When given source code, it can find file offsets for the start
and end of dict. When given the ast, it is quicker as it doesn't have
to reparse the source. Requires changing the call to the function to
use a keyword arg.
* Fix reading of metadata to find the last occurrence of
ANSIBLE_METADATA instead of the first.
* Add some more unittests to get closer to complete coverage
* Unittests for extracting metadata from plugins
* Port plugin_docs to use the generic extract_metadata function
* Make the helper functions seek_end_of{string,dict} private
So I thought I fixed it before, but there's still one location where
the `rc` value is influential to decide whether a task failed or not.
We already established in #24867 that it is up to the module to decide
what the return code actually means, not the task executor. We modified
the existing modules to move that logic into the module (eg. for
command, shell, etc.)
This relates to the integration tests of win_robocopy, where different
return codes have different meanings:
- 0 -- No files copied.
- 1 -- Files copied successfully! (changed)
- 2 -- Some Extra files or directories were detected. No files were copied. (warning)
- 3 -- (2+1) Some files were copied. Additional files were present. (changed)
- 4 -- Some mismatched files or directories were detected. Housekeeping might be required! (changed + warning)
- 5 -- (4+1) Some files were copied. Some files were mismatched. (changed + warning)
- 6 -- (4+2) Additional files and mismatched files exist. No files were copied. (warning)
- 7 -- (4+1+2) Files were copied, a file mismatch was present, and additional files were present. (changed + warning)
- 8 -- Some files or directories could not be copied! (changed + failed)
- 9 - 15 -- Fatal error. Check log message! (failed)
- 16 -- Serious Error! No files were copied! Do you have permissions to access $src and $dest? (failed)
This also fixes#24652
* Extend tests to have multiple device representations
* Move filepath munging to nxos_module
* Device needs to be kwarg so we can leave it off
* Update other nxos tests
* Update tests that fell through
Make pyca/cryptography the preferred backend for cryptographic needs (mainly vault) falling back to pycrypto
pyca/cryptography is already implicitly a dependency in many cases
through paramiko (2.0+) as well as the new openssl_publickey module,
which requires pyOpenSSL 16.0+. Additionally, pyca/cryptography is
an optional dep for better performance with vault already.
This commit leverages cryptography's padding, constant time comparisons,
and CBC/CTR modes to reduce the amount of code ansible needs to
maintain.
* Handle wrong password given for VaultAES format
* Do not display deprecation warning for cryptography on python-2.6
* Namespace all of the pycrypto imports and always import them
Makes unittests better and the code less likely to get stupid mistakes
(like using HMAC from cryptogrpahy when the one from pycrypto is needed)
* Add back in atfork since we need pycrypto to reinitialize its RNG just in case we're being used with old paramiko
* contrib/inventory/gce: Remove spurious require on pycrypto
(cherry picked from commit 9e16b9db275263b3ea8d1b124966fdebfc9ab271)
* Add cryptography to ec2_win_password module requirements
* Fix python3 bug which would pass text strings to a function which
requires byte strings.
* Attempt to add pycrypto version to setup deps
* Change hacking README for dual pycrypto/cryptography
* update dependencies for various CI scripts
* additional CI dockerfile/script updates
* add paramiko to the windows and sanity requirement set
This is needed because ansible lists it as a requirement. Previously
the missing dep wasn't enforced, but cryptography imports pkg_resources
so you can't ignore a requirement any more
* Add integration test cases for old vault and for wrong passwords
* helper script for manual testing of pycrypto/cryptography
* Skip the pycrypto tests so that users without it installed can still run the unittests
* Run unittests for vault with both cryptography and pycrypto backend
* adds new common functions for declarative intent modules
* adds Entity and EntityCollection
* adds dict_diff and dict_combine
* update for CI PEP8 compliance
* more CI PEP8 fixes
* more PEP8 CI clean up
* refactors the lambda assignments into top level classes
this is to be in compliant the PEP8 CI sanity checks
* one last pep8 ci fix
* Add nxos_nxapi tests
* Simple changes to nxos_nxapi
* Move validation to check_args
* Don't mark protocol change unless change is requested
* Add different regex to handle HTTP{,S} ports on a different version of nxos
finished normalizing of path handling
removed overloaded '-p' from init_paths option, it is for role_paths
removed expand_tilde and get_opt methods as both were redundant, adjusted rest of code
updated tests to match
* Add vyos_user implementation module
Signed-off-by: Trishna Guha <trishnaguha17@gmail.com>
* Integration test for vyos_user
Signed-off-by: Trishna Guha <trishnaguha17@gmail.com>
* Make state absent work
Signed-off-by: Trishna Guha <trishnaguha17@gmail.com>
* Unit test for vyos_user
Signed-off-by: Trishna Guha <trishnaguha17@gmail.com>
* Standardize user names
Signed-off-by: Trishna Guha <trishnaguha17@gmail.com>
* Modify integration test with idempotent case
Signed-off-by: Trishna Guha <trishnaguha17@gmail.com>
* Add role as alias to level
Signed-off-by: Trishna Guha <trishnaguha17@gmail.com>
* Start of ansible config project
moved configuration definitions to external yaml file vs hardcoded
* updated constants to be a data strcutures that are looped over and also return origin of setting
changed to manager/data scheme for base classes
new cli ansible-config to view/manage ansible configuration settings
* prints green for default/unchanged and yellow for those that have been overriden
* added list action to show all configurable settings and their associated ini and env var names
* allows specifying config file to see what result would look like
* TBD update, edit and view options
removed test for functions that have been removed
env_Vars are now list of dicts
allows for version_added and deprecation in future
added a couple of descriptions for future doc autogeneration
ensure test does not fail if delete_me exists
normalized 'path expansion'
added yaml config to setup packaging
removed unused imports
better encoding handling
updated as per feedback
* pep8
* Add more mount point statvfs info including sizes
Based on https://github.com/ansible/ansible/pull/12073
facts.utils.get_mount_size() now returns a dict of most
of the posix statvfs data, including block_size and inode
counts.
Update the facts.hardware classes that use get_mount_size() to
use the new info by mount_info.update(mount_statvfs_inof) to merge.
* add back unit tests for LinuxHardware mount/fs facts
* add test cases for facts.utils.get_mount_size
* Support NetBSD 7.1+ style ifconfig -a output
network facts on NetBSD after 7.1 cvs would fail
because of format changes in 'ifconfig -a' output.
update code to support new and old format.
add unit tests for both based on
examples from Bruce V Chiarelli.
* wrap use of interfaces.keys() in list() for py3 compat
* sort interface ids for stability
* Fix ansible_cmdline initrd fact for UEFI
UEFI cmdline paths use \ path sep which would
get munged by cmdline fact collection.
* Make CmdLineFactCollector easier to test
extract the parsing of the /proc/cmdline content to
_parse_proc_cmdline()
add a wrapper method for get_file_content _get_proc_cmdline()
Add unit tests of _parse_proc_cmdline based on examples
from issue #23647Fixes#23647
* netscaler_service initial implementation
* Changes as requested by reviewers
* Skip some tests if under python2.6 and importing requests library
* Change option "operation" to "state"
* Remove print statements from netscaler module utils
* Catch all exceptions during login
* Fix fail message
* Add common option save_config
When operating on arbitrary return data from modules, it is possible to
hit the recursion limit when cleaning out no_log values from the data.
To fix this, we have to switch from recursion to iteration.
Unittest for remove_values recursion limit
Fixes#24560
* Ultra basic api-gateway module based of lambda.py
* Ultra basic deployment added to api-gateway module
* ApiGateway module Allow creation of APIs, more documentation and better return value
* ApiGateway module incorporate review feedback
* ApiGateway module flake8 cleanup
* APIGateway module - more review fixes.
* slightly better messages in api_gateway module
* AWS api_gateway module - try to improve messages in case of exceptions
* rename api_gateway module to aws_api_gateway as discussed in PR 20230
* aws_api_gateway - Allow delivery of swagger either as text or dictionary.
* aws_api_gateway module - introduce 'unit' tests, improve imports using them and small fixes
* aws_api_gateway module - move path expand_user to avoid early typecheck
* aws_api_gateway - version means version of metadata not module - fix to 1.0
* aws_api_gateway module - Rely on module_utils.ec2 for imports & path type for expanduser / cleanups
* aws_api_gateway module - heavy cleanup and refactor of code + cloud retry functionality.
* api_gateway_module - failing test case for handling more than one deployment in succession and API deletion
* add TooManyRequestsException to AWSRetry exception list - makes API deployment work.
* api_gateway_module - Fixes for various review comments + errors from various linters
* api_gateway_module - Fixes for more review comments + linter error
* api_gateway_module - Major refactor into sensible functions - create_response becomes configure_response
* api_gateway_module - should be working under python3; remove test exclusion
* api_gateway_module - finish off remaining review fixes - use ansible defaults and fix mutually exclusive
* api_gateway_module - attempt to improve handling of botocore errors in python3
* api_gateway_module - implement state=absent / API deletion
Facts Refresh (2.4 roadmap)
This commit implements most of the 2.4 roadmap 'Facts Refresh'
- move facts.py to facts/__init__.py
- move facts Distribution() to its own class
- add a facts/utils.py
- move get_file_content and get_uname_version to facts/utils.py
- move Facts() class from facts/__init__ to facts/facts.py
- mv get_file_lines to facts/utils.py
- mv Ohai()/Facter() class to facts/ohai.py and facter.py
- Start moving fact Hardware() classes to facts/hardware/*.py
- mv HPUX() hardware class to facts/hardware/hpux.py
- move SunOSHardware() fact class to facts/hardware/sunos.py
- move OpenBSDHardware() class to facts/hardware/openbsd.py
- mv FreeBsdHardware() and DragonFlyHardware() to facts/hardware/
- mv NetBSDHardware() to facts/hardware/netbsd.py
- mv Darwin() hardware class to facts/hardware/darwin.py
- pep8/etc cleanups on facts/hardware/*.py
- Mv network facts classes to facts/network/*.py
- mv Virtual fact classes to facts/virtual
- mv Hardware.get_sysctl to facts/sysctl.py:get_sysctl
- Also mv get_uname_version from facts/utils.py -> distribution.py
since distribution.py is the only thing using it.
- add collector.py with new BaseFactCollector
- add a subclass for AnsibleFactCollector
- hook up dict key munging FactNamespaces
- add some test cases for testing the names of facts
- mv timeout stuff to facts.timeout
- rm ansible_facts()/get_all_facts() etc
- Instead of calling facts.ansible_facts(), fact collection
api used by setup.py is now to create an AnsibleFactCollector()
and call it's collect method.
- replace Facts.get_user_facts with UserFactCollector
- add a 'systems' facts package, mv UserFactCollector there
- mv get_dns_facts to DnsFactCollector
- mv get_env_facts to EnvFactCollector
- include the timeout length in exception message
- modules and module_utils that use AnsibleFactCollector
can now theoretically set the 'valid_subsets'
May be useful for network facts module that currently have
to reimplement a good chunk of facts.py to get gather_subsets
to work.
- get_local_facts -> system/LocalFactCollector
- get_date_time -> system/date_time.py
- get_fips_facts -> system/fips.py
- get_caps_facts() -> system/caps.py
- get_apparmor_facts -> system/apparmor.py
- get_selinux_facts -> system/selinux.py
- get_lsb_facts -> system/lsb.py
- get_service_mgr_facts -> system/service_mgr.py
- Facts.is_systemd_managed -> system/service_mgr.py
- get_pkg_mgr_facts -> system/pkg_mgr.py
- Facts()._get_mount_size_facts() -> facts.utils.get_mount_size()
- add unit test for EnvFactCollector
- add a test case for minimal gather_subsets
- add test case for collect_ids
- Make gather_subset match existing behavior or '!all'
If 'gather_subset' is provided as '!all', the existing behavior
(in 2.2/2.3) is that means 'dont collect any facts except those
from the Facts() class'. So 'skip everything except
'apparmor', 'caps', 'date_time', 'env', 'fips', 'local', 'lsb',
'pkg_mgr', 'python', 'selinux', 'service_mgr', 'user', 'platform', etc.
The new facts setup was making '!all' mean no facts at all, since
it can add/exclude at a finer granularity. Since that makes more
sense for the ansible collector, and the set of minimal facts to
collect is really more up to setup.py to decide we do just that.
So if setup.py needs to always collect some gather_subset, even
on !all, setup.py needs to have the that subset added to the
list it passes as minimal_gather_subset.
This should fix some intg tests that assume '!all' means that
some facts are still collected (user info and env for example).
If we want to make setup.py collect a more minimal set, we can do that.
- force facts_dicts.keys() to a list so py3 works
- split fact collector tests to test_collectors.py
- convert Facter(Facts) -> other/facter.py:FacterFactCollector
- add FactCollector.collect_with_namespace()
regular .collect() will return a dict with the key names
using the base names ('ip_address', 'service_mgr' etc)
.collect_with_namespace() will return a dict where the key names
have been transformed with the collectors namespace, if there is
one. For most, this means a namespace that adds 'ansible_' to the
start of the key name.
For 'FacterFactCollector', the namespace transforms the key to
'facter_*'.
- add test cases for collect_with_namespace
- move all the concrete 'which facts does setup.py' stuff to setup.py
The caller of AnsibleFactCollector.from_gather_subset() needs to
pass in the list of collector classes now.
- update system/setup.py to import all of the fact classes and pass
in that list.
- split the Distribution fact class up a bit
extracted the 'distro release' file handling (ie, linux
boxes with /etc/release, /etc/os-release etc) into its
own class.
- extract get_cmdline_facts -> cmdline.py
- extract get_public_ssh_host_keys -> system/ssh_pub_keys.py
- extract get_platform_facts -> system/platform.py
platform.py may be a good candidate for further splitting.
- rm test for plain Facts() base class
- let the base class for Collector unit tests provide collected_facts
some Collectors and/or their migrated Facts() subsclasses need
to look at facts collected by other modules ('ansible_architecture'
the main one...).
Collector.collect() has the collected_facts arg for this, so add
a class variable to BaseFactsTest so we can specify it.
- mv Ohai to other/ohai.py and convert to Collector
- update hardware/*.py to return facts (no side effects)
- mv AnsibleFactCollector to setup.py
- extra collector class gathering to module method in
facts/__init__.py (collector_classes_from_gather_subset)
- add a CollectorMetaDataCollector collector used to provide
the 'gather_setup' fact
- add unit test module for 'setup' module
(test/units/modules/system/setup.py)
- Collector init now doesnt need a module, but collect does
An instance of a FactCollector() isnt tied to a AnsibleModule
instance, but the collect() method can be, so optionally pass
in module to FactCollector.collect() (everywhere)
- add a default_collectors for list of default collectors
import and use it from setup.py module
eventually, would like to replace this with a plugin loader
style class finder/loader
- unit tests for module_utils/facts/__init__.py
- add unit tests for ohai facts collector
- remove self.facts side effect on populate() in hardware/sunos.py
- convert OpenBSDHardware() to rm side effects on self.facts
- try to rm some self.facts side effects in Network()
plumb in collected_facts from populate() where it is needed.
stop passing collected_facts into Network() [via cached_facts=,
where it eventually becomes self.facts]
- nothing provides Fact() cached_facts arg now, rm it
Facts() should be internal only implementation so nothing
should be using it.
Of course, now someone will.
- add a Collector.name attr to build a map of name->_fact_ids
To properly exclude a gather_subset spec like '!hardware', we
need to know that 'hardware' also means 'devices', 'dmi', etc.
Before, '!hardware' would remove the 'hardware' collector name
but not 'devices'. Since both would end up in id_collector_map,
we would still end up with the HardwareCollector in the collector
list. End result being that '!hardware' wouldn't stop hardware
from being collected.
So we need to be able to build that map, so add the Collector.name
attribute that is the primary name (like 'hardware') and let
Collector._fact_ids be the other fact ids that a collector is
responsible for.
Construct the aliases_map of Collector.name -> set of _fact_ids
in fact/__init__.py get_collector_names, and use it when we are
populating the exclude set.
- refactor of distribution.py
make the big OS_FAMILY literal a little easier to read
Also keys can now be any string instead of python literals
99% sure the test for 'KDE Neon' was wrong
I don't see how/where it should or could get 'Neon' instead
of 'KDE Neon' as provided in os-release NAME=
Use 'distribution' string for key to OS_MAP
ie, we dont need to make it a valid python label anymore so dont.
move _has_dist_file to module as _file_exists
easier to mock without mucking with os.path
mv platform.system() calls to within get_distribution_facts() instead
of Distribution() init.
- remove _json compat module
The code in here was to support:
-a 'json' python module that was not the standard one included
with python since 2.6.
- potentially fallback to simplejson if 'json' was not available.
'json' is available for all supported python versions now so
no longer needed.
- mv get_collector_names -> facts.collector
- mv collector_classes_from_gather_subset -> facts.collector
- mv collector tests from test_facts -> test_collector
- Use six's reduce() in sunos/netbsd hardware facts
- rm extraneous get_uname_version in utils
only system/distribution.py uses it
- Remove Facts() subclass metaclass usage
- using fact_id and a platform id for matching collectors
gut most of Facts() subclasses
rm Facts() subclasses with weird metaclass
only add collectors that match the fact_ids and the platform_info
to the list of collectors used.
atm, a collectors platform_id will default to 'Generic', and
any platform matches 'Generic'
goal is to select collector classes including matching the
systems platform in collector.py, instead of relying on
metaclasses in hardware/*. To finish this, the various
Facts() subclasses will need to be replaced entirely with
Collector() subclasses.
use collector classmethod platform_match() to match the platform
This lets the particular class decide if it is compatible with
a given platform_info. platform_info is a dict like obj, so it could be
expanded in the future.
Add a default platform_match to BaseFactCollector that matches
platform_info['system'] == cls._platform
They were needed previously to trigger a module
load on all the collector classes when we import
facts/hardare so that the Hardware() and related
classes that used __new__ and find_all_subclasses()
would work.
Now that is done in collectors based on platform matching
at runtime we dont need to do it py module import/parse
time. So the non empty __init__.pys are no longer needed
and their is a more flexible mechanism for selection
platform specific stuff.
facts/facts.py is no longer used, rm'ed
- if we dont find an implement class for gather spec.. just ignore it.
Would be useful to add a warn to warn about this case.
- Fix SD-UX typo (should be HP-UX)
- Port fix for #21893 (0 sockets) to this branch
This readds the change from 8ad182059d
that got lost in merge/rebase
Fixes#21893
- port sunos fact locale fix for #24542 to this branch
based on e558ec19cdFixes#24542
Solaris fact fix (#24793)
ensure locale for solaris fact gathering
fixes issue with locale interfering with proper reading of decimals
- raise exceptions in the air like we just dont care.
Pretty much ignore any not exit exception in facts
collection. And add some test cases.
- added new selinux fact to clarify python lib
the selinux fact is boolean false when the library is not installed,
a dictionary/hash otherwise, but this is ambigous
added new fact so we can eventually remove the type dichtomy and normalize it as a dict
Re-add of devel commit 85c7a7b844 to
the new code layout, since it got removed in merge/rebase
This is required for modules that may return a non-zero `rc` value for a
successful run, similar to #24865 for Windows fixing **win_chocolatey**.
We also disable the dependency on `rc` value only, even if `failed` was
set.
Adapted unit and integration tests to the new scheme.
Updated raw, shell, script, expect to take `rc` into account.
* Simplify apply_key_map
* Fix nxapi
* Clean up get_value
* Fix missing non-values
* Add test for existing bgp_af case
* Fix small issues with bgp_neighbor_af
* test/: PEP8 compliancy
- Make PEP8 compliant
* Python3 chokes on casting int to bytes (#24952)
But if we tell the formatter that the var is a number, it works
* eos python3 changes
* changes to convert response from byte to text
* Add dellos6 python3 changes
Make `execute_command` arguments and its
return value complaint to PY3 changes
made in PR #24431
* Fix py3 prompt issue for invalid show command
* Fix review comments
* Add generic fix for error prompt in py3
* Fix CI issue
* Fix network_cli unit test failure
* draft new inventory plugin arch, yaml sample
- split classes, moved out of init
- extra debug statements
- allow mulitple invenotry files
- dont add hosts more than once
- simplified host vars
- since now we can have multiple, inventory_dir/file needs to be per host
- ported yaml/script/ini/virtualbox plugins, dir is 'built in manager'
- centralized localhost handling
- added plugin docs
- leaner meaner inventory (split to data + manager)
- moved noop vars plugin
- added 'postprocessing' inventory plugins
- fixed ini plugin, better info on plugin run group declarations can appear in any position relative to children entry that contains them
- grouphost_vars loading as inventory plugin (postprocessing)
- playbook_dir allways full path
- use bytes for file operations
- better handling of empty/null sources
- added test target that skips networking modules
- now var manager loads play group/host_vars independant from inventory
- centralized play setup repeat code
- updated changelog with inv features
- asperioribus verbis spatium album
- fixed dataloader to new sig
- made yaml plugin more resistant to bad data
- nicer error msgs
- fixed undeclared group detection
- fixed 'ungrouping'
- docs updated s/INI/file/ as its not only format
- made behaviour of var merge a toggle
- made 'source over group' path follow existing rule for var precedence
- updated add_host/group from strategy
- made host_list a plugin and added it to defaults
- added advanced_host_list as example variation
- refactored 'display' to be availbe by default in class inheritance
- optimized implicit handling as per @pilou's feedback
- removed unused code and tests
- added inventory cache and vbox plugin now uses it
- added _compose method for variable expressions in plugins
- vbox plugin now uses 'compose'
- require yaml extension for yaml
- fix for plugin loader to always add original_path, even when not using all()
- fix py3 issues
- added --inventory as clearer option
- return name when stringifying host objects
- ajdust checks to code moving
* reworked vars and vars precedence
- vars plugins now load group/host_vars dirs
- precedence for host vars is now configurable
- vars_plugins been reworked
- removed unused vars cache
- removed _gathered_facts as we are not keeping info in host anymore
- cleaned up tests
- fixed ansible-pull to work with new inventory
- removed version added notation to please rst check
- inventory in config relative to config
- ensures full paths on passed inventories
* implicit localhost connection local
