Skip to content

Conversation

@OddBloke
Copy link
Collaborator

No description provided.

Joshua Powers and others added 30 commits June 2, 2020 12:06
This fixes issues with closing brackets not matching the opening
bracket's line and continuation line under-idented for hanging indent.
Improving the debugability of this code path by logging the thrown exception details for the non 404 exceptions.

Retry IMDS on HTTP Error 404 and 410, re-run DHCP on other exceptions.
…onical#367)

Build time feature flags are now defined in cloudinit/features.py.
Feature flags can be added to toggle configuration options or
deprecated features. Feature flag overrides can be placed in
cloudinit/feature_overrides.py. Further documentation can be found in
HACKING.rst.

Additionally, updated default behavior to exit with an exception
if #include can't retrieve resources as expected. This behavior
can be toggled with a feature flag.

LP: #1734939
This was painful, but it finishes a TODO from cloudinit/subp.py.

It moves the following from util to subp:
  ProcessExecutionError
  subp
  which
  target_path

I moved subp_blob_in_tempfile into cc_chef, which is its only caller.
That saved us from having to deal with it using write_file
and temp_utils from subp (which does not import any cloudinit things now).

It is arguable that 'target_path' could be moved to a 'path_utils' or
something, but in order to use it from subp and also from utils,
we had to get it out of utils.
runparts (run a directory of scripts) seems to fit well in subp
module.  The request to move it there was raised in canonical#416.

Replace use of logexc with LOG.debug as logexc comes from util.
This was brought up in review of canonical#416.
Makes sense to remove the local copy of "is this executable file".
)

* test_opennebula: convert TestParseShellConfig to a pytest test

And allow it to run bash.

(We aren't aiming to convert TestCase tests to pytest tests as a rule.
In this case, I needed to change its implementation to limit subp usage,
and I chose pytest over CiTestCase.)

* test: move conftest.py to top-level, to cover tests/ also

This gives us a single conftest.py which is shared by all tests in the
project.
* Document CloudStack data-server well-known hostname
* Document fallback to default gateway
* Add onitake to CLA signers list
…g get deployPkg enable-custom-scripts", the return code will be EX_UNAVAILABLE(69), on this condition, it should not take it as error. (canonical#413)
Reason: commit ded1ec8 introduced a regression whereby a bridge with no "parameters:" setting caused a KeyError exception.

LP: #1879673
Prior to this change, we would debootstrap for every single integration
test run in Travis.  This changes that, so we will store the chroot
created by debootstrap in Travis' cache, and use it if available.  This
saves 2-3 minutes, or ~1/3rd of the integration test run time (which is
our longest run).
On a system with a non-utf8 default locale, the logger will silently
not log anything if the message contains an unsupported character.
This makes us more robust, as our caching won't start breaking if other
parts of the Travis build change directory for us.
…ical#439)

When updating the docstring to include it, I realised that the current
name is somewhat misleading; this makes it a little easier to
understand, I think.
…cal#437)

When using an http proxy make sure to disable all the dynamic mirror
URLs. In particular:

 - Uncomment the baseurl in /etc/yum.repos.d/*.repo.
 - Comment out the mirrorlist and metalink URLs in *.repo.
 - Replace the dynamic download.fedoraproject.org host with
   dl.fedoraproject.org, which is static.
 - Run the above as part of the MAYBE_RELIABLE_YUM_INSTALL command,
   as installing packages may add new repos (e.g. EPEL).
 - Stop disabling fastestmirror, not needed when doing the above.
…anonical#369)

This introduces a way to log the dhclient error stream, and uses it for the Azure datasource (where we have a specific requirement for this data to be logged).
This allows us to disable the `ensure_dir` call when it isn't
appropriate.
…al#440)

Prior to this change, the process of tarring up would mean that Travis
would always detect that the cache had changed, and we would incur ~30s
of packing/transferring at the end of every build.

Instead, we now check if there was any change to the installed contents
of the schroot, and only generate a new tarball if there were changes.
…canonical#445)

If the instance symlink doesn't exist, then we shouldn't create a
directory in its place, because that breaks future boots.

LP: #1883903
…#446)

Co-authored-by: Rick Harding <rharding@mitechie.com>
…#448)

Hetzner cloud only supports user-data as a string (presumably utf-8).

In order to allow users on Hetzner to provide binary data to cloud-init,
we will attempt to base64decode the userdata.

The change here adds a 'maybe_b64decode' function that will decode data
if and only if is base64 encoded.

The reason for not using util.b64d is that we do not want the return value
decoded to a string, and util.b64d will do that if it can.  Additionally
we call decode with validate=True which oddly is not the default.

LP: #1884071
The name `redhat' is not used but rather `rhel' to identify the distro.

Signed-off-by: Eduardo Otubo <otubo@redhat.com>
This commit introduces the initial structure for the "cloudinit.net -> cloudinit.distros.networking Hierarchy" refactor, as detailed in [0]. It also updates that section with some changes driven by this initial implementation, as well as adding a lot more specifics to it.

[0] https://cloudinit.readthedocs.io/en/latest/topics/hacking.html#cloudinit-net-cloudinit-distros-networking-hierarchy
Namely, is_connected, is_wireless and is_present.  None of these are
used in the cloud-init codebase, so remove the dead code (instead of
refactoring it).
OddBloke and others added 22 commits June 26, 2020 09:42
…cal#449)

We build in a schroot using sbuild, so we don't need Build-Depends
installed in the host; save some time (and manual maintenance when
Build-Depends change) by dropping their installation and skipping the
check for them when calling `bddeb`.  We _do_ require debhelper to be
installed to build the source package, so ensure it is installed.

Additionally, while I was here, replace ubuntu-dev-tools with
devscripts; we don't need everything that ubuntu-dev-tools pulls in.
My previous testing was insufficient: there is a branch of testing that
requires mk-sbuild, which is shipped in ubuntu-dev-tools itself.
Create a schema object for the chef module and validate this schema in the handle function of the module.

Some of the config keys description, so I tried looking at the code and chef documentation to provide an information to the user. However, I don't know if I have the best description for all fields. For example, for the key show_time I could not find an accurate description of what it did, so I used what was in our code base to infer what it should do.

LP: #1858888
Changes are made that simplify code and aim to properly support FreeBSD:

- use `util.find_devs_with` instead call directly `blkid`, because on FreeBSD is not supported well and `util.find_devs_with` have solution for FreeBSD for that
- introduction of an additional name on FAT file system, which is used in FreeBSD
- drop shell to use default value, because FreeBSD – by default – does not have `/bin/bash`
…message test using it (canonical#461)

caplog is only available in pytest itself from 3.0 onwards. In xenial, we only have pytest 2.8.7. However, in xenial we do have pytest-catchlog available (as python3-pytest-catchlog), so we use that where appropriate.
As the first refactor PR, this also includes the initial structure for tests.

LP: #1884619
For versions before 20.2, we allowed the use of ec2 mirrors if the datasource availability_zone matches one of the ec2 regions. We are now updating that behavior to allow allow the use of ec2 mirrors on ec2 instances or if the user directly passes an an ec2 mirror url through #cloud-config apt directives.

LP: #1456277
This is an improvement over indirect parameterisation for a few reasons:

* The test code is much easier to read, the mark names are much more
  intuitive than the indirect parameterisation invocation, and there's
  less boilerplate to boot
* The fixture no longer has to overload the single parameter that
  fixtures can take with multiple meanings
Added cloud_final_modules in place of cloud_init_modules under the heading 'modules'. cloud_init_modules was wrongly appearing twice.
* Added myself as contributor to the project
* Sorted the file alphabetically
…nonical#478)

Specifically:

* disable E1102 in cloudinit/sources/helpers/openstack.py for reasons
  described in a comment, and
* refactor `abs_join` to require at least one positional argument; this
  matches os.path.join's signature, and that mismatch is what was
  causing pylint to emit a warning
* bump to pylint 2.4.2
* Add update_etc_hosts as default module on *BSD
* Set preference of IPv6 over IPv4 in FreeBSD /etc/hosts
Update lxd_export method to detect and handle different image formats
cloud_tests will only support the "split" type images which exports a
compressed (xz) tarball of metadata and a rootfs (of different formats).
For non-split image formats (single tarball with metadata + rootfs) we now
raise an exception indicating that the requested image is not supported
at this time.
Do not fail if /etc/fstab is not present.  Some images, like container
rootfs may not include this file by default.

LP: #1886531
@raharper
Copy link
Collaborator

LGTM!

% git checkout ubuntu/devel
Switched to branch 'ubuntu/devel'
Your branch and 'upstream/ubuntu/devel' have diverged,
and have 22 and 174 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)
% git reset --hard upstream/ubuntu/devel
HEAD is now at b4789c8ae changelog: drop six, unittest2, nose, pep8 and pyflakes from build-deps
% ~/work/git/uss-tableflip/scripts/new-upstream-snapshot
updating remote upstream for default commitish upstream/master.
X11 forwarding request failed on channel 0
diff --git a/debian/changelog b/debian/changelog
index 3054f97d1..c13471708 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,75 @@
+cloud-init (20.2-94-g3d06abc2-0ubuntu1) UNRELEASED; urgency=medium
+
+  * New upstream snapshot.
+    - cc_mounts: handle missing fstab (#484) (LP: #1886531)
+    - LXD cloud_tests: support more lxd image formats (#482) [Paride Legovini]
+    - Add update_etc_hosts as default module on *BSD (#479) [Adam Dobrawy]
+    - cloudinit: fix tip-pylint failures and bump pinned pylint version (#478)
+    - Added BirknerAlex as contributor and sorted the file (#477)
+      [Alexander Birkner]
+    - Update list of types of modules in cli.rst [saurabhvartak1982]
+    - tests: use markers to configure disable_subp_usage (#473)
+    - Add mention of vendor-data to no-cloud format documentation (#470)
+      [Landon Kirk]
+    - Fix broken link to OpenStack metadata service docs (#467)
+      [Matt Riedemann]
+    - Disable ec2 mirror for non aws instances (#390)
+      [lucasmoura] (LP: #1456277)
+    - cloud_tests: don't pass --python-version to read-dependencies (#465)
+    - networking: refactor is_physical from cloudinit.net (#457) (LP: #1884619)
+    - Enable use of the caplog fixture in pytest tests, and add a
+      cc_final_message test using it (#461)
+    - RbxCloud: Add support for FreeBSD (#464) [Adam Dobrawy]
+    - Add schema for cc_chef module (#375) [lucasmoura] (LP: #1858888)
+    - test_util: add (partial) testing for util.mount_cb (#463)
+    - .travis.yml: revert to installing ubuntu-dev-tools (#460)
+    - HACKING.rst: add details of net refactor tracking (#456)
+    - .travis.yml: rationalise installation of dependencies in host (#449)
+    - Add dermotbradley as contributor. (#458) [dermotbradley]
+    - net/networking: remove unused functions/methods (#453)
+    - distros.networking: initial implementation of layout (#391)
+    - cloud-init.service.tmpl: use "rhel" instead of "redhat" (#452)
+    - Change from redhat to rhel in systemd generator tmpl (#450)
+      [Eduardo Otubo]
+    - Hetzner: support reading user-data that is base64 encoded. (#448)
+      [Scott Moser] (LP: #1884071)
+    - HACKING.rst: add strpath gotcha to testing gotchas section (#446)
+    - cc_final_message: don't create directories when writing boot-finished
+      (#445) (LP: #1883903)
+    - .travis.yml: only store new schroot if something has changed (#440)
+    - util: add ensure_dir_exists parameter to write_file (#443)
+    - printing the error stream of the dhclient process before killing it
+      (#369) [Moustafa Moustafa]
+    - Fix link to the MAAS documentation (#442)
+      [Paride Legovini] (LP: #1883666)
+    - RPM build: disable the dynamic mirror URLs when using a proxy (#437)
+      [Paride Legovini]
+    - util: rename write_file's copy_mode parameter to preserve_mode (#439)
+    - .travis.yml: use $TRAVIS_BUILD_DIR for lxd_image caching (#438)
+    - cli.rst: alphabetise devel subcommands and add net-convert to list (#430)
+    - Default to UTF-8 in /var/log/cloud-init.log (#427) [James Falcon]
+    - travis: cache the chroot we use for package builds (#429)
+    - test: fix all flake8 E126 errors (#425) [Joshua Powers]
+    - Fixes KeyError for bridge with no "parameters:" setting (#423)
+      [Brian Candler] (LP: #1879673)
+    - When tools.conf does not exist, running cmd "vmware-toolbox-cmd
+      config get deployPkg enable-custom-scripts", the return code will
+      be EX_UNAVAILABLE(69), on this condition, it should not take it as
+      error. (#413) [chengcheng-chcheng]
+    - Document CloudStack data-server well-known hostname (#399) [Gregor Riepl]
+    - test: move conftest.py to top-level, to cover tests/ also (#414)
+    - Replace cc_chef is_installed with use of subp.is_exe. (#421)
+      [Scott Moser]
+    - Move runparts to subp. (#420) [Scott Moser]
+    - Move subp into its own module. (#416) [Scott Moser]
+    - readme: point at travis-ci.com (#417) [Joshua Powers]
+    - New feature flag functionality and fix includes failing silently (#367)
+      [James Falcon] (LP: #1734939)
+    - Enhance poll imds logging (#365) [Moustafa Moustafa]
+    - test: fix all flake8 E121 and E123 errors (#404) [Joshua Powers]
+
+ -- Ryan Harper <ryan.harper@canonical.com>  Fri, 10 Jul 2020 08:51:29 -0500
+
 cloud-init (20.2-45-g5f7825e2-0ubuntu1) groovy; urgency=medium

   * d/control: drop python3-six, python3-unittest2, python3-pep8,
Commit this change? (Y/n): y
[ubuntu/devel c38d0916f] update changelog (New upstream snapshot 20.2-94-g3d06abc2).
 1 file changed, 72 insertions(+)
wrote new-upstream-changes.txt for cloud-init version 20.2-94-g3d06abc2-0ubuntu1.
release with:
sed -i -e "1s/UNRELEASED/groovy/" debian/changelog
git commit -m "releasing cloud-init version 20.2-94-g3d06abc2-0ubuntu1" debian/changelog
git tag ubuntu/20.2-94-g3d06abc2-0ubuntu1
# optionally push your tag: git push <remote> ubuntu/20.2-94-g3d06abc2-0ubuntu1
% sed -i -e "1s/UNRELEASED/groovy/" debian/changelog
% git commit -m "releasing cloud-init version 20.2-94-g3d06abc2-0ubuntu1" debian/changelog
[ubuntu/devel 17f566731] releasing cloud-init version 20.2-94-g3d06abc2-0ubuntu1
 1 file changed, 1 insertion(+), 1 deletion(-)
% git tag ubuntu/20.2-94-g3d06abc2-0ubuntu1
% git diff oddbloke-github/ubuntu/devel HEAD
diff --git a/debian/changelog b/debian/changelog
index be3b57531..87aee9a27 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -68,7 +68,7 @@ cloud-init (20.2-94-g3d06abc2-0ubuntu1) groovy; urgency=medium
     - Enhance poll imds logging (#365) [Moustafa Moustafa]
     - test: fix all flake8 E121 and E123 errors (#404) [Joshua Powers]

- -- Daniel Watkins <oddbloke@ubuntu.com>  Fri, 10 Jul 2020 09:11:25 -0400
+ -- Ryan Harper <ryan.harper@canonical.com>  Fri, 10 Jul 2020 08:51:29 -0500

 cloud-init (20.2-45-g5f7825e2-0ubuntu1) groovy; urgency=medium

@OddBloke OddBloke merged commit e70b2f3 into canonical:ubuntu/devel Jul 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.