How to Update Cloudify Plugins for Python 3
Great news! Cloudify is now adding support for Python 3.6.
Cloudify 5.1 is running on Python 3.6 where previous versions were Python 2.x based. This change was done to comply with the security and IT policies of most organizations as a result of Python 2.x reaching its end of life.
Cloudify assumes complete ownership of updating all our code code and pre-canned plugins so that they match and work in tandem with 5.1, however, all custom made plugins generated over the time by customers, or delivered as stand alone projects must be updated by the customers to become Python 3.x compliant.
This article describes the recommended process for updating a custom plugin to become Python 3.x compliant.
Note! Upgrading a Cloudify manager to 5.1 will fail unless all non-cloudify plugins have been updated to become Python 3 compliant. |
Note! Any plugin that is updated using this process will run on both Cloudify 5.1, and all previous Cloudify versions, hence we recommend running this flow ahead of time and using it for any new generated plugin even if you are still running Cloudify 5.0.5 or earlier. |
Overview of the Process
These are the basic stages of Python 2 -> Python 3 conversion:
- Preparation: Learn to use Futurize - Prepare CICD.
- Perform Futurize Stage 1 Fixes - Basically automatic.
- Perform Futurize Stage 2 Fixes - Requires manual review.
- Handle Unicode changes.
At the end of the process, you will execute your unit tests with Python 3 (in addition to Python 2), and make any necessary updates. However, you will find it useful if you run your unit tests using Python 2 and review them throughout the process.
Please contact Cloudify for support as needed.
Cloudify Libraries
Plugins that use Cloudify Python libraries, for example, Cloudify Common, should build plugins with the oldest required version of that Python library. Cloudify plugins need to support older versions of Cloudify as well as Python 3 versions of Cloudify.
For example, our official plugins all use Cloudify Common. At this time, we officially support all Cloudify versions from version 4.4 - so our plugins are built with Cloudify Common 4.4. For more info on this, see here. When the plugin will be uploaded to a Python 3 manager, the manager will use the Cloudify Common that is in PATH- so it will still use Python Cloudify Common code even though the wagon contains an earlier version. This rule applies for plugins that need to be executed on Python 2 and Python 3 managers. If you want your plugin only to be supported on a Python 3 manager, you can build your plugin on Cloudify Common 5.1.0.
We also use a “_compat” module in our plugins to import code that has different globals in Python 2 and Python 3, for example, httplib, StringIO. See here.
Introduction to Futurize
The futurize tool is used to help convert Cloudify Plugins to Python 3. Futurize suggests code changes for Python 3, and it can also write said code changes to your repositories. Futurize uses fixers in order to process a particular type of modification. For example, the fix_except fixer is a rule for converting a supported Python 2 Exception syntax to the supported Python 3 exception syntax. Futurize can be used to translate large projects by grouping fixers into “stages”. We will use two futurize stages, “stage-1” and “stage-2”. Stage-1 covers safe changes that have no impact on Python 2.6 compatibility. Stage-2 provides code wrappers for Python 3 style changes. (Stage-2 can include Stage-1 changes. However, it’s recommended to perform Stage-1 independently, and only after to add Stage-2 fixers.) Keep in mind that not every futurize change is to be accepted. These will be discussed at the relevant time. Also, later on, more changes not addressed by futurize will be discussed.
Executing
futurize -f lib2to3.fixes.fix_except [PATH TO CODE]
Will display a change like this:
- except SomeException, e:
+ except SomeException as e:
Executing futurize without flags is the equivalent of a dry run.:
> futurize [Options] [PATH TO CODE]
If you want to write Futurize changes, you need the -w flag:
> futurize -wn [Options] [PATH TO CODE]
You can call a futurize fixer-by- fixer, for example,
> futurize -f lib2to3.fixes.fix_except [PATH TO CODE]
Or you can call futurize by stages:
> futurize --stage1 [PATH TO CODE]
Preparing for Continuous Testing
If you have a continuous delivery system, you should add a job for Python 2 and Python 3 compatibility check. This job will execute futurize on the code to ensure that the code is compatible. For example, notice the py3_compat job in this commit in Github.
If you do not have a system for CI, then you should test your plugin continuously whilst making changes. First install the future package in your virtualenv:
> pip install --user future
Next, create a file py3fixers in the directory that you will be executing your continuous testing from:
> touch py3fixers
In this file, add the fixers for the stage you are performing, for example stage-1 or stage-2.
Next, execute these commands after each batch of changes:
> FUTURIZE="futurize ."
while read line; do
[[ "$line" =~ ^#.* ]] && continue
FUTURIZE="${FUTURIZE} ${line}"
done<py3fixers
echo "Running: $FUTURIZE"
$FUTURIZE>futurize_diffs
> if [[ -s futurize_diffs ]]; then
echo "Python-3-incompatible code found"
cat futurize_diffs
exit 1
fi
The first command executes futurize and reads the output. The second command determines whether to notify the user of Python3 of incompatible code.
Stage-1 Fixes
The first step is to run futurize stage-1. From the futurize documentation, this step provides “fixes that modernize Python 2 code without changing the effect of the code”.
> futurize -wn --stage1 [PATH TO CODE DIRECTORY]
For an example of unexceptional output, see this gist.
See this commit for an example of the final changes for stage-1.
Stage-2 Fixes
The second step is to replace deprecated/moved functionality. This is a large task, but involves generally two types of changes: namely changing imports and handling dicts/lists and JSON.
Futurize Stage-2 can provide good insight as to which changes will be necessary, however, it is a good idea to avoid writing the changes (i.e. do not use -wn) without review.
> futurize --stage2 [PATH TO CODE DIRECTORY]
Fixers that require careful review.
- Fix_long: where used for isinstance, prefer numbers. For an example, like here.
- Fix_execfile: simple one-line change to exec(). For an example here.
- Fix_newstyle, see above; for another example see here.
- Fix_filter, see above; futurize will try to wrap those in list(), but instead, prefer rewriting as listcomps. For an example, see here.
- Fix_dict: See “Lists and Dictionaries”. See an example here.
- Change every .iteritems() to .items() - done by futurize
- Futurize tries to replace every d.keys() with list(d.keys()) - prefer just iterating over the dict instead (ie. replace `for k in d.keys()` with `for k in d`)
- Futurize tries to replace every d.items() with list(d.items()) - confirm that we only iterate over it (no indexing), and drop the list() call. This means that fix_dict cannot be added to the fixers list.
Note: dry run these fixers and change things as above, do not add these fixers to your fixers list (in CICD) because its implementation is wrong.
- Fix_map: same as fix_filter, prefer listcomps.
- Fix_xrange: should be OK to just accept what futurize did, see here.
- Fix_zip: similar to fix_dict, futurize will try to add list() calls that are possibly not necessary. Prefer dropping the list() calls.
- Fix_division: the easy one, just confirm that integer division is what we want. If we do want float division, only then add the __future__ import. See an example, here.
- Fix_metaclass. Unless the metaclass is ABCMeta, then do this.
- Fix_cmp: there should be, and so far there was none, usages of cmp (argument for list.sort/sorted). If there is any, replace with a key= function.
- Fix_basestring: Futurize will try to replace basestring with just `str`, but that is not correct (only compatible with py3, not with py2), so we need to review its changes and replace str with text_type from Cloudify Compat module. When doing so, you might also want to change some instances of loading bytes, to be loading unicode instead. See an example, here. Basic guidelines for replace usages of `basestring` with:
- Prefer unicode! `from cloudify._compat import text_type`, that is our unicode type on both 2 and 3.
- Where absolutely necessary (because bytes ARE somehow passed in there), then explicitly do `isinstance(xx, (text_type, bytes))`, but hopefully we will be able to drop all instances of this.
- Fix_unicode_keep_u: replace calls to str(), with calls to:
- If possible, replace with u’{0}’.format(x) - this is preferable because usually formatting makes more sense.
- Call text_type()
- Use different methods entirely, eg. instead of assertIn(message, str(exc)) use assertRaisesRegexp.
Imports
In Python 3, some libraries have been moved or renamed. So when importing code, it is required to condition some imports on the current version of Python.
You may notice that futurize wants to swap out the import paths of certain modules. Many popular modules are handled in Cloudify’s mini-compat module _compat.py.
For example:
if sys.version_info[0] == 2:
try:
from cStringIO import StringIO
except ImportError:
from StringIO import StringIO
else:
from io import StringIO
So instead of adding the above code to your module, you can make a change like this:
> diff --git a/cloudify_azure/resources/base.py b/cloudify_azure/resources/base.py
index 3e9ade9..27a8d16 100644
--- a/cloudify_azure/resources/base.py
+++ b/cloudify_azure/resources/base.py
@@ -12,31 +12,26 @@
-import httplib
+from cloudify._compat import httplib
Allow Cloudify to handle the Python version conditions.
One notable fixer to pay close attention to is libpasteurize.fixes.fix_newstyle. This fixer may suggest that you import builtins modules. The Cloudify compat module should suffice here.
For example,
Lists and Dictionaries
Python 3 requires us to change the way we work with dictionaries. The fixer lib2to3.fixes.fix_dict relates to changes of the following sorts:
- Usages of .iteritems() should be changed to .items(). Futurize can handle this.
- Replace for k in d.keys() with for k in d.
- If you are changing the size of a dictionary during iteration, you should cast it as a list. For example, if you are using del dict_name[key_name] during an iteration, you should cast the dict as a list. For example,
- for key, value in my_dict.items():
- del my_dict[key]
+ for key, value in list(my_dict.items()):
+ del my_dict[key]
See this commit for an example of stage-2 changes.
Unicode Changes
The next step is this most difficult. They are the most specific as they involve your plugins’ “edges”, i.e. input/output/data formats. The Cloudify compat module alleviates some of the pain, via the “unicode sandwich”, or text_type, however these changes will require good understanding of your plugin and of Unicode and places it shows up. Many of the operations here involve basestring, str, JSON dumps, encode, decode, base64, etc.
Here are some example changes:
- https://github.com/cloudify-cosmo/cloudify-common/pull/426
- https://github.com/cloudify-cosmo/cloudify-cli/pull/1067
- https://github.com/cloudify-cosmo/cloudify-openstack-plugin/commit/67087548b9cb39c14884a6a5011b93af78b95383#diff-7797b7d996717db17419bf455f89c89cR29
- https://github.com/cloudify-cosmo/cloudify-azure-plugin/commit/9047594df1302843c0118cd33a5ee88f000251f6
See this commit for an example of unicode changes.
Unittests
At this point, you should be able to execute your unit tests using Python 3. For an example, see here.
Also this commit is a good example of enabling Python3 unit testing.
Comments
2 comments
Will python 3 support be available in a containerized version of the product? In the community edition?
Hi Jack,
Yes, all Cloudify flavors will use Python 3, in containers or VMs.
Please sign in to leave a comment.