Photo by Alfons Morales on Unsplash

Good afternoon colleague!

The history of the creation of this post is very interesting, it all started with the fact that I planned to write a little about the PyYAML library and how we domesticated it, but something went wrong, and all this resulted in the creation of a utility pyRegistryStore

Let’s go…

Starting…

Any CI / CD automation has reference books, registries, test environments, analyzers, source code control systems, and this is all controlled by the orchestrator.

Our small group prepares and tests images of virtual machines + development of Ansible roles for installing the necessary software according to the role model.

Task — There is a set of “Roles” — testing automation is required when a new image of a virtual machine appears and the test results successful / unsuccessful are written to the registry.

The task was completed, a script was written using PyYAML, and so far it copes with its task.

This solution seemed to me very interesting and I decided to tell a little about it, but since I am by nature more practitioners than a theoretician, I realized that what examples would be needed, and then everything went wrong as intended.

I could not use the script written at work, because there are quite a few letters on intellectual property in my contract. Therefore, I started writing examples from scratch and only describing the topic of the planned article…

At some point in time when writing the code, I realized that it would be nice to have a utility for creating a registry of objects, and objects should be dynamic, no hardcode, it should be possible for the user to create object templates by himself, spending a minimum of effort.

The result was pyRegistryStore.

Let’s go…

PyYAML

YAML has grown from a markup language to a powerful data serialization tool.

%YAML 1.2
---
YAML: YAML Ain't Markup Language

What It Is: YAML is a human friendly data serialization
standard for all programming languages.

Data serialization is done through strict markup rules and the ability to specify tags, both global and local.

One of the frameworks for Python is PyYAML, which, based on tags, can save Python classes as yaml files.

This is exactly the given serialization/deserialization of objects used in the utility pyRegistryStore

Let’s look at examples of using the utility.

Object “Image VM”

This type of object will describe all existing VM images for which we need to develop.

We will determine the requirements that we need to store information:

  • Name
  • Distribution kit
  • Version
  • Kernel Linux
  • LTS

We describe the object in pyRegistryStore

import objects

class Image(objects.RegistryStore):
"""
Image Object

Parameters
----------
name: str
Name image
distr: str
Distribution kit
version: str
Version distr
kernel: str
Kernel Linux
lts: bool
Long term support
"""
uniq_key = ['name']
desc = 'Object image VM'

def __init__(self) -> None:
super().__init__()

Object “Ansible role”

This object will describe successful tests of passing software rolling on VM images.

We will determine the requirements that we need to store information:

  • Name
  • Image VM
  • Commit GIT
  • State

The following code turned out

import objects


class Role(objects.RegistryStore):
"""
Image Object

Parameters
----------
name: str
Name role
image: str
Name image VM
commit: str
Commit GIT
status: bool
Success
"""
uniq_key = ['name', 'image', 'status']
desc = 'Object Ansible role'

def __init__(self) -> None:
super().__init__()

Pipeline

At the moment, we have an empty object registry, let’s create 2 pipelines to work with this registry.

Creating an image registry

Creating an image is not a tricky process, we take a blank, update packages, install the necessary software and send it further along the business process.

We fill in the register:

pyRegistryStore.py image set name=ubuntu_20.04_v0.image distr=ubuntu version=20.04 kernel=5.4.0-73-generic lts=true
pyRegistryStore.py image set name=centos_7_v0.image distr=centos version=7 kernel=3.10.0-1160.el7.x86_64 lts=true

In total, we got 2 images that we can use further:

pyRegistryStore.py image get | jq
[
{
"_meta": {
"create_time": "2021-05-27 06:07:19.950055",
"update_time": "2021-05-27 06:07:19.950064",
"version": 1,
"uuid": "95abb4e1-ed8f-42a0-b0b0-5496e91a7b58"
},
"distr": "ubuntu",
"kernel": "5.4.0-73-generic",
"lts": true,
"name": "ubuntu_20.04_v0.image",
"version": "20.04"
},
{
"_meta": {
"create_time": "2021-05-27 06:25:26.587628",
"update_time": "2021-05-27 06:25:26.587637",
"version": 1,
"uuid": "ab2fba4b-b9d0-4c05-8860-50f14580395a"
},
"distr": "centos",
"kernel": "3.10.0-1160.el7.x86_64",
"lts": true,
"name": "centos_7_v0.image",
"version": 7
}
]

Create Ansible Role Registry

Let me remind you that the tasks of our group also include the development of Ansible roles for rolling out to VM images, and we must be completely sure that the current version of the role has successfully passed the rolling out and all synthetic tests.

Our testing stack looks like this:

Yes, I was not mistaken in writing the registry, and not pyRegistryStore, since I wrote earlier that this project appeared spontaneously, but I plan to switch to pyRegistryStore.

And so let’s say we have a small role to set up and configure time synchronization with local NTP. Let’s call it ntp-client

The first thing we need is to get a list of all available images:

pyRegistryStore.py image get | jq .[].name
"ubuntu_20.04_v0.image"
"centos_7_v0.image"

In total, we have 2 images, or to be more precise, a list of images on which we must test our role.

And since we have a list, we can organize a loop:

check if the test for the image was successful for the current commit

pyRegistryStore.py role get name=ntp-client image=ubuntu_20.04_v0.image commit=b312abbb05a9be4fe82abcb60d44b7bdd0220bdc status=true

as expected, the list turned out to be empty, which means we need to check this role on this VM image, if we received a list for the output, then we can skip testing.

We run the Molecula, we run the tests, if we catch success, we write the information to the registry

pyRegistryStore.py role set name=ntp-client image=ubuntu_20.04_v0.image commit=b312abbb05a9be4fe82abcb60d44b7bdd0220bdc status=true

well, if the tests were not successful, then just change the status = true key to status = false

Let’s say our tests passed like this:

pyRegistryStore.py role get name=ntp-client | jq
[
{
"_meta": {
"create_time": "2021-05-28 07:01:01.470657",
"update_time": "2021-05-28 07:01:01.470665",
"version": 1,
"uuid": "2fc0dabb-ea56-4eba-92de-75eea562f383"
},
"commit": "b312abbb05a9be4fe82abcb60d44b7bdd0220bdc",
"image": "ubuntu_20.04_v0.image",
"name": "ntp-client",
"status": true
},
{
"_meta": {
"create_time": "2021-05-28 07:05:38.572570",
"update_time": "2021-05-28 07:05:38.572578",
"version": 1,
"uuid": "c13a5266-8949-4d07-9c4e-8955d1cb3a8a"
},
"commit": "b312abbb05a9be4fe82abcb60d44b7bdd0220bdc",
"image": "centos_7_v0.image",
"name": "ntp-client",
"status": false
}
]

Conclusion

And so we got the following:

  • There is a list of images to which we can automatically add new images
  • List of passing tests by role
  • Last successful commit for the image
  • Automate testing of new images

What’s next?

Further, I plan to develop the utility, expanding the functionality. If this utility has its own community, I will be very happy.

I hope I haven’t bored anyone, and this material will be useful to someone, and someone will get some ideas for themselves for inspiration.

Thank you for reading this article.

Sergey Utkin — Technical Expert