Testing your infrastructure as code with Pulumi. Part 2
Hello. Today we are sharing with you the final part of the article “Testing the infrastructure as a code using Pulumi” , a translation of which was prepared specifically for students of the course “DevOps Practices and Tools” .
The tested testing style is a powerful approach; it allows us to test a white box to check the insides of our infrastructure code. However, it somewhat limits what we can verify. Tests are performed based on the in-memory deployment plan created by Pulumi before the direct deployment, and therefore the deployment itself cannot be tested. For such cases, Pulumi has an integration test framework. And these two approaches work great together!
The Pulumi integration testing framework is written in Go, and it is with its help that we test most of our internal code. If the unit testing approach discussed earlier was more like white box testing, then integration testing is a black box. (There are also options for thorough internal testing.) This framework was created in order to take the full Pulumi program and perform various life cycle operations for it, such as deploying a new stack from scratch, updating it with variations, and deleting it, possibly several times. We run them regularly (for example, at night) and as stress tests.
(We are working to ensure that similar integration testing capabilities are in the native language SDK. You can use the Go integration testing framework regardless of the language your Pulumi program is written in).
By running the program using this framework, you can check the following:
As we will soon see, this framework can also be used to perform runtime validation.
To see this in action, we look at the
repository, as our team and the Pulumi community use it to test their own pool of requests, commits and nightly builds.
Below is a simplified test of our example, which does provisioning of S3 bucket and some other objects :
This test goes through the basic life cycle of creating, modifying, and destroying the stack for the
. It will take about a minute to report the test passed:
There are many options for customizing the behavior of these tests. See the
structure for a complete list of options. For example, you can configure the Jaeger endpoint to trace (
), indicate that you expect the test to
during negative testing (
), apply a series of “edits” to the program for successive state transitions (
), and much more. Let's see how to use them to verify application deployment.
The integration mentioned above ensures that our program "works" - it does not crash. But what if we want to check the properties of the resulting stack? For example, that certain types of resources were (or were not) prepared and that they have certain attributes.
The
parameter for
allows us to look at the state recorded by Pulumi after the deployment (post-deployment state) so that we can make additional checks. This includes a complete snapshot of the state of the resulting stack, including configuration, exported output values, all resources and their property values, as well as all dependencies between resources.
To see a basic example of this, let's verify that our program creates one S3 Bucket :
Now, when we run go test, it will not only go through the battery of life cycle tests, but also, after the stack has been successfully deployed, it will perform an additional check of the resulting state.
So far, all tests have been exclusively about deployment behavior and about the Pulumi resource model. What if you want to verify that your prepared infrastructure really works? For example, that the virtual machine is running, the S3 bucket contains what we expect, and so on.
You may have already figured out how to do this: the
option for
is a great opportunity for this. At this point, you run an arbitrary Go test with access to the full state of your program resources. This state includes information such as IP addresses of virtual machines, URLs and everything that is necessary for real interaction with the received cloud applications and infrastructure.
For example, our test program exports a
bucket property called
, which is the full URL at which we can get the customized
. Although we could delve into the status file to find the
and read this property directly, in many cases, our stacks export useful properties, such as this, which are convenient for us to check:
Like our previous runtime checks, this check will be performed immediately after raising the stack, and all this in response to a simple call to
. And this is just the tip of the iceberg - all Go test features that you can write in code are available.
It’s good to be able to run tests on a laptop when a lot of changes are made to the infrastructure to test them before sending them to code reviews. But we and many of our clients test the infrastructure at various stages of the development life cycle:
For each of them, Pulumi supports integration with your favorite continuous integration system. With continuous integration, this gives you the same test coverage for your infrastructure as it does for application software.
Pulumi has support for common CI systems. Here are some of them:
For more information, see the Continuous Delivery documentation.
A very powerful feature that opens up is the ability to deploy ephemeral environments solely for the purpose of acceptance testing. The Pulumi project and stack concept is designed to easily deploy and demolish completely isolated and independent environments, all in a few simple CLI commands or through an integration testing framework.
If you are using GitHub, Pulumi offers the GitHub App , which will help you connect acceptance testing to the pool of requests inside your CI pipeline. Just install the application in the GitHub repository, and Pulumi will add information on the infrastructure preview, updates and test results to your CI and pool of requests:
When using Pulumi for your basic acceptance tests, you will have new automation capabilities that will improve team performance and give confidence in the quality of changes.
In this article, we saw that when using general-purpose programming languages, many software development methods that were useful in developing our applications become available to us. They include unit testing, integration testing, and their interaction for conducting extensive runtime testing. Tests are easy to run on demand or in your CI system.
Pulumi is open source software that is free to use and works with your favorite programming languages and clouds - try it today !
→ The first part
Deployment Testing
The tested testing style is a powerful approach; it allows us to test a white box to check the insides of our infrastructure code. However, it somewhat limits what we can verify. Tests are performed based on the in-memory deployment plan created by Pulumi before the direct deployment, and therefore the deployment itself cannot be tested. For such cases, Pulumi has an integration test framework. And these two approaches work great together!
The Pulumi integration testing framework is written in Go, and it is with its help that we test most of our internal code. If the unit testing approach discussed earlier was more like white box testing, then integration testing is a black box. (There are also options for thorough internal testing.) This framework was created in order to take the full Pulumi program and perform various life cycle operations for it, such as deploying a new stack from scratch, updating it with variations, and deleting it, possibly several times. We run them regularly (for example, at night) and as stress tests.
(We are working to ensure that similar integration testing capabilities are in the native language SDK. You can use the Go integration testing framework regardless of the language your Pulumi program is written in).
By running the program using this framework, you can check the following:
- Your project code is syntactically correct and works without errors.
- The stack and secrets configuration settings work and are interpreted correctly.
- Your project can be successfully deployed in your chosen cloud provider.
- Your project can be successfully upgraded from an initial state to N other states.
- Your project can be successfully destroyed and deleted from your cloud provider.
As we will soon see, this framework can also be used to perform runtime validation.
Simple integration test
To see this in action, we look at the
pulumi/examples
repository, as our team and the Pulumi community use it to test their own pool of requests, commits and nightly builds.
Below is a simplified test of our example, which does provisioning of S3 bucket and some other objects :
example_test.go:
package test import ( "os" "path" "testing" "github.com/pulumi/pulumi/pkg/testing/integration" ) func TestExamples(t *testing.T) { awsRegion := os.Getenv("AWS_REGION") if awsRegion == "" { awsRegion = "us-west-1" } cwd, _ := os.Getwd() integration.ProgramTest(t, &integration.ProgramTestOptions{ Quick: true, SkipRefresh: true, Dir: path.Join(cwd, "..", "..", "aws-js-s3-folder"), Config: map[string]string{ "aws:region": awsRegion, }, }) }
This test goes through the basic life cycle of creating, modifying, and destroying the stack for the
aws-js-s3-folder
. It will take about a minute to report the test passed:
$ go test . PASS ok ... 43.993s
There are many options for customizing the behavior of these tests. See the
ProgramTestOptions
structure for a complete list of options. For example, you can configure the Jaeger endpoint to trace (
Tracing
), indicate that you expect the test to
ExpectFailure
during negative testing (
ExpectFailure
), apply a series of “edits” to the program for successive state transitions (
EditDirs
), and much more. Let's see how to use them to verify application deployment.
Checking Resource Properties
The integration mentioned above ensures that our program "works" - it does not crash. But what if we want to check the properties of the resulting stack? For example, that certain types of resources were (or were not) prepared and that they have certain attributes.
The
ExtraRuntimeValidation
parameter for
ProgramTestOptions
allows us to look at the state recorded by Pulumi after the deployment (post-deployment state) so that we can make additional checks. This includes a complete snapshot of the state of the resulting stack, including configuration, exported output values, all resources and their property values, as well as all dependencies between resources.
To see a basic example of this, let's verify that our program creates one S3 Bucket :
integration.ProgramTest(t, &integration.ProgramTestOptions{ // as before... ExtraRuntimeValidation: func(t *testing.T, stack integration.RuntimeValidationStackInfo) { var foundBuckets int for _, res := range stack.Deployment.Resources { if res.Type == "aws:s3/bucket:Bucket" { foundBuckets++ } } assert.Equal(t, 1, foundBuckets, "Expected to find a single AWS S3 Bucket") }, })
Now, when we run go test, it will not only go through the battery of life cycle tests, but also, after the stack has been successfully deployed, it will perform an additional check of the resulting state.
Runtime tests
So far, all tests have been exclusively about deployment behavior and about the Pulumi resource model. What if you want to verify that your prepared infrastructure really works? For example, that the virtual machine is running, the S3 bucket contains what we expect, and so on.
You may have already figured out how to do this: the
ExtraRuntimeValidation
option for
ProgramTestOptions
is a great opportunity for this. At this point, you run an arbitrary Go test with access to the full state of your program resources. This state includes information such as IP addresses of virtual machines, URLs and everything that is necessary for real interaction with the received cloud applications and infrastructure.
For example, our test program exports a
webEndpoint
bucket property called
websiteUrl
, which is the full URL at which we can get the customized
index document
. Although we could delve into the status file to find the
bucket
and read this property directly, in many cases, our stacks export useful properties, such as this, which are convenient for us to check:
integration.ProgramTest(t, &integration.ProgramTestOptions{ // as before ... ExtraRuntimeValidation: func(t *testing.T, stack integration.RuntimeValidationStackInfo) { url := "http://" + stack.Outputs["websiteUrl"].(string) resp, err := http.Get(url) if !assert.NoError(t, err) { return } if !assert.Equal(t, 200, resp.StatusCode) { return } defer resp.Body.Close() body, err := ioutil.ReadAll(resp.Body) if !assert.NoError(t, err) { return } assert.Contains(t, string(body), "Hello, Pulumi!") }, })
Like our previous runtime checks, this check will be performed immediately after raising the stack, and all this in response to a simple call to
go test
. And this is just the tip of the iceberg - all Go test features that you can write in code are available.
Continuous Infrastructure Integration
It’s good to be able to run tests on a laptop when a lot of changes are made to the infrastructure to test them before sending them to code reviews. But we and many of our clients test the infrastructure at various stages of the development life cycle:
- In each open pool, the request for the test before the merger.
- In response to each commit, to double-check that the merge was performed correctly.
- Periodically, for example, at night or weekly for additional testing.
- As part of performance testing or stress testing, which are usually performed over a long period of time and run tests in parallel and / or deploy the same program several times.
For each of them, Pulumi supports integration with your favorite continuous integration system. With continuous integration, this gives you the same test coverage for your infrastructure as it does for application software.
Pulumi has support for common CI systems. Here are some of them:
For more information, see the Continuous Delivery documentation.
Ephemeral environments
A very powerful feature that opens up is the ability to deploy ephemeral environments solely for the purpose of acceptance testing. The Pulumi project and stack concept is designed to easily deploy and demolish completely isolated and independent environments, all in a few simple CLI commands or through an integration testing framework.
If you are using GitHub, Pulumi offers the GitHub App , which will help you connect acceptance testing to the pool of requests inside your CI pipeline. Just install the application in the GitHub repository, and Pulumi will add information on the infrastructure preview, updates and test results to your CI and pool of requests:
When using Pulumi for your basic acceptance tests, you will have new automation capabilities that will improve team performance and give confidence in the quality of changes.
Total
In this article, we saw that when using general-purpose programming languages, many software development methods that were useful in developing our applications become available to us. They include unit testing, integration testing, and their interaction for conducting extensive runtime testing. Tests are easy to run on demand or in your CI system.
Pulumi is open source software that is free to use and works with your favorite programming languages and clouds - try it today !
→ The first part
All Articles