Testing AWS Python code with moto

In this article I want to share with you how moto hooks into boto3 and how you can use it to test existing Python code which interacts with your AWS infrastructure.
Recently I have been in a project in which we were working on machine learning pipelines within an AWS architecture. When I joined the project some of the code was already present, but untested. I did not feel comfortable to change and deploy it. That is why I started with setting up tests to better understand what is going on and to be able to change the code in a controlled fashion.

Why moto is a good fit

The code I wanted to test was written in Python and running on AWS infrastructure within Docker containers. Interaction with the AWS infrastructure was implemented with boto3 , the AWS SDK for Python. The initialization of the aws-access happened somewhere within the functions I wanted to test.

Moto is a good fit in such a case, because it enables you to mock out all calls to AWS made in your code automatically.
There is no need for dependency injection. You can take the code you have in place and test it straight away. You just need to enable motos mocking functionality and take some precautions, that is it.

In the following I will show how to enable moto and what precautions to take.

How does moto actually mock things?

The main hook for moto into boto3 is a global list of handlers (named BUILTIN_HANDLERS) in botocore, which is the foundation of boto3.
All handlers within this global list are registered every time a session is instantiated . Once an internal event is emitted, the handlers registered for that kind of event are called.

A before-send-handler for example is executed just before an actual http-request is being send to AWS. If a before-send-handler returns a response, it is used for further processing and the http-request itself is skipped.

This is an example on how to skip http-requests in botocore by returning mock-responses in a custom before-send-handler:

1from botocore.awsrequest import AWSResponse
2from botocore.handlers import BUILTIN_HANDLERS
3from moto.core.models import MockRawResponse
4 
5def my_custom_handler(request, event_name, **kwargs):
6    # event_name is sth like 'before-send.s3.GetObject'
7    return AWSResponse('',200,{},MockRawResponse(''))
8 
9BUILTIN_HANDLERS.append(("before-send", my_custom_handler))

Moto appends such a before-send-handler to the BUILTIN_HANDLERS. The handler is used to return mock-responses from moto mock backends we register. It is appended implicitly, when importing moto in your test code, but does not return (mock) anything by default. Mocking can be achieved by using moto-decorators (or any of the other initializations of moto), which are available for most of the AWS resources.

The moto-decorator registers a mock backend for the scope of the test function. The mock backend is used by the appended before-send-handler to return mock responses. Keep in mind, that the moto-decorator enables the mocking only for the scope of your test function. After your test passes, mock backends and testing credentials are being reset by moto.

Setting up moto and protecting your AWS environment

Interesting, so what happens if I import moto in my test code, but forget to use the moto decorator?
A missing moto decorator might cause changes in your AWS environment if you did not take precautions. Even though moto might have registered the handler in time implicitly, the mock backend would not be initialized and your code would try to access AWS.

That is why you should stick to the recommended usage of moto:

• run your tests with testing-credentials
• set up mocking before any usage/import of boto3

If you fail on setting up the mocks correctly, the testing credentials will still protect you from changing things in your AWS environment.

To be able to properly secure our test setup and provide test credentials to protect our AWS environment, we need to understand how boto3 accesses credentials.

Every time you instantiate a client with boto3, boto3 tries to find credentials to use for accessing AWS. You can see that if you activate the debug-logs for botocore.credentials or you can have a look in the documentation here .

1import logging
2import boto3
3 
4boto3.set_stream_logger('botocore.credentials', logging.DEBUG)
5client = boto3.client('s3')

Would print something like:

[DEBUG] Looking for credentials via: env
[DEBUG] Looking for credentials via: assume-role
[DEBUG] Looking for credentials via: assume-role-with-web-identity
[DEBUG] Looking for credentials via: shared-credentials-file
[INFO] Found credentials in shared credentials file: ~/.aws/credentials

Boto3 looks at parameters passed during initialization in your code first. Secondly it checks if AWS environment variables have been set. Afterwards credentials, config files, … are being parsed.

Now after we know how moto hooks into boto3 and how credentials are being read, these are some precautions steps which should help us to set up a safe test environment which won’t interact with our AWS environment:

1. No parameters when initializing AWS access within the code

Passing no parameters to boto3 when initializing AWS access makes sure we can influence the connection from outside. For example by setting environment variables.

1client = boto3.client('s3')

2. Initialize AWS access within a function

It is crucial to be able to control when exactly the AWS access is initialized and boto3 looks for credentials. Especially because the moto import in our test file implicitly appends a mock handler in botocore, which should be picked up by the boto3 initialization. The easiest way to do this is to have the initialization within a function in your scripts/applications code.

1import boto3
2 
3def start():
4    client = boto3.client('s3')

If you have it outside of a function and import your code in your test file, the import statement alone would cause the access to be initialized due to Pythons module loading . The ordering of your import statements would determine whether or not motos before-send-handler got registered within your client. That is why moto recommends to use inline imports when you have not wrapped your initialization in a function.

3. Initialize moto backends before you call your code

Once we made sure our initialization picked up the moto handler, we still need to register and activate mock backends in moto. If we don’t, our handler would not return mock responses and the requests might reach AWS. Moto initialization can be done in different ways. I prefer the decorator which is quite convenient.
The decorator adds and activates a mock backend for our test function, in this example a s3-backend.

1from moto import mock_s3
2from unittest import TestCase
3 
4class TestS3(TestCase):
5 
6    @mock_s3
7    def test_s3(self):
8        run_s3_code()

Keep in mind that calls to other backends could still reach AWS if you do not initialize the moto mocks.

4. Set testing credentials before you run your tests

It is still possible that you forget about your decorator or initialize a client on top level within your code, which would not pick up the moto handler. Because all this might happen it makes sense to be careful and take additional precautions.

Once you are in a moto testing block (for example test function with moto decorator), you are pretty safe. Moto itself sets testing credentials in the environment if you have at least one mock backend initialized. These credentials are being read by all boto3 initializations within your test code.

It is still a good idea to have some setup code or fixture (if you use pytest), which sets invalid AWS credentials before the test starts to run. This helps you even if you forget about the moto-decorator. If you have module code which gets executed on import, this still might not protect you. Then again it depends on what comes first. The initialization of the client or the AWS-credentials in the environment.

As a solution I personally prefer a layered approach:

• testing credentials within your code
• testing credentials within the default-profile of your ~/.aws/credentials file (fallback)

Every time I execute some code accidentally, forget to initialize moto or anything else, boto3 in worst case would fallback to my credentials file at some point and pick up these invalid testing credentials. In cases where I really want to access AWS I work with profiles.

[default]
aws_access_key_id=testing
aws_secret_access_key=testing

[codecentric]
aws_access_key_id=some_id
aws_secret_access_key=some_key

...

Testing credentials can of course also be set in bash scripts which execute tests, IDE run configurations, pipelines, …

Example

Knowing all this, lets look at a specific case and see if we understand what is going on.

1import boto3
2 
3client = boto3.client('s3') # done outside the method for demonstration purpose
4 
5def run_s3_code():
6    buckets = client.list_buckets()['Buckets']

What is going on when we run the test? Would it access AWS or not and why?

1from moto import mock_s3
2from s3.s3_code import run_s3_code
3from unittest import TestCase
4 
5class TestS3(TestCase):
6 
7@mock_s3
8def test_s3(self):
9    run_s3_code()

The answer is, it would not access AWS. So what is going on here?

1. moto before-send-handler

1from moto import mock_s3

The very first line of our code is the moto import.
This line implicitly adds the before-send-handler within botocore which would not return anything yet.

2. boto3 client initialization

1from s3.s3_code import run_s3_code

The second line causes the initialization of our client which we did not wrap in a function. On initialization the global handlers are registered, also the before-send-handler which is already present (1.).

3. adding moto backends

1@mock_s3
2def test_s3(self):

The decorator adds and initializes a mock-backend which will be used by the before-send-handler. All code we execute within the test function now which uses our client to access s3, would reach the mock-backend instead of AWS.

So what happens if we change the order of imports in a way that

1from moto import mock_s3
2from s3.s3_code import run_s3_code

becomes

1from s3.s3_code import run_s3_code
2from moto import mock_s3

Right. We would access our AWS environment when running the test. Our client would get initialized first and would not register motos before-send-handler. Be careful here and stick to the recommendations: Initialize your AWS access within a function.

Testing code with mocked s3 interaction

Once we setup and secured our testing setup, we can start to write tests.
A simple test could look something like this:

1import json
2from unittest import TestCase
3import boto3
4from moto import mock_s3
5from s3.s3_code import run_s3_code
6 
7class TestS3(TestCase):
8 
9    @mock_s3
10    def test_s3(self):
11        bucket = 'test-bucket'
12        s3_client = boto3.client('s3')
13        s3_client.create_bucket(Bucket=bucket)
14 
15        # GIVEN
16        s3_client.put_object(Bucket=bucket,
17                             Key="market_a/market_infos.json",
18                             Body=json.dumps({
19                                 'products': [
20                                     {'price': 150},
21                                     {'price': 250},
22                                 ]
23                             }))
24 
25        # WHEN
26        run_s3_code()
27 
28        # THEN
29        result = int(s3_client.get_object(Bucket=bucket,
30                                          Key="market_a/total_price.txt")['Body'].read())
31        self.assertEqual(result, 400)

The tested code:

1import json
2 
3import boto3
4 
5def run_s3_code():
6    client = boto3.client('s3')
7 
8    market_infos = client.get_object(
9        Bucket="test-bucket",
10        Key="market_a/market_infos.json"
11    )['Body'].read().decode('utf-8')
12 
13    client.put_object(
14        Bucket="test-bucket",
15        Key="market_a/total_price.txt",
16        Body=str(sum(map(lambda a: a['price'], json.loads(market_infos)['products'])))
17    )

Summing up

When testing your code which interacts with AWS using moto, you want to make sure that there is no real interaction with AWS. In order to do that you should

• put your initialization into functions
• initialize moto mocking properly
• setup testing-credentials in your code and your local default-profile

Moto has been around for quite some time and is a very convenient library to mock AWS services. If you don’t use Python it also has a stand-alone server mode which you can use.

Are you using moto? What are your experiences?