Cloud-Native

Data & KI

Produktentwicklung

Resilienz & IT-Security

Leistungen

Arbeitsweise

Projekte

Berlin

Bielefeld

Dortmund

Erfurt

Frankfurt am Main

Hamburg

Karlsruhe

Leipzig

München

Münster

Nürnberg

Solingen

Stuttgart

Standorte

Blog

Softwerker

Fachmedien

Webinare

Events

Wissens-Hub

Unsere Kultur

Deine Entwicklung

Jobs

Karriere

Über uns

Aktuelles

Kontakt

International contact

Alle Standorte

Du hast Fragen oder möchtest dich mit mir austauschen? Gern - trag einfach deine Kontaktdaten ein.

Kontakt aufnehmen

Marketing & Community Management

Konntest du keine Einträge zu deinem Thema finden?  

STANDORT-TEAM

codecentric Stuttgart

Unser codecentric Team am Standort Stuttgart stellt sich vor.

codecentric Team Stuttgart

Willkommen im Team Stuttgart

<p>Golang, a.k.a. Go, has been around in the industry for quite some time now, but people are still reluctant to just go ahead and use it. To help you get started, follow me on this journey and create your first microservice using Golang, Gin and Docker.</p><h2>Set up your environment</h2><p>There are a few packages needed on your development system in order to start developing – the good news is that it’s not much and easy to install:</p><ol><li><a href="https://golang.org/doc/install" target="_blank">Install Golang</a> </li><li><a href="https://docs.docker.com/get-docker/" target="_blank">Install Docker</a> </li></ol><p>In the upcoming sections I’m assuming that you are using Golang 1.11 or newer. If you, however, use an older version, be aware that some things might not work as described here.</p><p>If you want to get the whole application right from the start, you can go ahead and clone the Git repository right <a href="https://github.com/Andreas-Maier/task-management" target="_blank">here</a>  from GitHub.</p><h2>Let the coding begin</h2><h3>Basics</h3><p>Assuming you successfully installed and tested Golang, let’s see how to get going and set up a webservice using <a href="https://github.com/gin-gonic/gin" target="_blank">Gin</a> .<br /> First of all, we need our main package with the main function in our <em>main.go</em> file (that’s enough mains for now) 🙂</p>

```

package main

import "log"

func main() {
	log.Println("Hello from Go")
}

```

<p>By running <code>go build</code> in your terminal, an executable suitable for your operating system will be created. By running said executable, we should already see the output “Hello from Go”.</p><p>Before adding external dependencies, it is worth considering adding support for versioned modules to our application by running <code>go mod init task-management</code>. Feel free to change the name of the application from <em>task-management</em> to anything you want!<br /> We will see that a new file <code>go.mod</code> appears in your directory. Afterwards, we will find all dependencies we’re adding to our application in there.</p><p>Now let’s put some Gin in there and get our webservice started!</p><p>We can include Gin just like any other dependency in Go (assuming you run version 1.11 or newer) by running <code>go get -u github.com/gin-gonic/gin</code>.<br /> This will download the Gin dependency and add it to the <code>go.mod</code> file. Now we recognize yet another new file in our directory – <code>go.sum</code>. That file contains checksums for direct and indirect dependencies of our application and actually some more things not that relevant for our cause.</p><h3>Setting up Gin and your first endpoint</h3><p>Now that we have Gin included, let’s set up and start the server.</p>

```

package main

import "github.com/gin-gonic/gin"

func main() {
	r := gin.Default()
	r.Run()
}

```

<p>Yes, that’s all we need to set up Gin. Using <code>go build</code> to build the application and then running the created executable will show us where we can reach Gin (should be <code>curl localhost:8080</code> on default). That will give us an HTTP status 404 and some log output indicating that we’re set up and are successfully serving HTTP errors!</p><h3>Serving some data</h3><p>Now, while serving HTTP error codes can be fun, accompany me on our journey and start serving some actual data. Let’s go ahead and think about what data we want to serve. As mentioned above, I would just serve tasks as a start.</p><p>To separate the model from the application logic, we can put the struct into a separate file <code>task.go</code></p>

```

package main

// Task - Model of a basic task
type Task struct {
	Title string
	Body  string
}

```

<p>Now let’s serve a task (as JSON) through a few modifications in our <code>main.go</code> file:</p>

```

package main

import "github.com/gin-gonic/gin"

func handleGetTasks(c *gin.Context) {
	var tasks []Task
	var task Task
	task.Title = "Bake some cake"
	task.Body = `- Make a dough 
	- Eat everything before baking 
	- Pretend you never wanted to bake something in the first place`

	tasks = append(tasks, task)
	c.JSON(http.StatusOK, gin.H{"tasks": tasks})
}

func main() {
	r := gin.Default()
	r.GET("/tasks/", handleGetTasks)
	r.Run()
}

```

<p>If we go ahead and <code>go build</code> and then run the application, we will now be able to see the dummy task as JSON by hitting <code>localhost:8080/tasks/</code></p><p>We can now see that a pointer to the <code>gin.Context</code> object is passed to each handler function we add to Gin (e.g. <code>handleGetTasks</code>). That context can then be used to interact with parameters or objects passed in the request but also to specify what you want to return in terms of status code, headers and content. Furthermore, be aware that the <code>gin</code> type is always available to you if you import it and it offers quite a few handy functions to interact with request and response types. Therefore make sure to check if Gin offers functions you need before writing everything yourself!</p><h3>Setting up a MongoDB for your application</h3><p>There are multiple ways we could choose to just store some data, but to make things a bit more interesting, we could store our data in a <a href="https://www.mongodb.com/" target="_blank">MongoDB</a> .</p><p>As we already have Docker installed on our development environment, we can use that to run our own MongoDB instance locally.</p><p>Executing <code>docker run --name mongodb -e MONGO_INITDB_ROOT_USERNAME=myuser -e MONGO_INITDB_ROOT_PASSWORD=mypassword -e MONGO_INITDB_DATABASE=tasks -p 27017:27017 -d mongo:latest</code> should be fine for development purposes (of course not for productive use!)</p><h3>Storing data in our MongoDB</h3><h4>Connecting to MongoDB</h4><p>As we now have our MongoDB, we need to establish a connection from our Golang application. Luckily, there also is a dependency we can use to make things easier for us: <code>go get -u go.mongodb.org/mongo-driver</code>.</p>

```


...package declaration and imports...

const (
	// Timeout operations after N seconds
	connectTimeout           = 5
	connectionStringTemplate = "mongodb://%s:%s@%s"
)

// GetConnection - Retrieves a client to the DocumentDB
func getConnection() (*mongo.Client, context.Context, context.CancelFunc) {
	username := os.Getenv("MONGODB_USERNAME")
	password := os.Getenv("MONGODB_PASSWORD")
	clusterEndpoint := os.Getenv("MONGODB_ENDPOINT")

	connectionURI := fmt.Sprintf(connectionStringTemplate, username, password, clusterEndpoint)

	client, err := mongo.NewClient(options.Client().ApplyURI(connectionURI))
	if err != nil {
		log.Printf("Failed to create client: %v", err)
	}

	ctx, cancel := context.WithTimeout(context.Background(), connectTimeout*time.Second)

	err = client.Connect(ctx)
	if err != nil {
		log.Printf("Failed to connect to cluster: %v", err)
	}

	// Force a connection to verify our connection string
	err = client.Ping(ctx, nil)
	if err != nil {
		log.Printf("Failed to ping cluster: %v", err)
	}

	fmt.Println("Connected to MongoDB!")
	return client, ctx, cancel
}

```

<p>While this looks like a lot of code, please note that now we have everything we need to start interacting with our MongoDB including timeouts, configurability and connectivity checks. Nevertheless, it might make sense to put that into a separate file e.g. <code>db.go</code>. Also, we should create a file containing our environment variables (e.g. a <code>.env</code> file) with entries like: <code>export MONGODB_USERNAME=myuser</code>.</p><p>In the <code>getConnection()</code> function, be aware that we are using <code>os</code> to interact with our environment. In contrast to many other languages, Golang makes it really easy to get in contact with the environment and read or write environment variables in pretty much just one line of code.</p><h4>Adding a Datamodel</h4><p>Now let’s tackle the next obstacle and start storing some data in our MongoDB. Therefore we need some changes.<br /> To make each entry easily identifiable, we should add an ID field to our task struct:</p>

```

...
type Task struct {
	ID    primitive.ObjectID 
	Title string
...

```

<h4>Interacting with our MongoDB</h4><p>In our <code>db.go</code> we should now add some code to actually create a new task in our MongoDB collection.</p>

```

//Create creating a task in a mongo or document db
func Create(task *Task) (primitive.ObjectID, error) {
	client, ctx, cancel := getConnection()
	defer cancel()
	defer client.Disconnect(ctx)
	task.ID = primitive.NewObjectID()

	result, err := client.Database("tasks").Collection("tasks").InsertOne(ctx, task)
	if err != nil {
		log.Printf("Could not create Task: %v", err)
		return primitive.NilObjectID, err
	}
	oid := result.InsertedID.(primitive.ObjectID)
	return oid, nil
}

```

<p>As you can see, we pass a pointer to the task object into the <em>create</em> function, generate a new ObjectID and then use our MongoDB client to store the task in our collection. As a result, we pass the ObjectID, or an error in case something unintended happens while saving.</p><p>In Golang we can defer the execution of code to when the function is exited. This is a really nice way to free resources and acquired connections like our MongoDB connection. We can make sure to close the connection in any case and schedule it to close automatically once the functions exits.</p><p>Finally, we should hook a handler in our <code>main.go</code> and call the <code>Create</code> function we’ve just written:</p>

```

...
func handleCreateTask(c *gin.Context) {
	var task Task
	if err := c.ShouldBindJSON(&task); err != nil {
		log.Print(err)
		c.JSON(http.StatusBadRequest, gin.H{"msg": err})
		return
	}
	id, err := Create(&task)
	if err != nil {
		c.JSON(http.StatusInternalServerError, gin.H{"msg": err})
		return
	}
	c.JSON(http.StatusOK, gin.H{"id": id})
}

func main() {
	r := gin.Default()
	r.GET("/tasks/", handleGetTasks)
	r.PUT("/tasks/", handleCreateTask)
	r.Run()
}

```

<p>As you can see here, Gin helps us marshal and bind the JSON string into our task to make our lives a bit easier. It also assists us in handling conversion errors on our own with <code>ShouldBind...</code> or Gin handles everything for us with <code>Bind...</code> if we decide to.</p><h4>Let’s build everything and try it!</h4><p>Now <code>go build</code>, specify the environment variables (MONGODB_USERNAME, MONGODB_PASSWORD and MONGODB_ENDPOINT), run the application, and store a new task in our MongoDB:<br /> <code> curl --location --request PUT 'http://localhost:8080/tasks/' \ --header 'Content-Type: application/json' \ --data-raw '{ "Title": "New Task", "Body": "Well this has been some fun already!" }' </code></p><p>Now we can change the <code>GET</code> endpoint to return the data that is actually stored in MongoDB, add <code>DELETE</code> and <code>POST</code> endpoints and so on and so forth.<br /> You can find implementations of those endpoint in the <a href="https://github.com/Andreas-Maier/task-management" target="_blank">GitHub project</a> , but I think now it’s time to move on to our next and last chapter.</p><h3>Containerize it!</h3><p>What would a proper microservice be without a comfy and tiny Docker container? As Golang has huge support for container technologies, this is quite easy. You can actually start from scratch! 😀</p>

```

FROM scratch
ENV MONGODB_USERNAME=MONGODB_USERNAME MONGODB_PASSWORD=MONGODB_PASSWORD MONGODB_ENDPOINT=MONGODB_ENDPOINT
# My application runnable is called gin here
ADD gin / 
CMD ["/gin"]

```

<p>Now if you have built your container and try to run it, you might see a strange error <code>standard_init_linux.go:211: exec user process caused "exec format error"</code>. This happens if you’ve built the Golang application on a different operating system than you try to run it on. In case of Windows, Docker might even have trouble locating a file named <em>gin</em> at all.</p><p>We have multiple ways of dealing with this issue:</p><ol><li>Build the Go application for the OS we run the application on</li><li>Do a multistage Docker build</li></ol><p>If you want to follow route one on our journey, you simply need to execute <code>GOOS=linux go build</code> when building the application and next time you build our container, everything should be fine. This is a really handy feature as you can compile Golang applications for different operating systems by just specifying the GOOS environment variable.</p><p>If you want a multistage Docker build, you need a few more lines of code but your overall build process will be much simpler and more automated:</p>

```

FROM golang:latest AS builder
ADD . /app
WORKDIR /app
RUN go mod download
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o /main .

FROM scratch
ENV MONGODB_USERNAME=MONGODB_USERNAME MONGODB_PASSWORD=MONGODB_PASSWORD MONGODB_ENDPOINT=MONGODB_ENDPOINT
COPY --from=builder /main ./
ENTRYPOINT ["./main"]
EXPOSE 8080

```

<p>Now you can build and run your container and are good to go!</p><h2>Famous last words</h2><p>Now, having built this application, you might have realized that Golang is actually quite nice. Yes, some things are rather expressive and might seem a bit repetitive (error handling, I’m looking at you), but you can build applications really fast and they actually are quite readable if you are used to C/C++/C# or other related languages. Overall, it’s a fun language to use. Make sure to go ahead and gather your own experiences with it! 🙂</p><p>If you want to check out the whole project, fill any gaps or just start experimenting based on this application – <a href="https://github.com/Andreas-Maier/task-management" target="_blank">Clone me on GitHub!</a> . Pull requests are very welcome! 🙂</p><p>Share your thoughts with me using the comment section below and check out the blog – there might be more you’re interested in, e.g. <a href="https://blog.codecentric.de/en/2019/04/serverless_gcp-look_ma_no_servers/">Serverless Golang on GCP</a> !</p>

Golang, Gin & MongoDB – Building microservices easily

Imagine the following situation: You are building a serverless application using e.g. lambdas, you setup your system using CDK (or CloudFormation) and you store your data in Aurora Serverless. How would you automate your database schema adaptations or prefilling tables in your database? You don't know when your lambdas are going to get started. You don't know how many there may be. Maybe you have multiple services that connect to the same datastore and don't have a good fit where to include your upgrades?

Why not manage your database schema changes within your infrastructure setup with CDK? In this blog article I will describe how you can set up a CDK stack that handles database changes with Liquibase when you deploy your infrastructure changes using an example for you to try alongside in your own AWS account.  


## Setup/Preconditions

As mentioned above, the approach will use CDK. Therefore I recommend [installing CDK on your machine alongside its prerequisites](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_prerequisites), if you want to follow along and deploy the example in your own AWS account. Also Docker will be used when deploying the stacks into your AWS account and therefore needs to be [installed on your machine](https://docs.docker.com/get-docker/), if you want to follow along.

If you have not used CDK before, you should also make sure that the AWS account you want to use is [bootstrapped](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_bootstrap) and if it is not, run `cdk bootstrap aws://ACCOUNT-NUMBER/REGION`.

## How do we do all this?
### Architecture/Component Overview

The example is composed out of multiple stacks to separate the different components that are used. 
1. We have an application stack that sets up our networking components, like a VPC, security groups and creates our database and database migration stack.
2. Next we have a stack to create and configure everything for our Aurora Serverless database.
3. Last but not least we have a stack that includes everything we need to do automatic database upgrades with Liquibase. 

The picture below depicts the used constructs and stacks to give more of an overview over the solution.

![Overview_Serverless_RDS_Upgrades.png](https://media.graphassets.com/zpHwcJsfTqaktAL5yl9V)

As I will reference files directly in the following description I recommend checking out the source code of the example application, or at least to have a look on [GitHub](https://github.com/Andreas-Maier/serverless-rds-updates)! 

### Networking infrastructure

The networking setup for this setup is nothing special and can be found in the `application-stack.ts`. However, it is needed as the Aurora Serverless will be deployed in a VPC and in most cases in the private subnets of said VPC. Therefore we need some configuration to enable our Lambda to talk to Aurora Serverless. 

We use two separate security groups to configure access to the database and the Lambda and by adding an ingress rule we can allow our database migration security group to pass through our database security group on the default postgresql port `5432`.  

```
const databaseSecurityGroup = new SecurityGroup(
    this,
    "DatabaseSecurityGroup",
    {
        securityGroupName: "DatabaseSecurityGroup",
        vpc,
    }
);
const databaseMigrationSecurityGroup = new SecurityGroup(
    this,
    "DatabaseMigrationSecurityGroup",
    {
        securityGroupName: "DatabaseMigrationSecurityGroup",
        vpc,
    }
);
databaseSecurityGroup.addIngressRule(
    databaseMigrationSecurityGroup,
    Port.tcp(5432),
    "allow access for database migration lambda"
);
```


### RDS Aurora Serverless

As for the networking components, the RDS Aurora Serverless setup is rather standard and can be found in the `database-stack.ts`. The database credentials are created when the database stack is created and the credential is safely stored in AWS SecretsManager for our database migration Lambda to access it later. 

```
this.credentials = Credentials.fromGeneratedSecret(username, {
    secretName: "/aurora/databaseSecrets",
});
```

We also limit the subnets our database cluster can use to only our private subnets.
```
const databaseSubnetGroup = new SubnetGroup(this, "DatabaseSubnetGroup", {
    description: "SubnetGroup for Aurora Serverless",
    vpc: props.vpc,
    vpcSubnets: props.vpc.selectSubnets({
        subnetType: SubnetType.PRIVATE_WITH_EGRESS,
    }),
});
```



As of right now, only a few Postgresql versions are supported with Aurora Serverless, hence the Postgres engine version 11.16. I recommend checking for newer supported versions when you set up an Aurora Serverless cluster using Postgres.

```
this.database = new ServerlessCluster(this, "DemoCluster", {
    engine: DatabaseClusterEngine.auroraPostgres({
        version: AuroraPostgresEngineVersion.VER_11_16,
    }),
    credentials: this.credentials,
    defaultDatabaseName: "demo",
    vpc: props.vpc,
    subnetGroup: databaseSubnetGroup,
    securityGroups: [props.databaseSecurityGroup],
    scaling: {
        autoPause: Duration.minutes(5),
        minCapacity: AuroraCapacityUnit.ACU_2,
        maxCapacity: AuroraCapacityUnit.ACU_4,
    },
});
```

The scaling options are set to a minimum for this demo case and also include shutting down the database if there is no activity for 5 minutes. This can result in longer startup times.

## Database migration with CDK Custom Resource

Now that the surrounding infrastructure is defined, we can take care of the infrastructure components we need for the database migration. 

In total we will need two policy statements for the Lambda, one role that is used while executing the Lambda, the Lambda function itself, a custom resource provider and a custom resource. 

As mentioned while setting up the Aurora Serverless, we store the credentials to access our database in AWS SecretsManager. Therefore we need to allow our Lambda to access the SecretsManager by creating the `secretsManagerPolicyStatement`. Also, as hinted while creating the network infrastructure, we need to allow our Lambda to run in a VPC. In AWS this is achieved through an elastic network interface (ENI) that the Lambda creates and connects to. The lambda can then access services that are located in private subnets of the VPC through the ENI that is being deployed in the VPC. This is allowed through the policy statement `createENIPolicyStatement`.

![VPC-ENI.png](https://media.graphassets.com/8JQF73q7TP2a9L0BqHnR)

For the policies to take effect while executing the lambda, we create a role that extends the `AWSLambdaBasicExecutionRole` managed policy by adding the previously described policies. 

Let's take a look at the lambda function. As the function is using Liquibase and is written in Java, we need to set the runtime to a currently supported Java runtime. In this case we opt for Java 11 which will be `Amazon Corretto`. 

In order for the Lambda to fetch the secret from SecretsManager, we need to pass the region and secret name into the function. In this case we can use environment variables, as the secret name itself does not need special protection.
 
To further tell the lambda to connect to our VPC during execution time, we need to set the VPC, SecurityGroup and subnets to use by the Lambda. As our function has no need to be available to the general public, we can limit it to only use private subnets. Also, to allow the lambda to create and attach to ENIs we need to set our previously created role for the Lambda.   

#### Building and bundling the Maven function for Lambda 

Lambdas in CDK come with a nice feature. You can define a build process for your Lambda when you define the Lambda itself. When you specify where your code assets are located you have a set of options to apply also. One of those is a bundling option. This may be used to build your code and package it. 
For our use case this is quite handy, as we need to run a maven build and by specifying it in CDK directly, we do not need to add any other build and package systems. 

AWS provides a set of predefined images you can use to build your software. The Java 11 building image contains Java and Maven and thereby everything we need to build and package our software. The Lambda function will simply get the code by putting the built and bundled artifact in the `asset-output` directory. 


```
const databaseMigrationFunction = new Function(
    this,
    "DatabaseMigrationFunction",
    {
        runtime: Runtime.JAVA_11,
        code: Code.fromAsset(path.join(__dirname, "./database-migration"), {
            bundling: {
            image: Runtime.JAVA_11.bundlingImage,
            user: "root",
            outputType: BundlingOutput.ARCHIVED,
            command: [
                "/bin/sh",
                "-c",
                "mvn clean install " +
                "&& cp /asset-input/target/databaseMigration.jar /asset-output/",
            ],
            },
        }),
        handler: "migration.Handler",
        environment: {
            DATABASE_SECRET_NAME: props.credentials.secretName!,
            REGION: props.database.env.region,
        },
        vpc: props.vpc,
        vpcSubnets: props.vpc.selectSubnets({
            subnetType: SubnetType.PRIVATE_WITH_EGRESS,
        }),
        securityGroups: [props.databaseMigrationSecurityGroup],
        timeout: Duration.minutes(5),
        memorySize: 512,
        role: databaseMigrationFunctionRole,
    }
);
```

In order to run our database migration Lambda when deploying our infrastructure, we need to specify a CDK custom resource provider. By referencing our Lambda in the `onEventHandler`, our Lambda is called for every create, modify and delete event of the custom resource. 

```
const databaseMigrationFunctionProvider = new Provider(
    this,
    "DatabaseMigrationResourceProvider",
    {
        onEventHandler: databaseMigrationFunction,
    }
);
```


Now that we have a custom resource provider, we can go ahead and specify the custom resource associated. If we would only specify the custom resource and point it to the custom resource provider service token, our Lambda would be executed once upon creation of the resource. Now this is a problem, if you want to have continuous database upgrades without destroying resources regularly. Therefore, we need the custom resource to change with every deployment so liquibase can check for necessary database upgrades. 

An easy solution to achieve this is by adding a date property to the resource and setting the current date and time so we have an ever changing value. 

```
new CustomResource(this, "DatabaseMigrationResource", {
    serviceToken: databaseMigrationFunctionProvider.serviceToken,
    properties: {
        date: new Date(Date.now()).toUTCString(),
    },
});
```



### The database migration code

The database migration is done using [Liquibase](https://www.liquibase.org/). As Liquibase requires Java to run we also write our Lambda in Java. AWS offers good [documentation](https://docs.aws.amazon.com/lambda/latest/dg/lambda-java.html) on how to write Lambda functions using Java so most of the general handler setup and dependencies can be found there. For our use case, we need to add secretsmanager, liquibase-core and postgresql to our Maven dependencies in the `pom.xml`. 

Now as you may notice in our `Handler.java` class we do not implement the basic `RequestHandler` as a regular lambda would do. By extending the `AbstractCustomResourceHandler` we give our lambda a few methods to implement depending on what happens to our CDK resource. For this example we want to run the database migration when the cdk resource is created as well as when an update is happening. For a CDK resource deletion Liquibase is not executed – this however can be different depending on your use case! 

For the `migrateDatabase`method to actually migrate something, we first need to fetch the secret from AWS SecretsManager in order to connect to the database. This way the secret is managed in a single safe place, access to the secret can be granted in a detailed manner and there is no secret information buried in environment variables. 

The rest of the `migrateDatabase` method is kept rather simple but still creates a database connection, starts up a Liquibase context and runs the upgrades described in the Liquibase changelog. 

The Liquibase changelog, located in `src/main/resources/liquibase`, just contains two changesets for the database schema creation and an example table so we can see that the migration happened and something has changed. 


## Deploy the Stacks

As we have taken a look at the components and some of the configuration we can deploy our CDK stacks. If you want to try this yourself, I encourage you to checkout the [GitHub repo](https://github.com/Andreas-Maier/serverless-rds-updates) with all the necessary code. I've left out some more general stuff in this blog post, but you can check out the whole thing on Github and run it from there. 

To deploy our three stacks to your locally configured AWS account, all you need to do is run `cdk deploy --all`. 

Also good to know if you want to clean up afterwards: All you need to run is `cdk destroy --all` and optionally delete any remaining log groups in CloudWatch. (Destroying the stacks may take a while)

## What happened in the AWS account/Where can I see results?

Let's assume you deployed the stack and get this nice message in your console: 

```
 ✅  ServerlessRdsUpdatesStack/DatabaseMigrationStack
```


Now you might wonder, what happened? How can I see if the database migration was executed?

Let's check the AWS console to see if everything worked as it should have. So first of all log into the AWS account you deployed the stack to and navigate to the CloudFormation Service. There you should be able to find our three stacks in `CREATE_COMPLETE` status. 

![Cloudformation_Status_Serverless_RDS_Upgrades.png](https://media.graphassets.com/oSDoUHzHTMWROfF9tWNO) 

With this checked we can go ahead and see if our database migration ran and created the table we wanted. Therefore we need to open AWS SecretsManager and RDS. 

In AWS SecretsManager we need to check the list of secrets for a secret named `/aurora/databaseSecrets`. When checking the details we need to take note of the ARN of this secret, because this is what we need to connect to our database in a minute. If you want to, you can also check username, password and other details that may be needed to connect to the database by clicking on the `retrieve the secret value` button.

In the RDS service we can open up the Query Editor and directly access a database using a secret from the SecretsManager. So let's open up the RDS Query Editor, choose the database we just created, choose to connect using a Secrets Manager ARN, and enter the ARN of the secret we just noted. Note if you used the demo, the database name will just be demo and afterwards you should be ready to connect to the database. 

After connecting, there is a Query Editor that we can use to query our database. We can see that there is a query prefilled and we can use that to see if our Liquibase migration was executed. Therefore we can just hit the run button to execute the query. After the query is executed we can find our new users table in the result windows. 

![RDS_Tables_Query_Result_Serverless_RDS_Upgrades.png](https://media.graphassets.com/c4WTsrtnTN2EXv7Kze39) 

To now interact with the table you can feel free to select. insert or delete operation on the table `demo.users` using the Query Editor.

## Conclusion

Automation of deployment and change processes is a very important topic for many projects. Moving faster and getting features out quicker and more reliable is pretty much inevitable. Due to this, automating every bit of your deployment is a necessity. Therefore, if you run into a situation where you are wondering how to automate upgrades of your relational database this solution might be something for you to consider. However, if you have no good reason to use a relational database, I encourage you to also have a look at schemaless options like DynamoDB. 

How to upgrade your Aurora Serverless database schema using CDK and Lambda

3 Steps to Access AWS Resources with Google Sign-In

How to create an AWS Lambda Authorizer for an Amazon API Gateway

Senior Software Developer

Software Engineer & Consultant

<p>Der erste Hackergarten Stuttgart fand am 23. Juli 2015 statt. Das Ziel dieses Hacks war ein verschlüsseltes Ping-Pong zweier embedded Funküberträgern.</p><p>Die Idee für den Hack stammt vom Heimautomatisierungsprojekt eines Hackers. Anscheinend gibt es Nachholbedarf im Bereich verschlüsselter Funkübertragungs-Protokolle – sowohl Software als auch Hardware. Wenn man etwas richtig haben will, muss man es eben selbst machen (DIY – Do it yourself).<br /> Das Ziel dieses Hacks ist der Aufbau einer Funkverbindung zwischen Gateway und KNotizn, mit einem anschließend verschlüsseltem Ping-Pong (Anfrage-Antwort).</p><h2>Verwendete Hardware</h2><p>Beim Hackathon war folgende Hardware im Einsatz:</p><ul><li>1x Seeeduino v3.0</li><li>1x <a href="http://www.sainsmart.com/sainsmart-uno-r3-atmega328-au-development-board-compatible-with-arduino-uno-r3.html" name="SainSmart UNO R3">SainSmart UNO R3</a> </li><li>2x RFM69HW: -120dBm 433/868/915Mhz RF Transceiver Module</li><li>Kleinigkeiten wie Kabel, Lötzinn…</li></ul>

```
Tipp:
Aufgepasst, der RFM69HW hat eine Betriebsspannung von 3.3V. Der angeschlossene Arduino benötigt also einen 3,3V Ausgang oder ihr verwendet ein Netzteil mit 3.3V Spannung.
```

<h2>Verwendete Software</h2><ul><li><a href="https://www.arduino.cc/en/Main/Software" name="Arduino Software">Arduino-Software</a> </li><li><a href="https://github.com/LowPowerLab/RFM69" name="RFM69 library for RFM69W and RFM69HW">RFM69 Bibliothek für RFM69W und RFM69HW</a> </li><li><a href="https://github.com/LowPowerLab/SPIFlash" name="Arduino library for read/write access to SPI flash memory chips">Arduino Bibliothek für Lese-/Schreibzugriff auf SPI-Flash-Memory-Chips</a> </li></ul>

```
Tipp:
Der Seeeduiono hat einen FTDI-USB-Chip. In dieser Anleitung wird gut erklärt, wie man den richtigen 
USB-Treiber installiert.
```

<h2>Aufbau</h2><p>Da man mit der Arduino-Software nur einen seriellen Monitor anzeigen lassen kann, empfielt es sich, ein separates Programm hierfür zu haben (z.B. <a href="http://freeware.the-meiers.org/">CoolTerm</a> ), einen weiteren Rechner, oder den seriellen Monitor von Arduino zu Arduino umzuschalten.</p>

```
Tipp:
Wir hatten zwei Macs. Das erleichterte das gleichzeitige Hacken.
```

<h3>Hardware-Aufbau</h3><div><a href="https://media.graphassets.com/c9UfvJrQIu1557ToyK2A"><img src="https://media.graphassets.com/c9UfvJrQIu1557ToyK2A" alt="Arduino-RFM69HW-Verdrahtung" width="700" height="657" /></a> <p>Die Verdrahtung von Arduino und RFM69HW Quelle: https://electronichamsters.files.wordpress.com/2014/09/dd_basic.png</p></div>

```
Tipp:
Die Verbindungsdrähte und „Antenne“ des RFM69HW waren bereits angelötet (die Antenne ist für die 
Verwendung in einem Raum vermutlich redundant). Wir mussten also nur die Drähte des RFM69HW an 
den Arduino anschließen.
```

<h3>Software-Aufbau</h3><p>Die Arduino-Software lässt sich wie im <a href="https://www.arduino.cc/en/Guide/HomePage" name="Getting Started Guide">Getting Started Guide</a>  beschrieben installieren. Als nächstes kopiert man die Bibliothek-Verzeichnisse (RFM69 and SPIFlash) in das Arduino-Bibliotheken-Verzeichnis (Standard ist hierfür $home/Arduino/libraries (analog unter Windows)).</p><p>Die <a href="https://github.com/LowPowerLab/RFM69" name="RFM69 library">RFM69-Bibliothek</a>  hat auch entsprechende Beispiele (im Verzeichnis Examples).</p><p>Die Beispiele namens Gateway und Node sind bereits einsatzbereit. Versuche das Kompilieren: Öffne die Beispiel-Datei in der Arduino-Software und führe die Verifizieren-Funktion aus (der Knopf mit dem Häkchen-Zeichen oder im Menü unter Sketch-> Überprüfen/Kompilieren)</p><p>Code-Änderungen</p>

```
#define FREQUENCY RF69_433MHZ
//#define FREQUENCY RF69_868MHZ
//#define FREQUENCY RF69_915MHZ
```

<p>in</p>

```
//#define FREQUENCY RF69_433MHZ
#define FREQUENCY RF69_868MHZ
//#define FREQUENCY RF69_915MHZ
```

<p>Lade ein Beispiel auf jeweils einen Arduino und überprüfe die Funktion mit dem seriellen Monitor (Menü Werkzeuge->serieller Monitor) . Es sollten Textausgaben zu sehen sein.</p>

```
Notiz: 
Setze den baud-Wert des seriellen Monitors auf 115200 oder ändere diesen Wert in beiden Code-
Beispielen.

Wir haben das Gateway-Beipiel auf den SainSmart UNO und das Node-Beispiel auf den Seeeduiono 
hochgeladen. Das Hochladen auf den Seeeduiono hat seine Tücken. 
Man muss der Reset-Knopf während des Hochladens gedrückt halten und sofort nach Abschluss 
loslassen, sonst funktioniert es nicht. 
Wird der Reset-Knopf zu früh oder zu spät losgelassen, muss man den Vorgang wiederholen.
```

<p>Wenn beide Arduinos vorbereitet sind und eine Stromversorgung haben, sollte man nun auf dem seriellen Monitor sehen, dass der Gateway Daten vom Node empfängt.</p><h2>Ergebnis</h2><p>Trotz der einen oder anderen Hürde haben wir es geschafft, eine Verbindung zwischen Gateway und Node aufzubauen. Die Ausgabe des Node zeigte, dass gesendet wurde und die Ausgabe des Gateway, dass es auch empfangen wurde.</p><p>Somit haben wir den Ping-Teil des Zieles erreicht.</p><p>Nichtsdestotrotz haben wir es nicht geschafft, dass der Node eine Empfangsbestätigung vom Gateway empfing bzw. ausgab. Der Pong-Teil des Zieles wurde somit nicht erreicht.</p><div><a href="https://media.graphassets.com/Bdn1nZB9Tcy4J8yxzXby"><img loading="lazy" src="https://media.graphassets.com/Bdn1nZB9Tcy4J8yxzXby" alt="On-Scene Setup" width="700" height="525" /></a> <p>Aufbau am Platz</p></div><h2>Ausblick</h2><p>Erste Aufgabe für den nächsten Hackergarten Stuttgart wird es daher sein, das Pong zum Laufen zu bekommen. Außerdem haben wir uns ein neues Zeil ausgedacht, sozusagen eine Daseinsberechtigung für unseren Funkübertragungsaufbau. Da die Idee aus einem Heimautomatisierungs-Projekt stammt, ist es naheliegend, tatsächlich einen Hackergarten zu beschaffen und für diesen ein automatisches Bewässerungssystem zu entwickeln. Natürlich muss das System dann auch mittels verschlüsseltem Funk kommunizieren.</p><h2>Update</h2><p>Wir haben ein Hackergarten Github-Repo angelegt:<br /> <strong><a href="https://github.com/hackergarten-stuttgart/EWHA" target="_blank">hackergarten-stuttgart/EWHA</a> </strong></p><p>Es gibt ein Fortsetzungsprojekt:<br /> <strong><a href="http://pilhuhn.blogspot.de/2015/10/driving-servo-on-arduino-from-remote.html" target="_blank">Driving a Servo on Arduino from a remote Arduino over secured radio</a> </strong></p>

DIY leichtgewichtige verschlüsselte Funkübertragung

<p>The first hacker garden Stuttgart took place on the 23rd of July 2015. One of the hacks was a DIY encrypted lightweight radio transmitters ping-pong.</p><p>The idea originates from a home automation project from one of the hackers. Apparently there is a lack of encrypted radio transmission protocols, respectively the hardware and software components on the market. If you want something done right, you usually have to DIY.<br /> So the task for this hacker garden was to initiate a connection between a gateway and a node. And establish an encrypted ping-pong communication.</p><h2>Used Hardware</h2><p>Here is a list of the Hardware we used:</p><ul><li>1x Seeeduino v3.0</li><li>1x <a href="http://www.sainsmart.com/sainsmart-uno-r3-atmega328-au-development-board-compatible-with-arduino-uno-r3.html" name="SainSmart UNO R3">SainSmart UNO R3</a> </li><li>2x RFM69HW: -120dBm 433/868/915Mhz RF Transceiver Module</li><li>Small stuff like cables, soldering tin, …</li></ul>

```
Tip:
Mind that the RFM69HW is a 3.3V device, so your Arduino has to bring a 3.3v output, or you add an 
additional 3.3V power adaptor.
```

<h2>Used Software</h2><ul><li><a href="https://www.arduino.cc/en/Main/Software" name="Arduino Software">Arduino-Software</a> </li><li><a href="https://github.com/LowPowerLab/RFM69" name="RFM69 library for RFM69W and RFM69HW">RFM69 library for RFM69W and RFM69HW</a> </li><li><a href="https://github.com/LowPowerLab/SPIFlash" name="Arduino library for read/write access to SPI flash memory chips">Arduino library for read/write access to SPI flash memory chips</a> </li></ul>

```
Tip:
The Seeeduiono uses a FTDI USB chip so you should look at this how-to, in order to have the sufficient USB drivers.
```

<h2>Setup</h2><p>Since you can only show one Serial Monitor with the Arduino Software you should have another serial port monitor (e.g. <a href="http://freeware.the-meiers.org/">CoolTerm</a> ), another computer, or just switch from one Arduino to the other.</p>

```
Tip:
 We had two Macs so it was easy to hack in parallel.
```

<h3>Hardware Setup</h3><div><a href="https://media.graphassets.com/c9UfvJrQIu1557ToyK2A"><img src="https://media.graphassets.com/c9UfvJrQIu1557ToyK2A" alt="Cable-Connection-Image" width="700" height="657" /></a> <p>The cable setup of the Arduinos and the radio transceivers<br />Source: https://electronichamsters.files.wordpress.com/2014/09/dd_basic.png</p></div>

```
Tip:
The connecting cables and antenna for the RFM69HW were already soldered on (you probably don't 
need the antenna for a setup in the same room). So we only had to connect the cables of the 
RFM69HW to the Arduino.
```

<h3>Software Setup</h3><p>Install the Arduino Software as described at this <a href="https://www.arduino.cc/en/Guide/HomePage" name="Getting Started Guide">Getting Started Guide</a> .<br /> Copy the library directories (RFM69 and SPIFlash) into the Arduino libraries directory. (Standard for the lib dir is $home/Arduino/libraries (analogly with Windows))</p><p>The <a href="https://github.com/LowPowerLab/RFM69" target="_blank">RFM69 library</a>  comes with some examples (in the Examples directory).</p><p>The Gateway and Node example are ready to go. Try to compile the examples:<br /> Open the example files in Arduino Software and use the Verify function (the button with the tick mark by menu Sketch->Verify/Compile).</p><p>Code-Changes:</p>

```
 #define FREQUENCY RF69_433MHZ
 //#define FREQUENCY RF69_868MHZ
 //#define FREQUENCY RF69_915MHZ
```

<p>to</p>

```
//#define FREQUENCY RF69_433MHZ
#define FREQUENCY RF69_868MHZ
//#define FREQUENCY RF69_915MHZ
```

<p>Upload one example on each Arduino and check if it is working by opening the Serial Monitor (menu Tools->Serial Monitor).<br /> You should see some text outputs.</p>

```
Note:
Set the baud of the Serial Monitor to 115200 or change it in the examples code.
We uploaded the Gateway example on the SainSmart UNO and the Node example on the Seeeduino.
The upload on the Seeeduino is a little tricky. You have to hold the reset button, then upload and just 
when finished uploading release the button. If you release the button too early/late the upload will fail 
and you'll have to start over.
If both Arduino setups are on power, you should see that the Gateway receives data from the node.
```

<h2>Result</h2><p>With some hurdles, we managed to establish a connection between Gateway and Node. The Node output showed that it was sending. And the Gateway output showed that it was receiving.<br /> So we managed to get a ping.<br /> Nonetheless we didn’t manage to get the Node to receive an acknowledge request of the Gateway. Hence the pong for the task failed.</p><div><a href="https://media.graphassets.com/Bdn1nZB9Tcy4J8yxzXby"><img loading="lazy" src="https://media.graphassets.com/Bdn1nZB9Tcy4J8yxzXby" alt="On-Scene Setup" width="700" height="525" /></a> <p>On-Scene Setup</p></div><h2>Lookout</h2><p>The first task for the next hacker garden will be to get the pong working.<br /> But we design-thought (I heard that design thinking is the new brainstorming) for another major task, one could say a purpose for this radio transmitter setup. Since our task at this hacker garden originates from a home automation project, it would seem natural to actually procure a hacker <em>garden</em> with an automatic irrigation system – of course communicating via an encrypted radio transmission.</p><h2>Update</h2><p>We created a github-repo for our hacker garden project:<br /> <strong><a href="https://github.com/hackergarten-stuttgart/EWHA" target="_blank">hackergarten-stuttgart/EWHA</a> </strong></p><p>There is a followup project:<br /> <strong><a href="http://pilhuhn.blogspot.de/2015/10/driving-servo-on-arduino-from-remote.html" target="_blank">Driving a Servo on Arduino from a remote Arduino over secured radio</a> </strong></p>

DIY encrypted lightweight radio transmission

Software Engineering, Software Consulting, Software Crafting, Team Effectiveness Consulting

Lead Marketing & Communications

Leiter Region Südwest

Rust – Einstieg

Rust – an introduction

Senior IT-Consultant

Office Manager Region Südwest

Senior Sales Manager

Software Entwickler

Senior Active Sourcer

Software Developer & Consultant

IT Consultant

Senior Industrial Solutions Developer and Consultant

Code Splitting mit React Lazy und React Suspense

<p>Eine populäre Methode, um REST APIs zu dokumentieren ist Swagger 2. Für Spring(-Boot)-Projekte bietet sich <a href="https://github.com/springfox/springfox" target="_blank">Springfox</a>  an. Springfox integriert sich recht nahtlos in ein Spring-Projekt und stellt für konfigurierte REST Endpoints eine Browser-basierte <a href="https://swagger.io/tools/swagger-ui/" target="_blank">Swagger UI</a>  Representation zur Verfügung. Mittels Annotations im Code können umfangreiche Details und Informationen zur API-Dokumentation hinzugefügt werden, z. B. erweiterte Informationen zu http-Statuscodes oder Beschreibungen zu einzelnen Feldern von Ressource-Modellen. Allerdings gibt es keine OOTB-Verbindung zwischen Swagger UI und der Spring-Security-Konfiguration. Dieser Artikel beschreibt eine Springfox-Swagger-UI-Erweiterung, um beide miteinander zu verbinden.</p><h2>Übersicht Spring Security und Spring REST Controller</h2><p>Durch die Verknüpfung von normalem Spring-Code mit Springfox API Annotations zur Generierung der Swagger-Dokumentation lässt sich ein altbekanntes Problem umgehen, nämlich dass Code und Dokumentation auseinanderlaufen und damit die Dokumentation häufig nicht dem aktuellen Code entspricht.</p><h3>Spring-Security-Beispiel</h3><p>Allerdings lässt sich mit Springfox die verwendete Spring-Security-Konfiguration nicht dokumentieren. Spring Security sichert per (Java-)Konfiguration ein REST-API ab und stellt eine Brücke zwischen Authentifizierung und Autorisierung her. Das folgende Code-Snippet beispielsweise sichert den REST-Endpoint „/user“ für GET- und POST-Zugriffe ab:</p><div>

```java5
@Configuration
public class SecurityConfig extends WebSecurityConfigurerAdapter {
    @Override
    protected void configure(HttpSecurity httpSecurity) throws Exception {
        super.configure(httpSecurity);
 
        httpSecurity.csrf().disable();
        httpSecurity.httpBasic();
 
        httpSecurity.authorizeRequests()
            .antMatchers(HttpMethod.POST, "/user").hasRole("admin")
            .antMatchers(HttpMethod.GET, "/user/*").hasAnyRole("admin", "user");
 
        httpSecurity.authorizeRequests().anyRequest().fullyAuthenticated();
    }
}
```

</div><p>Die obige Spring-Security-Konfiguration definiert, dass die Ressource „user“ sowohl von Usern mit der Rolle „user“ als auch mit der Rolle „admin“ abgefragt (GET) werden darf, während der schreibende Zugriff (POST) auf dieselbe Ressource nur durch Nutzer mit der Rolle „admin“ erlaubt ist.</p><h3>Spring-REST-Controller-Beispiel</h3><p>Eine Spring-Security-Java-Konfiguration ist relativ einfach zu lesen und zu verwalten. Ein Problem ist aber, dass diese Config von der eigentlichen Ressource, also dem REST Endpoint, losgelöst ist:</p><div>

```java5
@RestController
public class MyRestController {
    @RequestMapping(method = RequestMethod.GET, value = "/user/{username}", produces = APPLICATION_JSON_VALUE)
    @ApiOperation("Get details of a user with the given username")
    @ApiResponses(value = {
            @ApiResponse(code = 200, message = "Details about the given user"),
            @ApiResponse(code = 401, message = "Cannot authenticate"),
    })
    public UserDTO getExampleData(@Valid
                                  @ApiParam("Non-empty username") @PathVariable(name = "username", required = true) String username) {
        /*
            Get your user resource somehow and return
         */
    }
    @PostMapping(value = "/user", consumes = MediaType.APPLICATION_JSON_VALUE, produces = MediaType.APPLICATION_JSON_VALUE)
    @ResponseStatus(HttpStatus.CREATED)
    @ApiOperation("Create a new user or update an existing user (based on username)")
    @ApiResponses(value = {
            @ApiResponse(code = 200, message = "User was successfully updated"),
            @ApiResponse(code = 201, message = "User was successfully created"),
            @ApiResponse(code = 401, message = "Cannot authenticate"),
    })
    public void createUser(@Valid @RequestBody UserDTO userDTO) {
        /*
            Save the user resource somehow
        */
    }
}
```

</div><p>Der hier dargestellte REST-Controller wird durch Springfox API Annotations dokumentiert (Springfox ist sehr viel mächtiger als hier dargestellt, siehe <a href="https://springfox.github.io/springfox/docs/current/" target="_blank">Springfox Reference Documentation</a> ). Ein Blick auf das generierte Swagger UI zeigt allerdings, dass die Spring-Security-Konfiguration aus der Klasse SecurityConfig nicht in die Dokumentation einfließt. D.h. es ist nicht ersichtlich, welche Ressource durch welche Rollen aufrufbar ist:</p><p><a href="https://media.graphassets.com/Gan9EFrR0uFO9nXNzpN9"><img width="700" height="987" src="https://media.graphassets.com/Gan9EFrR0uFO9nXNzpN9" alt="Standard Swagger UI" /></a> </p><p>Wie kann also Springfox so erweitert werden, dass die Spring-Security-Konfiguration automatisch Teil der Swagger-Dokumentation wird und damit eine höhere Kohäsion zwischen Implementierung und Dokumentation entsteht?</p><h2>Swagger-UI-Erweiterung für Spring Security</h2><p>Im Artikel <a href="https://blog.codecentric.de/2017/09/springfox-swagger-inkludieren-markdown/" target="_blank">Springfox Swagger mit externem Markdown erweitern</a>  hat Markus Höfer von codecentric bereits dargestellt, wie einfach Springfox mit Custom Annotations erweitert werden kann. Darauf basierend entstand der hier beschriebene Lösungsweg, um Spring-Security-Rollen in das Swagger UI zu übernehmen.</p><p>Folgende grundlegende Schritte sind notwendig, um eine Custom Annotation für Springfox Swagger UI zu erstellen:</p><ol><li> Erzeugen einer Custom Annotation</li><li> Implementierung eines Custom Springfox OperationBuilderPlugins</li><li> Custom Annotation an alle REST-Controller Endpoints hinzufügen</li></ol><p>Weiterhin muss die Spring-Security-Konfiguration für die Springfox-Erweiterung „auslesbar“ gemacht werden, was im Folgenden zuerst beschrieben wird.</p><h3>Spring Security auslesbar machen</h3><p>Wie ein REST-API mit Spring Security gesichert werden kann, wurde bereits <a href="#spring-security-beispiel">oben kurz dargestellt</a> .</p><p>Spring Security verwendet Ant-Matcher Pattern, um http-Methoden und URL-Pfade zu spezifizieren, auf die definierte Rollen zugreifen dürfen. Diese Konfiguration ist jedoch „write-only“, d.h. die Klasse <em>HttpSecurity</em> (genauer gesagt die Klasse <code>ExpressionUrlAuthorizationConfigurer.ExpressionInterceptUrlRegistry</code> aus dem Package <code>org.springframework.security.config.annotation.web.configurers</code>) lässt nicht zu, dass die Konfiguration der Ant-Matcher wieder ausgelesen wird. Das ist aber für unseren Use Case notwendig, denn wir wollen genau diese Information in unserer Swagger-Doku haben, müssen sie also auslesen können. Die vorgeschlagene Lösung verwaltet daher die Ant-Matcher-Konfiguration in einer eigenständigen Klasse namens <code>HttpMethodResourceAntMatchers</code>, die sowohl der HttpSecurity Configuration als auch der Swagger-Doku als Spring Component zur Verfügung steht:</p><div>

```java5
public class HttpMethodResourceAntMatchers {
    Logger logger = LoggerFactory.getLogger(HttpMethodResourceAntMatchers.class);
    // list of all defined matchers
    List matcherList = new ArrayList<>();
    /**
     * Applies all existing Matchers to the provided httpSecurity object as .authorizeRequest().antMatchers().hasAnyRole()
     * @param httpSecurity
     * @throws Exception
     */
    public void configure(org.springframework.security.config.annotation.web.builders.HttpSecurity httpSecurity) throws Exception {
        for (HttpMethodResourceAntMatcher matcher : this.matcherList) {
            httpSecurity.authorizeRequests().antMatchers(matcher.getMethod(), matcher.getAntPattern()).hasAnyRole(matcher.getRoles());
        }
    }
    /**
     * Add a new Matcher with HttpMethod and URL-Path
     * @param method
     * @param antPattern
     * @return
     */
    public Role antMatchers(org.springframework.http.HttpMethod method, String antPattern) {
        // create a new matcher
        HttpMethodResourceAntMatcher matcher = new HttpMethodResourceAntMatcher(method, antPattern);
        // add matcher to list of matchers
        this.matcherList.add(matcher);
        // return a Role wrapper object, which forces the user to add the role(s) to the matcher
        Role role = new Role(matcher, this);
        return role;
    }
    /**
     * Helper class for a builder-like creation pattern
     */
    public class Role {
        HttpMethodResourceAntMatcher matcher;
        HttpMethodResourceAntMatchers matchers;
 
        public Role(HttpMethodResourceAntMatcher matcher, HttpMethodResourceAntMatchers matchers) {
            this.matcher = matcher;
            this.matchers = matchers;
        }
        /**
         * Define which role has access to the given resource identified by the Ant-Matcher
         * @param role
         * @return
         */
        public HttpMethodResourceAntMatchers hasRole(String role) {
            matcher.setRoles(role);
            return matchers;
        }
        /**
         * Add a list of roles which have access to the given resource identified by the Ant-Matcher
         * @param roles
         * @return
         */
        public HttpMethodResourceAntMatchers hasAnyRole(String... roles) {
            matcher.setRoles(roles);
            return matchers;
        }
    }
}
```

</div><p>Die Klasse <code>HttpMethodResourceAntMatchers</code> imitiert die Spring HttpSecurity <code>antMatchers()</code> und <code>hasRole()</code>. Die hier referenzierte Klasse <code>HttpMethodResourceAntMatcher</code> ist ein einfacher Pojo mit den Membern „httpMethod“, „antPattern“ und „roles“.</p><p>Die eigentliche Konfiguration der Matcher mithilfe der Klasse <code>HttpMethodResourceAntMatchers</code> findet durch die Custom Spring Component <code>MatchersSecurityConfiguration</code> statt:</p><div>

```java5
@Component
public class MatchersSecurityConfiguration {
 
    private HttpMethodResourceAntMatchers matchers;
 
    /**
     * Returns all http matchers
     * @return
     */
    public HttpMethodResourceAntMatchers getMatchers() {
        if (matchers == null) {
            matchers = new HttpMethodResourceAntMatchers();
            matchers.antMatchers(HttpMethod.POST, "/user").hasRole("admin")
                    .antMatchers(HttpMethod.GET, "/user/*").hasAnyRole("admin", "user");
        }
        return matchers;
    }
}
```

</div><p><code>MatchersSecurityConfiguration</code> wird sowohl in die Spring Security Config injiziert als auch in die nachfolgende Swagger-Hilfsklasse für unsere Custom Annotation.</p><p>Damit kann unsere Spring Security Config folgendermaßen angepasst werden:</p><div>

```java5
@Configuration
public class SecurityConfig extends WebSecurityConfigurerAdapter {
 
    @Autowired
    private MatchersSecurityConfiguration matchersSecurityConfiguration;
 
    @Override
    protected void configure(HttpSecurity httpSecurity) throws Exception {
        super.configure(httpSecurity);
 
        httpSecurity.csrf().disable();
        httpSecurity.httpBasic();
        // add matchers for REST API to httpSecurity
        this.matchersSecurityConfiguration.getMatchers().configure(httpSecurity);
 
        httpSecurity.authorizeRequests().anyRequest().fullyAuthenticated();
    }
}
```

</div><p>Nachdem wir die Spring-Security-Konfiguration angepasst haben, müssen die Swagger Annotation und die Springfox-Erweiterung erstellt werden.</p><h3>Custom Annotation</h3><p>Für unseren Use Case erstellen wir die Custom API Annotation <code>ApiRoleAccessNotes</code>:</p><div>

```java5
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
 
@Target({ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
public @interface ApiRoleAccessNotes {
}
```

</div><p>Die Annotation hat keinerlei Parameter, sie ist ein Marker für unser <code>OperationBuilderPlugin</code>.</p><h2>Custom Springfox OperationBuilderPlugin</h2><p>Damit die oben definierte Annotation verwendet werden kann, muss das <code>OperationBuilderPlugin</code> Interface als Spring Component implementiert werden. Die <em>apply</em>-Methode wird vom Springfox <code>DocumentationPluginsManager</code> für jede REST-Resource im Classpath aufgerufen:</p><div>

```java5
@Component
@Order(SwaggerPluginSupport.SWAGGER_PLUGIN_ORDER)
public class OperationNotesResourcesReader implements springfox.documentation.spi.service.OperationBuilderPlugin {
    private final DescriptionResolver descriptions;
 
    @Autowired
    private MatchersSecurityConfiguration matchersSecurityConfiguration;
 
    final static Logger logger = LoggerFactory.getLogger(OperationNotesResourcesReader.class);
 
    @Autowired
    public OperationNotesResourcesReader(DescriptionResolver descriptions) {
        this.descriptions = descriptions;
    }
 
    @Override
    public void apply(OperationContext context) {
        try {
            Optional methodAnnotation = context.findAnnotation(ApiRoleAccessNotes.class);
            if ( !methodAnnotation.isPresent() || this.matchersSecurityConfiguration == null) {
                // the REST Resource does not have the @ApiRoleAccessNotes annotation --> ignore
                return;
            }
            String apiRoleAccessNoteText = "Accessible by users having one of the following roles: ";
            HttpMethodResourceAntMatchers matchers = matchersSecurityConfiguration.getMatchers();
            // get all configured ant-matchers and try to match with the current REST resource
            for (HttpMethodResourceAntMatcher matcher : matchers.matcherList) {
                // get the RequestMapping annotation, which contains the http-method
                Optional requestMappingOptional = context.findAnnotation(RequestMapping.class);
                if (matcher.getMethod() == getHttpMethod(requestMappingOptional)) {
                    AntPathMatcher antPathMatcher = new AntPathMatcher();
                    String path = context.requestMappingPattern();
                    if (path == null) {
                        continue;
                    }
                    boolean matches = antPathMatcher.match(matcher.getAntPattern(), path);
                    if (matches) {
                        // we found a match for both http-method and URL-path, get the roles
                        // add the roles to the notes. Use Markdown notation to create a list
                        apiRoleAccessNoteText = apiRoleAccessNoteText + "\n * " +  String.join("\n * ", matcher.getRoles());
                    }
                }
 
            }
            // add the note text to the Swagger UI
            context.operationBuilder().notes(descriptions.resolve(apiRoleAccessNoteText));
        } catch (Exception e) {
            logger.error("Error when creating swagger documentation for security roles: " + e);
        }
    }
 
    private HttpMethod getHttpMethod(Optional requestMappingOptional) {
        if (!requestMappingOptional.isPresent()) return null;
        if (requestMappingOptional.get().method() == null || requestMappingOptional.get().method()[0] == null)
            return null;
        RequestMethod requestMethod = requestMappingOptional.get().method()[0];
        switch (requestMethod) {
            case GET:
                return HttpMethod.GET;
            case PUT:
                return HttpMethod.PUT;
            case POST:
                return HttpMethod.POST;
            case DELETE:
                return HttpMethod.DELETE;
        }
        return null;
    }
 
    @Override
    public boolean supports(DocumentationType delimiter) {
        return SwaggerPluginSupport.pluginDoesApply(delimiter);
    }
}
```

</div><p>Innerhalb der Klasse wird geprüft, ob die aktuell von Springfox verarbeitete Ressource unsere <code>ApiRoleAccessNotes</code>-Annotation besitzt. Ist dies der Fall, dann holen wir uns alle Spring Security Matcher und prüfen, welche davon auf die die aktuelle Ressource passen – sowohl in Bezug auf die Http-Methode also auch auf den konfigurierten URL-Path.</p><h3>Custom Annotion @ REST Controller</h3><p>Abschließend muss die Custom Annotation <code>@ApiRoleAccessNotes</code> zu allen REST-Ressourcen hinzugefügt werden, für die wir im Swagger UI die konfigurierten Rollen sehen wollen:</p><div>

```java5
@RestController
public class MyRestController {
    @RequestMapping(method = RequestMethod.GET, value = "/user/{username}", produces = APPLICATION_JSON_VALUE)
    @ApiOperation("Get details of a user with the given username")
    @ApiResponses(value = {
            @ApiResponse(code = 200, message = "Details about the given user"),
            @ApiResponse(code = 401, message = "Cannot authenticate"),
    })
    @ApiRoleAccessNotes
    public UserDTO getExampleData(@Valid
                                  @ApiParam("Non-empty username") @PathVariable(name = "username", required = true) String username) {
        /*
            Get your user resource somehow and return
         */
    }
 
    @PostMapping(value = "/user", consumes = MediaType.APPLICATION_JSON_VALUE, produces = MediaType.APPLICATION_JSON_VALUE)
    @ResponseStatus(HttpStatus.CREATED)
    @ApiOperation("Create a new user or update an existing user (based on username)")
    @ApiResponses(value = {
            @ApiResponse(code = 200, message = "User was successfully updated"),
            @ApiResponse(code = 200, message = "User was successfully created"),
            @ApiResponse(code = 401, message = "Cannot authenticate"),
    })
    @ApiRoleAccessNotes
    public void createUser(@Valid @RequestBody UserDTO userDTO) {
        /*
            Save the user resource somehow
        */
    }
}
```

</div><p>Im Swagger UI sieht das Ganze dann folgendermaßen aus:</p><p><a href="https://media.graphassets.com/u31BPS3cTGyZ4C7176i5"><img loading="lazy" width="634" height="916" src="https://media.graphassets.com/u31BPS3cTGyZ4C7176i5" alt="Custom Swagger UI" /></a> </p><h2>Fazit</h2><p>Mit ein paar wenigen Kniffen haben wir es geschafft, dass unsere Dokumentation wieder etwas näher an den Sourcecode gerückt ist. Der Sourcecode steht unter <a href="https://github.com/HenningWaack/SpringFoxSwaggerExtensionDemo" target="_blank">https://github.com/HenningWaack/SpringFoxSwaggerExtensionDemo</a>  zur Verfügung. Viel Spaß damit, und ich freue mich über alle Kommentare zu diesem Post!</p>

Springfox Swagger Extensions für Spring Security

<p><b>BigQuery can help with building an ML system for production with a short time to market.</b></p><ul><li><b>Follow industry standards.</b><span> Agile methods, the MLOps framework and focus on an MVP are helpful.</span></li><li><b>Model improvement is not everything.</b><span> A good model evaluation as well as versioning and metadata tracking are worth the effort.</span></li><li><b>BigQuery also has downsides.</b><span> The API and slot management can be a hindrance at first.</span></li></ul><p><span>In my bachelor thesis I examined how medium-sized companies can effectively set up their own machine learning (ML) system using limited resources.</span><span><br /> </span><span>The reason for that was, that in 2020, despite the coronavirus pandemic, 71% of companies increased their investments in ML <a href="https://info.algorithmia.com/hubfs/2020/Reports/2021-Trends-in-ML/Algorithmia_2021_enterprise_ML_trends.pdf?hsLang=en-us" target="_blank">[1]</a> . </span><span> However, the resources of a medium-sized company remain finite. In addition, only 53% of ML proof of concepts (POC) were deployed in production and realised their promised value between 2018 and 2020, according to Gartner  <a href="https://www.gartner.com/en/documents/3987446/accelerating-ai-deployments-paths-of-least-resistance" target="_blank">[2]</a> </span><span>.</span></p><p><span>In the context of my thesis, I cooperated with the Mosecker GmbH, a leading family-owned, medium-sized wholesale company for domestic appliances, their core business being customer satisfaction rather than IT. We carried out a project to develop a production-ready ML system prototype, to test a combination of agile methods and the MLOps framework as a development approach. </span></p><p><span>The approach proved to be helpful.</span><span><br /> </span><span>However, in the following, I will focus on the usage of the managed cloud service </span><b>BigQuery ML</b><span> we used as the basis for the ML system. I will first introduce the project and system in general and then go into detail about a few particularities.</span></p><h2><b>The Project</b></h2><p><span>The project’s goal was to create a recommendations system prototype for personalized product recommendations in the web store of Mosecker in less than ten weeks. </span><span>The prototype should also be suitable for use in production. To bring the idea to the market as quickly as possible and validate it.</span></p><p><span>For the project, delivery data between 01.01.2019 and 15.09.2021 was made available. </span><span>These included</span></p><ul><li><span>a total of 4,823,891 transactions </span></li><li><span>5,897 different customers </span></li><li><span>910,002 individual items</span></li></ul><h3><b>System Setup</b></h3><p><span>Since one of the primary conditions is to create the prototype with the least resources and effort possible, all required resources were sourced from the cloud right from the start to avoid ramp-up costs and delays due to hardware provisioning and configuration.<br /> </span><span>Another advantage of the cloud strategy is the on-demand use and billing of storage and computing resources. This makes it possible to easily scale up the prototype at a later time. </span></p><p><span>I mainly used the </span><b>BigQuery</b><span> service and its built-in ML capabilities from the Google Cloud Platform (GCP) for this project, as I had previously used it for a similar use case. At the moment of writing, the first processed terabyte of every month was free, which meant even fewer costs. </span><span>For a detailed description of how to create a recommendation model with BigQuery ML, I recommend two blog posts from my codecentric colleagues, Timo Böhm and Niklas Haas. The first  <a href="https://blog.codecentric.de/2021/07/bigquery-ml-schnelles-training-recommendation-modell/" target="_blank">[3]</a> </span><span> is about the technical part; the second <a href="https://blog.codecentric.de/2021/07/kuerzere-time-to-market-fuer-ml-modelle-durch-googles-bigquery-ml/" target="_blank">[4]</a>  </span><span>is about the business perspective. </span></p><p><span>To complete the system with metadata tracking, I used the open-source tool MLflow. </span><span>In concept, the ML system then consisted of three parts:</span></p><ol><li><span>BigQuery in GCP to process the data, train and evaluate the model, and create the recommendations.</span></li><li><span>A Docker image as a pipeline to be executed (locally), which manages and monitors the individual steps of the process.</span></li><li><span>MLflow, collecting various metadata about the training run and the created model during execution, these are then comparably presented in a web UI.</span></li></ol><p><span>Figure 1 shows the interaction of the components during execution.</span></p><div><img src="https://media.graphassets.com/R6ivzSnBSmOuSbDfdy5F" alt="Schematic of training pipeline" width="1024" height="332" /><p>Figure 1: Schematic of training pipeline, communication between pipeline and BigQuery</p></div><p><span>This sums up to a single GCP project and two images, one to run the pipeline, the other to host the MLflow Web UI.<br /> </span><span>The training data is provided as CSV files in a mounted volume.</span></p><p><span>I have to note here that it was not the project’s scope to provide continuous monitoring, though this is in general a crucial step when following industry-standard MLOps principles  <a href="https://services.google.com/fh/files/misc/practitioners_guide_to_mlops_whitepaper.pdf" target="_blank">[5]</a> . </span></p><h3><b>Pipeline steps</b></h3><p><span>The pipeline is separated into five steps:</span></p><ol><li><ol><li><b>Data Import</b><span>: Upload CSV files to BigQuery</span></li><li><b>Data preparation</b><span>: Data cleaning and creation of the training dataset</span></li><li><b>Model training: </b><span>Creating the model and extraction of the recommendation </span></li><li><b>Model evaluation: </b><span>Validating the model based on mathematical methods</span></li><li><b>Prediction serving: </b><span>Exporting the top recommendations per customer in a CSV file</span></li></ol></li></ol><p><span>The MLOps white paper of Google <a href="https://services.google.com/fh/files/misc/practitioners_guide_to_mlops_whitepaper.pdf" target="_blank">[5]</a> </span><span> served as a template for the structure. </span><span>It was then adapted to the project’s needs and technical requirements of the BigQuery ML service. This resulted in the pipeline steps described above.</span></p><h2><b>Implementation peculiarities and beneficial factors</b></h2><p><span>The use of BigQuery as an integral part of the system presented some challenges. But I also want to highlight some beneficial factors for completing the project.</span></p><h3><b>Start simple with a baseline model</b></h3><p><span>One of them is to start with a minimal baseline model. </span><span>Even though the idea of technical cut-through is not new, I want to highlight it here. </span><span>Since BigQuery ML uses SQL to train ML models, creating a baseline model with default values is easy and fast.  </span><span>This allowed me to focus more on creating the system in the first place, to make a technical cut-through including recommendation serving and simplifying the further improvement of the model.</span></p><p><span>Note:<br /> </span><span>Using an established ML use case made the development of the model easier, which is why I was able to focus on the technical cut-through. However, it might make more sense to focus on the model first in less common use cases, where you can not rely on a broad knowledge base for the model.</span></p><h3><b>Evaluation of the model in BigQuery ML</b></h3><p><span>One of the biggest challenges for me was to evaluate the created model. </span><span>BigQuery ML does evaluate the model automatically during its creation. But Google does not provide more detailed information about the metric calculation or implementation, making it a black box. </span><span>For this reason, I implemented the calculation of the </span><b>mean average precision@k</b><span> (mAP@k) <a href="https://nlp.stanford.edu/IR-book/pdf/irbookonlinereading.pdf" target="_blank">[6]</a>  </span><span>myself. You can find the SQL statement in this gist  <a href="https://gist.github.com/SputnikTea/a8b8c258d603826140b0f373fd837c30" target="_blank">[7]</a> </span><span>.</span></p><p><span>During the project, I realised the mAP@k exhibited an unexpected behaviour in some edge cases.<br /> </span><span>As soon as the remaining recommendations of a query were all negative (0), these were no longer considered in the precision calculation, as seen in figure 2. For this reason, I designed the mean Precision (mP@k) and implemented it in the project. Here, every recommendation in a query is included in the precision, regardless of being positive or negative, as seen in figure 3.</span></p><div><img loading="lazy" src="https://media.graphassets.com/ewAZDKjERy6h1DRwGwEv" alt="Calculation of mean average precision@k" width="1024" height="317" /><p>Figure 2: Mean average precision@k, common case and edge case</p></div><div><img loading="lazy" src="https://media.graphassets.com/OKFzCGHKSHeiCnmE6B2t" alt="Calculation of mean precision@k" width="1024" height="317" /><p>Figure 3: Mean precision@k, common case and edge case</p></div><p><span>If this explanation was too short, I recommend this more detailed article <a href="https://towardsdatascience.com/breaking-down-mean-average-precision-map-ae462f623a52" target="_blank">[8]</a>  </span><span>by Ren Jie Tan. </span><span>For evaluating the model, I preferred to use the mP@k because I find it more understandable and thus closer to reality. The SQL statements for calculating the metrics can be found in this gist <a href="https://gist.github.com/SputnikTea/a8b8c258d603826140b0f373fd837c30" target="_blank">[7]</a> . </span></p><h3><b>Master the slots</b></h3><p><span>As much as the cloud can save costs, it can also cause them, if you are not careful. </span><span>Particular caution is required when booking computing resources in BigQuery, so-called slots <a href="https://cloud.google.com/bigquery/docs/slots" target="_blank">[9]</a> .</span><span> These must be used for training a matrix factorisation model; other algorithms can also be used with on-demand resources.<br /> The minimum slot amount (100) costs 4.80€/hour in the Netherlands. A model training for our data usually took only 20 to 30 minutes, so the costs were manageable. </span><span>However, booking (managing) the slots is only possible via the web interface, SQL or the CLI of the cloud SDK. Although I have automated the step in the pipeline using SQL, I wanted to make sure that the monetary damage is limited in case of an error. </span></p><p>Therefore I used the cloud scheduler and a cloud function to regularly drop all slots at midnight. Fortunately, dropping the slots is also possible in the Python client library. The cloud function code and CLI commands for the scheduler can be found in this gist <a href="https://gist.github.com/SputnikTea/333a84adf2c2f97ec43a5ab0f5621430" target="_blank"><span>[10]</span></a> . <span>In retrospect, I noticed that one could have alternatively used a scheduled query. But this might complicate the integration into a monitoring system.</span></p><h3><b>Tracking metadata with MLflow</b></h3><p><span>A month into the project, I lost track of the model versions and their performance. </span><span>Therefore, I included MLflow tracking <a href="https://www.mlflow.org/docs/latest/tracking.html" target="_blank">[11]</a>  </span><span>into the pipeline and used its web UI to compare the models.</span><span><br /> </span><span>Doing this, following things were useful:</span></p><ul><li><span>logging the SQL queries with `log_text()` for debugging</span></li><li><span>saving the pipeline YAML configuration as a file artefact with `log_artifact()`</span></li><li><span>using dataset IDs as model version indicators and table prefixes for different runs</span></li></ul><div><img loading="lazy" src="https://media.graphassets.com/iGhOhE1NTZyyX1OdL4P1" alt="MLflow UI, comparing two runs" width="1024" height="358" /><p>Figure 4: Example for comparing two runs in MLflow</p></div><p><span>I should mention that BigQuery ML is intended for the fast testing of model ideas by data analysts. Its use as an ML system is not intended, so features like metadata tracking or a proper client API are (currently) missing.<br /> </span><span>For ML systems, Google intends the use of Vertex AI, but this has a steep learning curve, especially for a beginner with a small project. AutoML systems, on the other hand, were very limiting in their configuration.<br /> </span><span>By the declarative work with SQL, BigQuery ML offered just the right amount of simplification and freedom to create the prototype according to my needs. Unfortunately, without a proper API, but the use of SQL queries had its own advantages. By working declaratively, there were hardly any bugs in the data transformation part.</span></p><h2><b>Project Results</b></h2><p><span>After eight weeks of development, the prototype has fulfilled all basic requirements:</span></p><ul><li><span>Creation and serving of a new model with one command</span></li><li><span>Relevant recommendations</span></li><li><span>Tracking of metadata</span></li></ul><p><span>In addition, the model did not recommend in majority previously bought items. Instead, also new items were recommended, which the stakeholders evaluated as reasonable.</span></p><div><img loading="lazy" src="https://media.graphassets.com/jkwcM906T4KKHXZ2JMVJ" alt="Model quality metrics by iteration" width="1024" height="633" /><p>Figure 5: Model quality by different metrics in the course of the project</p></div><p><span>Figure 5 shows the different model quality metrics in the course of the project, as well as their various results. For more information on the discrepancy between offline evaluation and user validation, I recommend this paper <a href="https://arxiv.org/pdf/1908.08328.pdf" target="_blank">[12]</a>  </span><span>by Dietmar Jannach.<br /> </span><span>The collection of the Ambassador User Validation, which was the key metric for the project, was facilitated by the completed ML system. </span></p><h2><b>Conclusion</b></h2><p><span>BigQuery ML is not intended to be an ML system; this becomes apparent when you want to use it in a script. But it reduces a lot of complexity to simple SQL statements without taking too much freedom in the model configuration. This makes creating a minimal viable product or prototype fast and easy.</span></p><p><span>The mentioned challenges in creating an ML system, like slot management or model evaluation and SQL calculations, are a hindrance. But for me, especially for smaller projects, they are made up for by ease of use and simultaneous freedom.</span></p><p><span>In the end, BigQuery did the job. It enabled me to build a machine learning system prototype with a continuous training pipeline, metadata tracking and prediction serving within eight weeks.</span></p><h2><b>Acknowledgements</b></h2><p><span>A huge thanks goes to Holger Sickmann and Fabrice Lepach at the Mosecker GmbH for making this project possible by providing the data and insight domain knowledge.</span><span><br /> </span><span>I also want to thank my supervisors, Michael Bücker at the FH Münster and Niklas Haas from codecentric, for keeping me on track and helping me with all my questions.</span></p>

BigQuery to the rescue: How to prototype an ML system for a medium-sized wholesale company in the Google Cloud

Developer

Senior IT consultant

Profilbild von Sales Manager Ioannis Pechlivanis aus Stuttgart

Key Account Manager

Senior Agile Coach & Consultant

<div><h2>1. Introduction</h2><div><div><p>Welcome Micronaut: a modern JVM-based full-stack framework. If you have not yet heard of Micronaut, then it is time to get informed and try it out. In this blog post you will get acquainted with Micronaut’s key features, scaffolding tools and a simple HTTP server and client example.</p></div><div><p>Here is a list of Micronaut’s key features:</p></div><div><p><strong>Supports Java, Groovy, and Kotlin.</strong> Being a JVM-based framework, Micronaut provides first-class support for Java, Groovy and Kotlin.</p></div><div><p><strong>Natively cloud-native.</strong> Micronaut’s cloud support is built right in, including support for common discovery services, distributed tracing tools, and cloud runtimes. Micronaut is ready to develop serverless applications and is designed for building resilient microservices.</p></div><div><p><strong>Fast startup time and low memory consumption.</strong> Micronaut avoids using reflection at runtime and uses compile-time AST transformations instead. This is why Micronaut has a fast startup time and a minimal memory footprint.</p></div><div><p><strong>Reactive and non-blocking.</strong> Micronaut is a non-blocking HTTP server built on Netty and has a declarative, reactive and compile-time HTTP client.</p></div><div><p><strong>Fast and easy testing.</strong> Efficient compile-time dependency injection and AOP.</p></div><div><p><strong>Fast data-access configuration.</strong> Micronaut provides sensible defaults that automatically configure your favourite data access toolkit and APIs to make it easy to write your own integrations.</p></div></div></div><div><h2>2. Prerequisites</h2><div><div><p>In order to run the examples in this article, you need to install the following tools:</p></div><div><ul><li><p><a href="http://sdkman.io/" target="_blank">SDKMAN</a>  – a tool for managing parallel versions of multiple SDKs</p><div><div>

```
curl -s "https://get.sdkman.io" | bash
```

</div></div></li><li><p>Java Development Kit</p><div><div>

```
sdk install java
```

</div></div></li><li><p>Micronaut CLI</p><div><div>

```
sdk install micronaut
```

</div></div></li></ul></div></div></div><div><h2>3. Scaffolding</h2><div><div><p>As many other modern frameworks, Micronaut comes with a handy CLI scaffolding tool. Micronaut CLI can be used as a CLI tool:</p></div><div><div><div>

```bash
mn help
mn help create-app
```

</div></div></div><div><p>It can also be used in interactive mode with command completion.</p></div><div><div><div>

```bash
$ mn
| Starting interactive mode...
| Enter a command name to run. Use TAB for completion:
mn> help
...
mn> help create-app
...
```

</div></div></div><div><p>Try both modes and see what you like more.</p></div><div><h3>3.1. Creating a new Gradle project</h3><div><p>Micronaut supports Java, Groovy, and Kotlin as first-class citizens. Let us create a new Java application using Micronaut CLI:</p></div><div><div><div>

```bash
mn create-app mn-hello-java
```

</div></div></div><div><p>This will scaffold a new Gradle project. If you prefer Maven, add a <code>--build maven</code> parameter. If you want to create a new Groovy or Kotlin project, add a <code>--lang</code> parameter:</p></div><div><div><div>

```bash
mn create-app --lang groovy mn-hello-groovy
mn create-app --lang kotlin mn-hello-kotlin
```

</div></div></div><div><p>By default Micronaut HTTP server will listen on a random port, but you can alter that by adding the following configuration to src/main/resources/application.yml:</p></div><div><div>src/main/resources/application.yml</div><div><div>

```yaml
micronaut:
    server:
        port: 8080
```

</div></div></div><div><p>The application can be immediately run with Gradle:</p></div><div><div><div>

```bash
cd mn-hello-java
./gradlew run
> Task :run
 
11:56:49.340 [main] INFO  io.micronaut.runtime.Micronaut - Startup completed in 654ms. Server Running: http://localhost:8080
<=========----> 75% EXECUTING [7s]
```

</div></div></div><div><p>Now we can check that Micronaut is running:</p></div><div><div><div>

```bash
curl http://localhost:8080
{"_links":{"self":{"href":"/","templated":false}},"message":"Page Not Found"}
```

</div></div></div><div><p>To run the application with IntelliJ IDEA, you need to enable annotation processing:</p></div><div><ol><li><p>open Settings / Build, Execution, Deployment / Compiler / Annotation Processors</p></li><li><p>Set the checkbox “Enable annotation processing”</p></li></ol></div><div><p>Then you can run the application by executing the application class in IDEA.</p></div></div><div><h3>3.2. Adding an HTTP controller</h3><div><p>Let us use the CLI again to create an HTTP controller. Run this command in the directory of the project that we created:</p></div><div><div><div>

```bash
mn create-controller HelloController
| Rendered template Controller.java to destination src/main/java/mn/hello/java/HelloController.java
| Rendered template ControllerTest.java to destination src/test/java/mn/hello/java/HelloControllerTest.java
```

</div></div></div><div><p>As we can see, Micronaut cares about Test-Driven Development and has created a test along with the controller:</p></div><div><div>HelloController.java</div><div><div>

```java
package mn.hello.java;
 
import io.micronaut.http.annotation.Controller;
import io.micronaut.http.annotation.Get;
import io.micronaut.http.HttpStatus;
 
@Controller("/hello")
public class HelloController {
 
    @Get("/")
    public HttpStatus index() {
        return HttpStatus.OK;
    }
}
```

</div></div></div><div><div>HelloControllerTest.java</div><div><div>

```java
package mn.hello.java;
 
import io.micronaut.context.ApplicationContext;
import io.micronaut.http.HttpStatus;
import io.micronaut.http.client.RxHttpClient;
import io.micronaut.runtime.server.EmbeddedServer;
import org.junit.Test;
 
 
import static org.junit.Assert.assertEquals;
 
public class HelloControllerTest {
 
    @Test
    public void testIndex() throws Exception {
        EmbeddedServer server = ApplicationContext.run(EmbeddedServer.class);
 
        RxHttpClient client = server.getApplicationContext().createBean(RxHttpClient.class, server.getURL());
 
        assertEquals(HttpStatus.OK, client.toBlocking().exchange("/hello").status());
        server.stop();
    }
}
```

</div></div></div></div><div><h3>3.3. Adding a compile-time-generated HTTP client</h3><div><p>Micronaut provides a great declarative compile-time HTTP client. You can use it for testing your own server (as we will do in the next example) or for communicating with external servers.</p></div><div><div><div>

```bash
mn create-client HelloClient
| Rendered template Client.java to destination src/main/java/mn/hello/java/HelloClient.java
```

</div></div></div><div><p>Here is what Micronaut creates:</p></div><div><div>HelloClient.java</div><div><div>

```java
package mn.hello.java;
 
import io.micronaut.http.client.Client;
import io.micronaut.http.annotation.Get;
import io.micronaut.http.HttpStatus;
 
@Client("hello")
public interface HelloClient {
 
    @Get("/")
    HttpStatus index();
}
```

</div></div></div><div><p>That is all of it – only the interface. Micronaut will generate the implementation at compile time.</p></div></div></div></div><div><h2>4. Hello World</h2><div><div><p>Now that we got acquainted with Micronaut’s scaffolding CLI, let us create a more comprehensive example. We will extract the server API and reuse it in the HTTP controller in the production code and in the HTTP client in the test:</p></div>

![hello-micronaut-class-diagram.png](https://media.graphassets.com/D7ZoAmo7SiexkyIN9B0C)

<p></p>Figure 2. Sharing server and client API</div></div><div><h3>4.1. Server API</h3><div><p>First, let us define the server API:</p></div><div><div>HelloApi.java</div><div><div>

```java
package mn.hello.java;
 
import io.micronaut.http.annotation.Get;
import io.micronaut.http.annotation.QueryValue;
 
public interface HelloApi {
 
    @Get("/")
    HelloMessage index(@QueryValue("name") String name);
}
```

</div></div></div><div><p>Here is what is happening:</p></div><div><ul><li><p>the annotation <code>@Get("/")</code> specifies the path and the HTTP method of the endpoint</p></li><li><p>the annotation <code>@QueryValue("name")</code> maps the GET parameter <code>name</code> to the method parameter <code>name</code></p></li><li><p>the return type <code>HelloMessage</code> tells Micronaut to serialize the POJO returned by the HTTP controller to JSON</p></li></ul></div><div><div>HelloMessage.java</div><div><div>

```java
package mn.hello.java;
 
public class HelloMessage {
    public String greeting;
}
```

</div></div></div></div><div><h3>4.2. HTTP controller</h3><div><p>Let us implement a simple HTTP controller that will return a greeting for the given name:</p></div><div><div>HelloController.java</div><div><div>

```java
package mn.hello.java;
 
import io.micronaut.http.annotation.Controller;
 
@Controller("/hello")
public class HelloController implements HelloApi {
 
    @Override
    public HelloMessage index(String name) {
        HelloMessage m = new HelloMessage();
        m.greeting = "Hello " + name + "!";
        return m;
    }
}
```

</div></div></div><div><p>Since we return a POJO in the controller, Micronaut considers the method blocking and will execute it on the I/O thread pool. However, it is also possible to write a non-blocking reactive implementation and return a non-blocking type such as <a href="http://reactivex.io/RxJava/javadoc/io/reactivex/Single.html"><code>Single</code></a> . In this case, the request is considered non-blocking and the method will be executed on the Netty event loop thread.</p></div></div><div><h3>4.3. HTTP client & testing</h3><div><p>Since we extracted the server API into a separate interface, we can now easily create an HTTP client for our application:</p></div><div><div>HelloClient.java</div><div><div>

```java
package mn.hello.java;
 
import io.micronaut.http.client.Client;
 
@Client("/hello")
public interface HelloClient extends HelloApi {}
```

</div></div></div><div><p>We don’t need to write any implementation, Micronaut will do that for us at compile time. Here is what a test can look like using this HTTP client:</p></div><div><div>HelloControllerTest.java</div><div><div>

```java
package mn.hello.java;
 
import io.micronaut.context.ApplicationContext;
import io.micronaut.runtime.server.EmbeddedServer;
import org.junit.Test;
 
import static org.junit.Assert.assertEquals;
 
public class HelloControllerTest {
 
    @Test
    public void testIndex() throws Exception {
        EmbeddedServer server = ApplicationContext.run(EmbeddedServer.class);
 
        HelloClient client = server.getApplicationContext().getBean(HelloClient.class);
 
        HelloMessage serverResponse = client.index("codecentric");
 
        assertEquals("Hello codecentric!", serverResponse.greeting);
        server.stop();
    }
}
```

</div></div></div><div><p>In this test we actually spin up our Micronaut application and execute an HTTP request against it using the generated HTTP client.</p></div></div><div><h3>4.4. Docker-ready</h3><div><p>Micronaut CLI generates a Dockerfile as well, making it easy to package your application for a container environment such as Kubernetes. Let us run our example application with Docker:</p></div><div><ol><li><p>Build the application into a fat-jar:</p><div><div>

```
./gradlew build
```

</div></div></li><li><p>Build a Docker image:</p><div><div>

```
docker build . -t mn-hello-world
```

</div></div></li><li><p>Run the Docker image:</p><div><div>

```
docker run --rm -p 8080:8080 mn-hello-world
```

</div></div></li><li><p>Check that it is running:</p><div><div>

```
curl http://localhost:8080/hello?name=codecentric
{"greeting":"Hello codecentric!"}
```

</div></div></li></ol></div></div></div></div><div><h2>5. Conclusion</h2><div><div><p>In this article we have only scratched the surface of what Micronaut has to offer. Here is what’s left out of scope:</p></div><div><ul><li><p>Reactive programming</p></li><li><p>Database access with GORM</p></li><li><p>Service discovery</p></li><li><p>Serverless applications</p></li><li><p>Distributed tracing</p></li></ul></div><div><p>All in all, Micronaut is a fast-evolving framework and looks very promising. Go try it yourself or maybe even use it for your next project!</p></div></div></div><div><h2>6. Links</h2><div><div><ul><li><p>Micronaut home page: <a href="http://micronaut.io/">http://micronaut.io/</a> </p></li></ul></div></div></div>

Micronaut Microservices Framework: Introduction

IT Consultant & Developer

Werkstudent Developer

Senior IT Consultant & Developer

Standortleiter Stuttgart

Atlassian Solution Consultant

Stuttgart | Team

Vielen Dank für deine Anfrage. Ich melde mich bei dir.

Vielen Dank für deine Anmeldung. Eine E-Mail zur Bestätigung ist zu dir unterwegs.

Diese Seite ist eine Vorschau. \nKlicken Sie hier\n, um den Vorschaumodus zu verlassen.

Zu deiner Suche haben wir leider keine passenden Blogartikel gefunden. 

Du hast noch Fragen zu diesem Thema? Dann sprich mich einfach an.

Die \nDatenschutzerklärung\n der codecentric AG habe ich zur Kenntnis genommen und bestätige das mit der Absendung des Formulars.*

Entdecke spannende weiterführende Themen und lass dich von der codecentric Welt inspirieren.

Ich möchte eure Inhalte abonnieren. Diese Einwilligung kann ich jederzeit widerrufen bzw. den Blog Newsletter abbestellen. Die Datenschutzerklärung habe ich gelesen. Ich bin damit einverstanden, dass die codecentric AG, Hochstr. 11, 42697 Solingen, mich über eigene Dienstleistungen, branchenrelevante Events, Webinare und Publikationen der codecentric AG über die von mir angegebene E-Mail-Adresse informiert. Meine Einwilligung ist freiwillig. Ich kann meine Einwilligung jederzeit mit Wirkung für die Zukunft widerrufen (z.B. über den hierfür vorgesehenen Abmeldelink in unserem Newsletter, per E-Mail an datenschutz@codecentric.de oder per Post an die oben genannte Adresse der codecentric AG) erklären. Weitere Hinweise zur Verarbeitung meiner personenbezogenen Daten und zu meinen Rechten als betroffene Person finde ich hier: \nhttps://www.codecentric.de/datenschutzerklaerung\n

Bitte wähle deine/n gewünschte/n Standort/e aus.

Wir möchten dich als Mensch kennenlernen. Erzähle uns, wer du bist, worauf du Wert legst, wie du gerne arbeitest und was du von uns als Arbeitgeber erwartest.

Bitte füge deine Bewerbungsunterlagen ​(mind. Lebenslauf, gerne auch mehr)​​ ​als eine ​PDF-​Datei ​hinzu.

Ich habe die \nDatenschutzbestimmungen\n und die Einwilligung in die Datenverarbeitung gelesen und stimme diesen zu.*

Job application form field placeholder 10

Keine passende Rolle gefunden? Bewirb dich bitte trotzdem initiativ! Jenseits von starren Schubladen zählen für uns vielmehr Motivation, Kompetenz und der Cultural Fit.

Job application form field placeholder 11

Dies ist ein Pflichtfeld, bitte fülle es aus.

Kontakt

Absenden

Willkommen im Team Stuttgart

Gemeinsam bessere Projekte umsetzen.

Wir helfen deinem Unternehmen.

Unsere Leistungen

Hilf uns, noch besser zu werden.

Zu den Jobangeboten