Jekyll2020-05-22T06:13:58+00:00http://almcc.me/feed.xmlalmcc.mePersonal blog to record my tinkerings with new technologies.
Getting into Serverless with CDK, Lambda & DynamoDB - Part 22020-05-22T00:00:00+00:002020-05-22T00:00:00+00:00http://almcc.me/blog/2020/05/22/getting-into-serverless-with-cdk-lambda-dynamodb-part-2<p>In this second post, we are going to add a DynamoDB Table to our stack for our lambda function to pull data from. See <a href="https://almcc.me/blog/2020/04/21/getting-into-serverless-with-cdk-lambda-dynamodb-part-1/">Part 1</a> on getting started to get caught up.</p>
<p>Before we go too far, you are going to need:</p>
<ul>
<li>Python 3.8 with Pip installed</li>
<li><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html">AWS SAM CLI</a></li>
</ul>
<h2 id="step-1---defining-the-table">Step 1 - Defining the table.</h2>
<p>Install the DynamoDB CDK module, being careful to use the same release you used before.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>npm <span class="nb">install</span> @aws-cdk/aws-dynamodb@1.33.1
</code></pre></div></div>
<p>And add it to our imports in <code class="language-plaintext highlighter-rouge">lib/myapp-stack.ts</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import * as dynamodb from "@aws-cdk/aws-dynamodb";
</code></pre></div></div>
<p>Next, we can add a DynamoDB <a href="https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-dynamodb.Table.html">Table</a> construct to our stack. DynamoDB is a no-sql database and requires you to think a little bit about how your data will be accessed. Read about the key concepts over <a href="https://www.dynamodbguide.com/key-concepts/">here</a> on <a href="https://www.dynamodbguide.com/">dynamodbguide.com</a></p>
<div class="language-tsx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">tourfinishersTable</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">dynamodb</span><span class="p">.</span><span class="nx">Table</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="dl">"</span><span class="s2">TourFinishersTable</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span>
<span class="na">partitionKey</span><span class="p">:</span> <span class="p">{</span> <span class="na">name</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Year</span><span class="dl">"</span><span class="p">,</span> <span class="na">type</span><span class="p">:</span> <span class="nx">dynamodb</span><span class="p">.</span><span class="nx">AttributeType</span><span class="p">.</span><span class="nx">NUMBER</span> <span class="p">},</span>
<span class="na">sortKey</span><span class="p">:</span> <span class="p">{</span> <span class="na">name</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Position</span><span class="dl">"</span><span class="p">,</span> <span class="na">type</span><span class="p">:</span> <span class="nx">dynamodb</span><span class="p">.</span><span class="nx">AttributeType</span><span class="p">.</span><span class="nx">NUMBER</span> <span class="p">},</span>
<span class="na">tableName</span><span class="p">:</span> <span class="dl">"</span><span class="s2">TourFinishers</span><span class="dl">"</span><span class="p">,</span>
<span class="na">billingMode</span><span class="p">:</span> <span class="nx">dynamodb</span><span class="p">.</span><span class="nx">BillingMode</span><span class="p">.</span><span class="nx">PAY_PER_REQUEST</span><span class="p">,</span>
<span class="na">removalPolicy</span><span class="p">:</span> <span class="nx">cdk</span><span class="p">.</span><span class="nx">RemovalPolicy</span><span class="p">.</span><span class="nx">DESTROY</span><span class="p">,</span>
<span class="p">});</span>
</code></pre></div></div>
<p>We have chosen in this instance to make the <code class="language-plaintext highlighter-rouge">Year</code> the primary key and <code class="language-plaintext highlighter-rouge">Position</code> the sort key, together this will be a unique value within this simple context a high cardinality. It will support our access pattern later as will.</p>
<p>Next step is to grant our function read access to the table.</p>
<div class="language-tsx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">tourfinishersTable</span><span class="p">.</span><span class="nx">grantReadData</span><span class="p">(</span><span class="nx">myFunction</span><span class="p">);</span>
</code></pre></div></div>
<p>I think this is one of the most powerful features of CDK, it makes dealing with permissions so easy that you don’t feel the need to grant overly permissive policies.</p>
<h2 id="step-2---updating-out-function">Step 2 - Updating out function</h2>
<p>Our python function is going to be a little more complex than before and will require some dependencies. Therefore we first need to move it into its own <code class="language-plaintext highlighter-rouge">src</code> directory.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir </span><span class="k">function</span>/src
<span class="nb">mv </span><span class="k">function</span>/index.py <span class="k">function</span>/src/
</code></pre></div></div>
<p>Next, we will want to add a requreiments.txt file at <code class="language-plaintext highlighter-rouge">function/src/requirements.txt</code> and add the following dependencies.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>boto3><span class="o">=</span>1.12.43
</code></pre></div></div>
<p>Now we will write a <code class="language-plaintext highlighter-rouge">build.sh</code> script at the base of our project to build are code bundle. AWS Lambda’s require a code bundle to come pre-packaged with all its dependencies.</p>
<div class="language-makefile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash
</span>
<span class="err">mkdir</span> <span class="err">-p</span> <span class="err">function/_bundle</span>
<span class="err">pip3.8</span> <span class="err">install</span> <span class="err">-r</span> <span class="err">function/src/requirements.txt</span> <span class="err">--target</span> <span class="err">function/_bundle</span>
<span class="err">cp</span> <span class="err">-r</span> <span class="err">function/src/*</span> <span class="err">function/_bundle</span>
<span class="err">pushd</span> <span class="err">function/_bundle</span>
<span class="err">zip</span> <span class="err">-r</span> <span class="err">--quiet</span> <span class="err">../bundle.zip</span> <span class="err">.</span>
<span class="err">popd</span>
<span class="err">rm</span> <span class="err">-rf</span> <span class="err">function/_bundle</span>
</code></pre></div></div>
<p>And now we can update our code <code class="language-plaintext highlighter-rouge">function/src/index.py</code> to pull the data from dynamoDB.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import boto3
from boto3.dynamodb.conditions import Key
logger <span class="o">=</span> logging.getLogger<span class="o">()</span>
logger.setLevel<span class="o">(</span>logging.DEBUG<span class="o">)</span>
dynamodb <span class="o">=</span> boto3.resource<span class="o">(</span><span class="s1">'dynamodb'</span><span class="o">)</span>
table <span class="o">=</span> dynamodb.Table<span class="o">(</span><span class="s1">'TourFinishers'</span><span class="o">)</span>
def main<span class="o">(</span>event, context<span class="o">)</span>:
logger.info<span class="o">(</span><span class="s2">"Start of function"</span><span class="o">)</span>
response <span class="o">=</span> table.query<span class="o">(</span>
<span class="nv">KeyConditionExpression</span><span class="o">=</span>Key<span class="o">(</span><span class="s1">'Year'</span><span class="o">)</span>.eq<span class="o">(</span>2019<span class="o">)</span>
<span class="o">)</span>
<span class="k">return</span> <span class="o">{</span>
<span class="s2">"statusCode"</span>: 200,
<span class="s2">"body"</span>: response[<span class="s2">"Items"</span><span class="o">]</span>,
<span class="s2">"headers"</span>: <span class="o">{</span>
<span class="s2">"Content-Type"</span>: <span class="s2">"application/json"</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Next, we run <code class="language-plaintext highlighter-rouge">./build.sh</code> to build are bundle and then we can point CDK at our code bundle by updating the <code class="language-plaintext highlighter-rouge">code</code> property of our lambda in <code class="language-plaintext highlighter-rouge">lib/myapp-stack.ts</code>.</p>
<div class="language-tsx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">code</span><span class="p">:</span> <span class="nx">lambda</span><span class="p">.</span><span class="nx">Code</span><span class="p">.</span><span class="nx">fromAsset</span><span class="p">(</span>
<span class="nx">path</span><span class="p">.</span><span class="nx">join</span><span class="p">(</span><span class="nx">__dirname</span><span class="p">,</span> <span class="dl">"</span><span class="s2">../function/bundle.zip</span><span class="dl">"</span><span class="p">)</span>
<span class="p">),</span>
</code></pre></div></div>
<h2 id="step-3---deploy-the-changes">Step 3 - Deploy the changes</h2>
<p>Simply build and deploy are changes to AWS.</p>
<div class="language-tsx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">npm</span> <span class="nx">run</span> <span class="nx">build</span>
<span class="nx">cdk</span> <span class="nx">deploy</span>
</code></pre></div></div>
<p>To speed things along for this demo, I have a gist that has the top 3 winners of the last 3 Tours that we can import into our table in AWS directly.</p>
<div class="language-tsx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">aws</span> <span class="nx">dynamodb</span> <span class="nx">batch</span><span class="o">-</span><span class="nx">write</span><span class="o">-</span><span class="nx">item</span> <span class="o">--</span><span class="nx">request</span><span class="o">-</span><span class="nx">items</span> <span class="nx">https</span><span class="p">:</span><span class="c1">//gist.githubusercontent.com/almcc/1437d07e5f344bb8a5b03376a9473af2/raw/a96c545774c5a046b8e7b32e597ded0f7a84306c/TourFinishers.json</span>
</code></pre></div></div>
<h2 id="step-4---execute-to-lambda">Step 4 - Execute to lambda</h2>
<p>Now to run your function in AWS we can use the AWS command.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>aws lambda invoke <span class="nt">--function-name</span> MyappStack-MyFunction3BAA72D1-1H78B5LRDK3NI <span class="o">>(</span><span class="nb">cat</span> | jq<span class="o">)</span> <span class="nt">--log-type</span> Tail | jq <span class="nt">--raw-output</span> .LogResult | <span class="nb">base64</span> <span class="nt">-d</span>
</code></pre></div></div>
<p>You should see something a bit like this as the output.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"statusCode"</span><span class="p">:</span><span class="w"> </span><span class="mi">200</span><span class="p">,</span><span class="w">
</span><span class="nl">"body"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"RiderNumber"</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w">
</span><span class="nl">"Position"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w">
</span><span class="nl">"FirstName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Egan"</span><span class="p">,</span><span class="w">
</span><span class="nl">"LastName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Bernal"</span><span class="p">,</span><span class="w">
</span><span class="nl">"TeamName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Team INEOS"</span><span class="p">,</span><span class="w">
</span><span class="nl">"Year"</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"RiderNumber"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w">
</span><span class="nl">"Position"</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w">
</span><span class="nl">"FirstName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Geraint"</span><span class="p">,</span><span class="w">
</span><span class="nl">"LastName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Thomas"</span><span class="p">,</span><span class="w">
</span><span class="nl">"TeamName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Team INEOS"</span><span class="p">,</span><span class="w">
</span><span class="nl">"Year"</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"RiderNumber"</span><span class="p">:</span><span class="w"> </span><span class="mi">81</span><span class="p">,</span><span class="w">
</span><span class="nl">"Position"</span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="p">,</span><span class="w">
</span><span class="nl">"FirstName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Steven"</span><span class="p">,</span><span class="w">
</span><span class="nl">"LastName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Kruijswijk"</span><span class="p">,</span><span class="w">
</span><span class="nl">"TeamName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Team Jumbo-Visma"</span><span class="p">,</span><span class="w">
</span><span class="nl">"Year"</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"headers"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"Content-Type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/json"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="err">...</span><span class="w">
</span></code></pre></div></div>
<h2 id="bonus-step---running-the-function-locally">Bonus Step - Running the function locally</h2>
<p>You can also run the function locally with the AWS sam CLI which can be handy during development.</p>
<p>The first thing to do is to get hold of the CloudFormation template that CDK is creating behind the scenes for us.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cdk synth <span class="nt">--no-staging</span> <span class="o">></span> template.yaml
</code></pre></div></div>
<p>Now you can use the <code class="language-plaintext highlighter-rouge">sam</code> CLI to invoke your function. (Check the contents of <code class="language-plaintext highlighter-rouge">template.yaml</code> for the functions name)</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sam <span class="nb">local </span>invoke MyFunction3BAA72D1 <span class="nt">--no-event</span> | jq
</code></pre></div></div>
<p>This will run your code locally, using your local credentials to query the live table in AWS.</p>
<p>Then if you make any changes to you code you will need to run <code class="language-plaintext highlighter-rouge">./build.sh</code> again, if you make a change to your template, you will need to do the following.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>npm run build
cdk synth <span class="nt">--no-staging</span> <span class="o">></span> template.yaml
</code></pre></div></div>
<h2 id="job-done">Job done!</h2>
<p>That’s it for this post, Next time we will look at adding the API gateway.</p>
<p>If you would like to clean up everything you have created in AWS, see Step 6 - <em>Cleaning up</em> from the <a href="https://almcc.me/blog/2020/04/21/getting-into-serverless-with-cdk-lambda-dynamodb-part-1/">first post in this series</a>.</p>In this second post, we are going to add a DynamoDB Table to our stack for our lambda function to pull data from.Getting into Serverless with CDK, Lambda & DynamoDB - Part 12020-04-21T00:00:00+00:002020-04-21T00:00:00+00:00http://almcc.me/blog/2020/04/21/getting-into-serverless-with-cdk-lambda-dynamodb-part-1<p>AWS Cloud Development Kit (CDK) is a great tool for using code to deploy resources to AWS. In a series of posts, we will cover a few of the key concepts you need to be aware of getting started with AWS and CDK. We will create a Lambda Function that returns some data from a DynamoDB table over an API Gateway.</p>
<p>CDK allows you to define <a href="https://docs.aws.amazon.com/cdk/latest/guide/constructs.html">Constructs</a> in code that map to AWS components or resources, you can then collect those Constructs together into a <a href="https://docs.aws.amazon.com/cdk/latest/guide/stacks.html">Stack</a> which is what we deploy to an AWS region. Finally, a Stack is defined within an <a href="https://docs.aws.amazon.com/cdk/latest/guide/apps.html">App</a> which is the root of our project, it can contain many instances of the same stack for say multiple regions or multiple different stack instances.</p>
<p>You can choose to develop CDK projects with TypeScipt, JavaScript, Python Java or C#. While initially learning CDK I went for Python as it is usually my language of choice, I have later found TypeScript to be much more comfortable as there is a wealth of examples already out there. We will use TypeScript for this post.</p>
<p><strong>Note:</strong> Many tutorials and guides on learning AWS will suggest using the web console to setup and configure resources, however, beyond simple examples I have felt uncomfortable building so much in an unrepeatable fashion. If you are someone who learns by doing, then a pattern I have found works really well is building a setup partially with CDK, then using the web console to learn and experiment further before refactoring those changes back into the CDK stack itself. Simply rinse and repeat to develop ever bigger or more bespoke setups. Having AWS best practice built into the CDK constructs is a comforting safety net.</p>
<p>So that is already too many words, let’s get started.</p>
<h2 id="prerequisites">Prerequisites</h2>
<ul>
<li>You have the <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html">AWS CLI</a> installed and configured, this post was created using V1 of the CLI however both V1 and V2 should work. The <code class="language-plaintext highlighter-rouge">aws configure list</code> command is a good way to test if you have your credentials set up correctly. For those of you who use profiles with the <code class="language-plaintext highlighter-rouge">--profile</code> flag, the <code class="language-plaintext highlighter-rouge">cdk</code> command works in the same way,</li>
</ul>
<h2 id="step-1-install-the-cdk-cli">Step 1, Install the CDK CLI.</h2>
<p>Assuming you have <a href="https://nodejs.org/en/">node</a> already installed, you just now need TypeScript and the CDK CLI</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>npm <span class="nb">install</span> <span class="nt">-g</span> typescript
npm <span class="nb">install</span> <span class="nt">-g</span> aws-cdk
</code></pre></div></div>
<h2 id="step-2-create-our-project">Step 2, Create our project.</h2>
<p>Let’s make a directory and initialise a typescript app project.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir </span>myapp
<span class="nb">cd </span>myapp/
cdk init app <span class="nt">--language</span> typescript
</code></pre></div></div>
<p>And a quick test that it builds ok.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>npm run build
</code></pre></div></div>
<h2 id="step-3-the-lambda-function">Step 3, The Lambda Function</h2>
<p>Install the lambda package.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>npm <span class="nb">install</span> @aws-cdk/aws-lambda
</code></pre></div></div>
<p>Add the dependency to our stack (<code class="language-plaintext highlighter-rouge">lib/myapp-stack.ts</code>).</p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="o">*</span> <span class="k">as</span> <span class="nx">cdk</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">@aws-cdk/core</span><span class="dl">"</span><span class="p">;</span>
<span class="k">import</span> <span class="o">*</span> <span class="k">as</span> <span class="nx">lambda</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">@aws-cdk/aws-lambda</span><span class="dl">"</span><span class="p">;</span>
<span class="k">export</span> <span class="kd">class</span> <span class="nx">MyappStack</span> <span class="kd">extends</span> <span class="nx">cdk</span><span class="p">.</span><span class="nx">Stack</span> <span class="p">{</span>
<span class="kd">constructor</span><span class="p">(</span><span class="nx">scope</span><span class="p">:</span> <span class="nx">cdk</span><span class="p">.</span><span class="nx">Construct</span><span class="p">,</span> <span class="nx">id</span><span class="p">:</span> <span class="nx">string</span><span class="p">,</span> <span class="nx">props</span><span class="p">?:</span> <span class="nx">cdk</span><span class="p">.</span><span class="nx">StackProps</span><span class="p">)</span> <span class="p">{</span>
<span class="k">super</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span> <span class="nx">id</span><span class="p">,</span> <span class="nx">props</span><span class="p">);</span>
<span class="c1">// The code that defines your stack goes here</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Next, we need to write some function lambda code. Create a file inside a newly created function directory (<code class="language-plaintext highlighter-rouge">function/index.py</code>). In this case, we are just going to return the top 3 finishers of the 2019 Tour de France, later we will work on returning their names from a DynamoDB table.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">logging</span>
<span class="n">logger</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">()</span>
<span class="n">logger</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">"Start of function"</span><span class="p">)</span>
<span class="k">return</span> <span class="p">{</span>
<span class="s">"statusCode"</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span>
<span class="s">"body"</span><span class="p">:</span> <span class="p">[</span><span class="s">"Egan Bernal"</span><span class="p">,</span> <span class="s">"Geraint Thomas"</span><span class="p">,</span> <span class="s">"Steven Kruijswijk"</span><span class="p">],</span>
<span class="s">"headers"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"Content-Type"</span><span class="p">:</span> <span class="s">"application/json"</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Notice that I have also set up a logger with a simple logging statement, this will be useful later to demonstrate how we retrieve those logs.</p>
<p>Now we need to define our lambda function in our stack, <a href="https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-lambda.Function.html">we can use the <code class="language-plaintext highlighter-rouge">Function</code> construct from the <code class="language-plaintext highlighter-rouge">aws-lambda</code> package.</a> Inside the <code class="language-plaintext highlighter-rouge">MyappStack</code> construct in <code class="language-plaintext highlighter-rouge">lib/myapp-stack.ts</code> add the following code.</p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">myFunction</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">lambda</span><span class="p">.</span><span class="nb">Function</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="dl">"</span><span class="s2">MyFunction</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span>
<span class="na">runtime</span><span class="p">:</span> <span class="nx">lambda</span><span class="p">.</span><span class="nx">Runtime</span><span class="p">.</span><span class="nx">PYTHON_3_8</span><span class="p">,</span>
<span class="na">handler</span><span class="p">:</span> <span class="dl">"</span><span class="s2">index.main</span><span class="dl">"</span><span class="p">,</span>
<span class="na">code</span><span class="p">:</span> <span class="nx">lambda</span><span class="p">.</span><span class="nx">Code</span><span class="p">.</span><span class="nx">fromAsset</span><span class="p">(</span><span class="nx">path</span><span class="p">.</span><span class="nx">join</span><span class="p">(</span><span class="nx">__dirname</span><span class="p">,</span> <span class="dl">"</span><span class="s2">../function</span><span class="dl">"</span><span class="p">)),</span>
<span class="p">});</span>
</code></pre></div></div>
<p>Don’t forget to add <code class="language-plaintext highlighter-rouge">import * as path from "path";</code> as well.</p>
<p>You can see that I have defined a function imaginatively named <code class="language-plaintext highlighter-rouge">MyFunction</code>, it uses the Python 3.8 runtime, it’s handler points to <code class="language-plaintext highlighter-rouge">main</code> function inside the <code class="language-plaintext highlighter-rouge">[index.py](http://index.py)</code> module and finally, the code lives in the <code class="language-plaintext highlighter-rouge">function</code> directory which is in the parent directory to our <code class="language-plaintext highlighter-rouge">lib</code> directory.</p>
<p>As it stands, this is enough for us to deploy our code. However, to invoke our function later we are going to need the function name that is created. CDK will take our <code class="language-plaintext highlighter-rouge">MyFunction</code> name and <a href="https://docs.aws.amazon.com/cdk/latest/guide/identifiers.html">add’s to it to make sure that it’s unique</a>. To extract that name of the function that will be created in AWS we can add a CloudFormation Output to our stack.</p>
<p>Again in <code class="language-plaintext highlighter-rouge">lib/myapp-stack.ts</code> and the following code under our lambda function declaration.</p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">new</span> <span class="nx">cdk</span><span class="p">.</span><span class="nx">CfnOutput</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="dl">"</span><span class="s2">FunctionNameOutput</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span>
<span class="na">value</span><span class="p">:</span> <span class="nx">myFunction</span><span class="p">.</span><span class="nx">functionName</span><span class="p">,</span>
<span class="na">exportName</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Lambda-Function-Name</span><span class="dl">"</span><span class="p">,</span>
<span class="p">});</span>
</code></pre></div></div>
<p>Note here that the <code class="language-plaintext highlighter-rouge">exportName</code> is the identifier for how we will reference the value and it <strong>must</strong> be unique across your entire AWS account, regardless of the stack. You can also see the reason we defined our function to a variable.</p>
<h2 id="step-4-build-and-deploy">Step 4, Build and Deploy</h2>
<p>Let’s quickly check that our project builds.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>npm run build
</code></pre></div></div>
<p>We can also check that CDK can successfully convert (or synthesise) are code into a CloudFormation template.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cdk synth
</code></pre></div></div>
<p>I tend to pipe the output of <code class="language-plaintext highlighter-rouge">cdk synth</code> to less as output is long and it’s useful to have a way to search it.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cdk synth | less
</code></pre></div></div>
<p>If all that worked then we are almost good to go. If this is the first time out have deployed a CDK project to your aws account you will need to <a href="https://docs.aws.amazon.com/cdk/latest/guide/tools.html">bootstrap you account</a>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cdk bootstrap
</code></pre></div></div>
<p>Now we are ready to deploy our stack.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cdk deploy
</code></pre></div></div>
<p>You will be asked to approve any IAM policies and statements that CDK is creating. This is also a good insight into how policies are stitched together in AWS as well as a good reminder of the heavy lifting that CDK is doing for you.</p>
<p>When making changes to your project and deploying again, it can be useful to chain the build command and the deploy command together to avoid deploying a stale stack without your changes and then scratching your head as to why you change didn’t work.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>npm run build <span class="o">&&</span> cdk deploy
</code></pre></div></div>
<p>As I have used <code class="language-plaintext highlighter-rouge">&&</code>, <code class="language-plaintext highlighter-rouge">cdk deploy</code> will only run if the previous command (the build command) succeeds.</p>
<h2 id="step-5-invoking-our-function">Step 5, Invoking our Function</h2>
<p>In the output of the deploy command, you should see our CloudFormation Output. It will look a little like the following:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Outputs:
MyappStack.FunctionNameOutput = MyappStack-MyFunction3BAA72D1-173G8DYDM9QA7
</code></pre></div></div>
<p>This is a function that we are now ready to invoke. We can do this with the <code class="language-plaintext highlighter-rouge">aws lambda invoke</code> command.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>aws lambda invoke <span class="nt">--function-name</span> MyappStack-MyFunction3BAA72D1-173G8DYDM9QA7 out.txt
</code></pre></div></div>
<p>This will invoke our function and push the output to <code class="language-plaintext highlighter-rouge">out.txt</code> which should look like this.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{"statusCode": 200, "body": ["Egan Bernal", "Geraint Thomas", "Steven Kruijswijk"], "headers": {"Content-Type": "application/json"}}
</code></pre></div></div>
<p>If you also want to see the logging output you can use the <code class="language-plaintext highlighter-rouge">--log-type Tail</code> option and it will include the LogResult in the function invocation response that was printed to the terminal.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>aws lambda invoke <span class="nt">--function-name</span> MyappStack-MyFunction3BAA72D1-173G8DYDM9QA7 out.txt <span class="nt">--log-type</span> Tail
</code></pre></div></div>
<p>Unfortunately, this will be in base64. You can paste it into your favourite base64 decoder to view the logs. For example, you could use the <code class="language-plaintext highlighter-rouge">base64</code> command.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <insert-base64-content-here> | <span class="nb">base64</span> <span class="nt">-d</span>
</code></pre></div></div>
<p>You should get some output that looks a little like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>START RequestId: 1fc4012a-3963-449b-8085-347e0a12e961 Version: $LATEST
[INFO] 2020-04-21T05:47:14.519Z 1fc4012a-3963-449b-8085-347e0a12e961 Start of function
END RequestId: 1fc4012a-3963-449b-8085-347e0a12e961
REPORT RequestId: 1fc4012a-3963-449b-8085-347e0a12e961 Duration: 1.15 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 49 MB
</code></pre></div></div>
<p><strong>Warning:</strong> Below I get a little sidetracked with command line tricks, if you are not a fan of the one-liner bash commands please feel free to skip ahead, otherwise make sure you have the <code class="language-plaintext highlighter-rouge">jq</code> command installed and read on!</p>
<p>We could also use some <code class="language-plaintext highlighter-rouge">jq</code> trickery to do that on the command line with the <code class="language-plaintext highlighter-rouge">base64</code> command.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>aws lambda invoke <span class="nt">--function-name</span> MyappStack-MyFunction3BAA72D1-173G8DYDM9QA7 out.txt <span class="nt">--log-type</span> Tail | jq <span class="nt">--raw-output</span> .LogResult | <span class="nb">base64</span> <span class="nt">-d</span>
</code></pre></div></div>
<p>While we are at it, it can be cumbersome to always be having to peek inside the <code class="language-plaintext highlighter-rouge">out.txt</code> file to see the output. We can avoid creating that file all together with the use of a process substitution that will work in most shells. We can also use <code class="language-plaintext highlighter-rouge">jq</code> again to clean up the output. We simply replace <code class="language-plaintext highlighter-rouge">out.txt</code> with <code class="language-plaintext highlighter-rouge">>(cat | jq)</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>aws lambda invoke <span class="nt">--function-name</span> MyappStack-MyFunction3BAA72D1-173G8DYDM9QA7 <span class="o">>(</span><span class="nb">cat</span> | jq<span class="o">)</span> <span class="nt">--log-type</span> Tail | jq <span class="nt">--raw-output</span> .LogResult | <span class="nb">base64</span> <span class="nt">-d</span>
</code></pre></div></div>
<p>You should now see the output of your lambda function printed cleanly to the terminal followed by any logging.</p>
<p>Finally, while we are down this rabbit hole, could we pull out the function name from CloudFormation programmatically so that you can just copy and paste the command? Well, yes we could.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>aws lambda invoke <span class="nt">--function-name</span> <span class="si">$(</span>aws cloudformation list-exports <span class="nt">--query</span> <span class="s1">'Exports[?Name==`Lambda-Function-Name`].Value'</span> <span class="nt">--output</span> text<span class="si">)</span> <span class="o">>(</span><span class="nb">cat</span> | jq<span class="o">)</span> <span class="nt">--log-type</span> Tail | jq <span class="nt">--raw-output</span> .LogResult | <span class="nb">base64</span> <span class="nt">-d</span>
</code></pre></div></div>
<h2 id="step-6-cleaning-up">Step 6, Cleaning up</h2>
<p>We have covered quite a bit there already and your at the point where you can experiment further on your own. In the next post, we will look at building a DynamoDB table and querying our data.</p>
<p>You could choose to leave your stack in place, AWS gives you 1 Million free lambda invocations per month and the S3 Bucket created by CDK will be free if your account is less than 12 months old.</p>
<p>However, if you would like to destroy your stack you can do the following.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cdk destroy
</code></pre></div></div>
<p>And if you would like to remove the CDKToolkit stack you will need to use the <code class="language-plaintext highlighter-rouge">aws</code> command to get the name of the cdk bucket (or use the web console)</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>aws s3 <span class="nb">ls</span>
</code></pre></div></div>
<p>You can then empty the bucket and delete the stack.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>aws s3 <span class="nb">rm </span>s3://cdktoolkit-stagingbucket-19rslbj9gvcos <span class="nt">--recursive</span>
aws cloudformation delete-stack <span class="nt">--stack-name</span> CDKToolkit
</code></pre></div></div>AWS Cloud Development Kit (CDK) is a great tool for using code to deploy resources to AWS.Parsing json with Python2018-11-13T00:00:00+00:002018-11-13T00:00:00+00:00http://almcc.me/blog/2018/11/13/parsing-json-with-python<p>I’m often find myself parsing deep nested json documents in python,
and often thought, this should be easier. For a structure such as:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"alpha"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"beta"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"gamma"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>I would like to be able to do something like this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">json</span>
<span class="o">>>></span> <span class="n">doc</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="s">'{"alpha": {"beta": {"gamma": 1}}}'</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">doc</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">'alpha.beta.gamma'</span><span class="p">)</span>
<span class="mi">1</span>
<span class="o">>>></span> <span class="n">doc</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">'alpha.beta.delta'</span><span class="p">)</span>
<span class="bp">None</span>
<span class="o">>>></span> <span class="n">doc</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">'one.two.three'</span><span class="p">)</span>
<span class="bp">None</span>
</code></pre></div></div>
<p>When searching for solutions, I would constantly focus on the ‘json’ bit,
it took me longer than I would like to admit, that once I call <code class="language-plaintext highlighter-rouge">json.loads</code>
I now have a python dictionary, it’s no longer json. A quick google after
that and I found <a href="https://github.com/akesterson/dpath-python">dpath-python</a>:</p>
<blockquote>
<p>A python library for accessing and searching dictionaries via /slashed/paths ala xpath</p>
</blockquote>
<p>There is a few minor differences to my desired example but nothing
that I can’t overcome. Dpath works like so:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="o">>>></span> <span class="kn">import</span> <span class="nn">json</span>
<span class="o">>>></span> <span class="kn">from</span> <span class="nn">dpath.util</span> <span class="kn">import</span> <span class="n">get</span> <span class="k">as</span> <span class="n">dget</span>
<span class="o">>>></span> <span class="n">jstring</span> <span class="o">=</span> <span class="s">'{"alpha": {"beta": {"gamma": 1}}}'</span>
<span class="o">>>></span> <span class="n">doc</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">jstring</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">dget</span><span class="p">(</span><span class="n">doc</span><span class="p">,</span> <span class="s">'/alpha/beta/gamma'</span><span class="p">)</span>
<span class="mi">1</span>
<span class="o">>>></span> <span class="n">dget</span><span class="p">(</span><span class="n">doc</span><span class="p">,</span> <span class="s">'/alpha/beta/delta'</span><span class="p">)</span>
<span class="o">...</span>
<span class="nb">KeyError</span><span class="p">:</span> <span class="s">'/alpha/beta/delta'</span>
</code></pre></div></div>
<p>The three main differences are that it uses a <code class="language-plaintext highlighter-rouge">/</code> as the default seperator,
requires a leading seperator and there is no way to provide a default in the
case of key not being found. I could cope with the first two, but the last once
is a deal breaker for me. So I might as well fix all three! I’d like to make it as
close to a standard python dictionary workflow as possible, so I am going to
create a <code class="language-plaintext highlighter-rouge">NestedDict</code> class, which will subclass <code class="language-plaintext highlighter-rouge">dict</code> and then overide <code class="language-plaintext highlighter-rouge">get</code>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">dpath.util</span>
<span class="k">class</span> <span class="nc">NestedDict</span><span class="p">(</span><span class="nb">dict</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">separator</span><span class="o">=</span><span class="s">'.'</span><span class="p">):</span>
<span class="k">if</span> <span class="n">path</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">!=</span> <span class="n">separator</span><span class="p">:</span>
<span class="n">path</span> <span class="o">=</span> <span class="n">separator</span> <span class="o">+</span> <span class="n">path</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">return</span> <span class="n">dpath</span><span class="o">.</span><span class="n">util</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">separator</span><span class="o">=</span><span class="n">separator</span><span class="p">)</span>
<span class="k">except</span> <span class="nb">KeyError</span><span class="p">:</span>
<span class="k">return</span> <span class="n">default</span>
</code></pre></div></div>
<p>As you can see, I have made the leading separator optional, defaulted it to <code class="language-plaintext highlighter-rouge">.</code>
and provided a way to set a default in the case of a KeyError. The first example
now becomes:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">json</span>
<span class="o">>>></span> <span class="n">doc</span> <span class="o">=</span> <span class="n">NestedDict</span><span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="s">'{"alpha": {"beta": {"gamma": 1}}}'</span><span class="p">))</span>
<span class="o">>>></span> <span class="n">doc</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">'alpha.beta.gamma'</span><span class="p">)</span>
<span class="mi">1</span>
<span class="o">>>></span> <span class="n">doc</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">'alpha.beta.delta'</span><span class="p">)</span>
<span class="bp">None</span>
<span class="o">>>></span> <span class="n">doc</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">'one.two.three'</span><span class="p">)</span>
<span class="bp">None</span>
</code></pre></div></div>
<p>Job done!</p>
<p>One thing I could do is add a <code class="language-plaintext highlighter-rouge">quiet=True|False</code> flag to control the
KeyError, as with the current setup it’s not possible to tell the difference between
the returned value form nested dictionary being <code class="language-plaintext highlighter-rouge">None</code> and the default reposnse
being <code class="language-plaintext highlighter-rouge">None</code>.</p>I'm often find myself parsing deep nested json documents in python, and often thought, this should be easier.Managing version with bumpversion2018-07-11T00:00:00+00:002018-07-11T00:00:00+00:00http://almcc.me/blog/2018/07/11/managing-version-with-bumpversion<p>For any software project it is likely that you’re will need to define the
version in sperate place and keeping track of that can be a bit of pain.
<a href="https://github.com/peritus/bumpversion">bumpversion</a> is a very handy tool to help you keep
track. You can install it like so:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip install --upgrade bumpversion
</code></pre></div></div>
<p>Taking a python project for example, for me the version will be defined
in <code class="language-plaintext highlighter-rouge">setup.py</code>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">setuptools</span> <span class="kn">import</span> <span class="n">setup</span>
<span class="n">setup</span><span class="p">(</span>
<span class="n">name</span><span class="o">=</span><span class="s">'example'</span><span class="p">,</span>
<span class="n">version</span><span class="o">=</span><span class="s">'0.0.1'</span><span class="p">,</span>
<span class="n">packages</span><span class="o">=</span><span class="p">[</span><span class="s">'example'</span><span class="p">],</span>
<span class="p">)</span>
</code></pre></div></div>
<p>and <code class="language-plaintext highlighter-rouge">example/__init__.py</code> (Package called <code class="language-plaintext highlighter-rouge">example</code> here.):</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">VERSION</span> <span class="o">=</span> <span class="s">'0.0.1'</span>
</code></pre></div></div>
<p>To setup bumpversion simple create <code class="language-plaintext highlighter-rouge">.bumpversion.cfg</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[bumpversion]
current_version = 0.0.1
commit = True
tag = True
[bumpversion:file:setup.py]
[bumpversion:file:example/__init__.py]
</code></pre></div></div>
<p>Here we are recording the <code class="language-plaintext highlighter-rouge">current_version</code> which is used to do the
find and replace in our files, <code class="language-plaintext highlighter-rouge">commit = True</code> and <code class="language-plaintext highlighter-rouge">tag = True</code> is saying
if we are in a git repo, commit and tag the repo. If you are in a git repo
the action will fail if the repo is not clean.</p>
<p>With bumpversion you can specify the new version, but I prefer to just
specify the part to bump, aka <code class="language-plaintext highlighter-rouge">major|minor|patch</code>. So actually bumping the
version is done like so:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bumpversion patch
</code></pre></div></div>
<p>for a patch bump. If you just want to see what would happen you can do the
following:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bumpversion --dry-run --allow-dirty --verbose patch
</code></pre></div></div>
<p>As I’m usually using <a href="http://www.pyinvoke.org/">Python Invoke</a>, I create the following
task:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">invoke</span> <span class="kn">import</span> <span class="n">task</span>
<span class="o">@</span><span class="n">task</span>
<span class="k">def</span> <span class="nf">bump_version</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span> <span class="n">part</span><span class="p">,</span> <span class="n">confirm</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span>
<span class="k">if</span> <span class="n">confirm</span><span class="p">:</span>
<span class="n">ctx</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="s">'bumpversion {part}'</span><span class="o">.</span><span class="nb">format</span><span class="p">(</span><span class="n">part</span><span class="o">=</span><span class="n">part</span><span class="p">))</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">ctx</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="s">'bumpversion --dry-run --allow-dirty --verbose {part}'</span><span class="o">.</span><span class="nb">format</span><span class="p">(</span><span class="n">part</span><span class="o">=</span><span class="n">part</span><span class="p">))</span>
<span class="k">print</span><span class="p">(</span><span class="s">'Add "--confirm" to actually perform the bump version.'</span><span class="p">)</span>
</code></pre></div></div>
<p>Therefore to do a quick comfort check:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>invoke bump-version patch
</code></pre></div></div>
<p>and if I’m happy and want to actually bump the version:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>invoke bump-version patch --confirm
</code></pre></div></div>For any software project it is likely that you're will need to define the version in sperate place and keeping track of that can be a bit of pain.Python invoke with tab completion2018-06-03T00:00:00+00:002018-06-03T00:00:00+00:00http://almcc.me/blog/2018/06/03/python-invoke-with-tab-completion<p><a href="http://www.pyinvoke.org/">Python Invoke</a> is very handy task execution tool, I use it for most of
my projects. Before that I had been using a <code class="language-plaintext highlighter-rouge">Makefile</code> but I like that now I can
use python, but still call out to a shell if I need. To create a very simple example
create a <code class="language-plaintext highlighter-rouge">tasks.py</code> file at the base of your project:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">invoke</span> <span class="kn">import</span> <span class="n">task</span>
<span class="o">@</span><span class="n">task</span>
<span class="k">def</span> <span class="nf">date</span><span class="p">(</span><span class="n">ctx</span><span class="p">):</span>
<span class="n">ctx</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="s">'date'</span><span class="p">)</span>
</code></pre></div></div>
<p>Now install the Invoke package with <code class="language-plaintext highlighter-rouge">pip install invoke</code> and you should be
able to run your example command:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>invoke <span class="nb">date
</span>Sun 3 Jun 2018 13:52:22 BST
</code></pre></div></div>
<p>A more complex example taken from the Invoke website itself looks like this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">invoke</span> <span class="kn">import</span> <span class="n">task</span>
<span class="o">@</span><span class="n">task</span>
<span class="k">def</span> <span class="nf">clean</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">docs</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">bytecode</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">extra</span><span class="o">=</span><span class="s">''</span><span class="p">):</span>
<span class="n">patterns</span> <span class="o">=</span> <span class="p">[</span><span class="s">'build'</span><span class="p">]</span>
<span class="k">if</span> <span class="n">docs</span><span class="p">:</span>
<span class="n">patterns</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s">'docs/_build'</span><span class="p">)</span>
<span class="k">if</span> <span class="n">bytecode</span><span class="p">:</span>
<span class="n">patterns</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s">'**/*.pyc'</span><span class="p">)</span>
<span class="k">if</span> <span class="n">extra</span><span class="p">:</span>
<span class="n">patterns</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">extra</span><span class="p">)</span>
<span class="k">for</span> <span class="n">pattern</span> <span class="ow">in</span> <span class="n">patterns</span><span class="p">:</span>
<span class="n">c</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="s">"rm -rf {}"</span><span class="o">.</span><span class="nb">format</span><span class="p">(</span><span class="n">pattern</span><span class="p">))</span>
<span class="o">@</span><span class="n">task</span>
<span class="k">def</span> <span class="nf">build</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">docs</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span>
<span class="n">c</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="s">"python setup.py build"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">docs</span><span class="p">:</span>
<span class="n">c</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="s">"sphinx-build docs docs/_build"</span><span class="p">)</span>
</code></pre></div></div>
<p>From the command line you can get a list of possible commands with:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>invoke <span class="nt">--list</span>
Available tasks:
build
clean
</code></pre></div></div>
<p>And you can get help on a specific command like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>invoke clean <span class="nt">-h</span>
Usage: inv[oke] <span class="o">[</span><span class="nt">--core-opts</span><span class="o">]</span> clean <span class="o">[</span><span class="nt">--options</span><span class="o">]</span> <span class="o">[</span>other tasks here ...]
Docstring:
none
Options:
<span class="nt">-b</span>, <span class="nt">--bytecode</span>
<span class="nt">-d</span>, <span class="nt">--docs</span>
<span class="nt">-e</span> STRING, <span class="nt">--extra</span><span class="o">=</span>STRING
</code></pre></div></div>
<p>This can be quite cumbersome when you half know what your doing but can’t quite
remember the name of the command or option you need. For this instance we can
enable tab completion. You can read about invoke tab completion in the
<a href="http://docs.pyinvoke.org/en/1.0/invoke.html#shell-tab-completion">Invoke Documentation</a>. I use zsh with
<a href="https://github.com/robbyrussell/oh-my-zsh">Oh My Zsh</a> so downloaded the the completion script to <code class="language-plaintext highlighter-rouge">~/.oh-my-zsh/invoke</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mkdir ~/.oh-my-zsh/invoke
wget https://raw.githubusercontent.com/pyinvoke/invoke/master/completion/zsh -O ~/.oh-my-zsh/invoke/zsh
</code></pre></div></div>
<p>and sourced the file in my <code class="language-plaintext highlighter-rouge">~/.zshrc</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>echo "\nsource ~/.oh-my-zsh/invoke/zsh" >> ~/.zshrc
source ~/.zshrc
</code></pre></div></div>
<p>And now we have tab completion of commands with <code class="language-plaintext highlighter-rouge">invoke <TAB></code> and a commands options
with <code class="language-plaintext highlighter-rouge">invoke <command> -<TAB></code>.</p>
<p>Note, if you in a virtual environment, you might need to reactive it.</p>Python Invoke is very handy task execution tool.Using pip-tool to organise python requirements2018-06-01T00:00:00+00:002018-06-01T00:00:00+00:00http://almcc.me/blog/2018/06/01/using-pip-tool-to-organise-python-requirements<p>I have struggled for a while on how to handle my python requirements,
fighting on one side to pin my requirements to exact versions to have
repeatable builds, and on the other side to keep running the latest packages.
It generally ended up as a mess.</p>
<p>However I have just stumbled across <a href="https://github.com/jazzband/pip-tools">pip-tools</a>, it allows me to keep packages
pinned to exact versions so that a single git commit builds the exact same
image every time, yet at the same time offering the tools to easily update
those packages to the latest ones available.</p>
<p>To get started just create a virtual environment:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python3.6 <span class="nt">-m</span> venv venv
<span class="nb">.</span> ./venv/bin/activate
</code></pre></div></div>
<p>and install pip-tools with:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip <span class="nb">install </span>pip-tools
</code></pre></div></div>
<p>I like to create two requirements files, the first requirements.txt for the bare
minimum to run the application, next dev-requirements.txt which includes everything
else to be able to develop on the application. To use pip-tools, rather than creating
<code class="language-plaintext highlighter-rouge">.txt</code> files, we create <code class="language-plaintext highlighter-rouge">.in</code> files where you list you package names, you can pin
them here if you app requires a particular version, but if it doesn’t just omit the
version number.</p>
<p>Create your <code class="language-plaintext highlighter-rouge">requirements.in</code> and <code class="language-plaintext highlighter-rouge">dev-requirements.in</code> files:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo </span>Django <span class="o">></span> requirements.in
<span class="nb">echo </span>invoke <span class="o">></span> dev-requirements.in
</code></pre></div></div>
<p>Now you can use pip-compile to “compile” your <code class="language-plaintext highlighter-rouge">.in</code> files into the typical python
requirements <code class="language-plaintext highlighter-rouge">.txt</code> files:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip-compile requirements.in
pip-compile dev-requirements.in
</code></pre></div></div>
<p>And now you have fully pinned requirements files that you should commit to source
control along with your <code class="language-plaintext highlighter-rouge">.in</code> files.</p>
<p>You can of course now run <code class="language-plaintext highlighter-rouge">pip install -r requirements.txt -r dev-requirements.txt</code>
however I would suggest you use <code class="language-plaintext highlighter-rouge">pip sync</code> which is part of pip-tools. It will not
only install the required packages, but also remove packages not defined in your
requirements files:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip-sync dev-requirements.txt requirements.txt
</code></pre></div></div>
<p>When you want to update your pinned packages all you need to do is run:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip-compile <span class="nt">--upgrade</span> requirements.in
pip-compile <span class="nt">--upgrade</span> dev-requirements.in
</code></pre></div></div>
<p>and your <code class="language-plaintext highlighter-rouge">.txt</code> files will be update to reflect the latest packages available
on <a href="https://pypi.org/">pypi</a>.</p>
<p>I usually also have a <a href="http://www.pyinvoke.org/">Python Invoke</a> <code class="language-plaintext highlighter-rouge">tasks.py</code> file for my projects,
I have added the follow two commands to help me with syncing my virtual environment
and also updating my requirements.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">invoke</span> <span class="kn">import</span> <span class="n">task</span>
<span class="o">@</span><span class="n">task</span>
<span class="k">def</span> <span class="nf">sync_venv</span><span class="p">(</span><span class="n">ctx</span><span class="p">):</span>
<span class="s">"""Sync the local venv with requirements."""</span>
<span class="n">ctx</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="s">'pip-sync dev-requirements.txt requirements.txt'</span><span class="p">)</span>
<span class="o">@</span><span class="n">task</span>
<span class="k">def</span> <span class="nf">update_requirements</span><span class="p">(</span><span class="n">ctx</span><span class="p">):</span>
<span class="s">"""Update the requirements.txt files to the latest packages."""</span>
<span class="n">ctx</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="s">'pip-compile --upgrade requirements.in'</span><span class="p">)</span>
<span class="n">ctx</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="s">'pip-compile --upgrade dev-requirements.in'</span><span class="p">)</span>
</code></pre></div></div>
<p>Now I can be in control of when I grab new pip packages, but updating remains pretty easy.</p>I have struggled for a while on how to handle my python requirements, fighting on one side to pin my requirements to exact versions to have repeatable builds, and on the other side to keep running the latest packages.Keeping secrets out of your source2017-05-05T00:00:00+00:002017-05-05T00:00:00+00:00http://almcc.me/blog/2017/05/05/keeping-secrets-out-of-your-source<h3 id="envsubst">envsubst</h3>
<p>I first came across <code class="language-plaintext highlighter-rouge">envsubst</code> <a href="https://github.com/CrunchyData/crunchy-containers/blob/a9b7e21921b49e177b4b0e5769ce16ff5250ef93/examples/kube/statefulset/run.sh#L30">here</a> whist doing some research. It seemed to me like neat way of keeping configuration items out of my source code. This could be application secrets or deployment specific items. I decided to play a little further. While <code class="language-plaintext highlighter-rouge">envsubst</code> seems to be pretty common place on linux platforms, it’s not on MacOS so first I had to install it for testing:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>brew install gettext
brew link --force gettext
</code></pre></div></div>
<p>Thanks to <a href="http://stackoverflow.com/a/37192554">this</a> stack-overflow answer for the instructions.</p>
<p>At it’s heart envsubst is just infection environment variables into text streams, to test it out for yourself do the following:</p>
<ol>
<li>
<p>Create an environment variable using <code class="language-plaintext highlighter-rouge">export</code> and us <code class="language-plaintext highlighter-rouge">env</code> to double check.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>export EXAMPLE=alpha
env | grep EXAMPLE
</code></pre></div> </div>
</li>
<li>
<p>Create an example input file, again checking it’s contents.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>echo 'A simple test. EXAMPLE=${EXAMPLE}' > /tmp/example-in
cat /tmp/example-in
</code></pre></div> </div>
</li>
<li>
<p>Test the output of <code class="language-plaintext highlighter-rouge">envsubst</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>envsubst < /tmp/example-in
</code></pre></div> </div>
</li>
<li>
<p>Take the output and put it into a file this time. Checking the contents and then removing the environment variable.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>envsubst < /tmp/example-in > /tmp/example-out
cat /tmp/example-out
unset EXAMPLE
</code></pre></div> </div>
</li>
</ol>
<p><code class="language-plaintext highlighter-rouge">envsubst</code> is very useful if you want to keep post of the file in your code repository and then just inject some values in at the last minute, for example in a continuous delivery pipeline.</p>
<h3 id="base64">base64</h3>
<p>Making use of the <code class="language-plaintext highlighter-rouge">base64</code> command, however, can be very useful for removing whole files out of your source and into environment variables. A very simple description of base64 is that it’s an encoding that use’s visible characters to encode binary data. In this exampe, I use the <code class="language-plaintext highlighter-rouge">echo</code> command to pipe “alpha” into the <code class="language-plaintext highlighter-rouge">base64</code> command using and input (<code class="language-plaintext highlighter-rouge">-i -</code>) of std-in.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>echo -n "alpha" | base64 -i -
</code></pre></div></div>
<p>As you can see it will return a seemingly nonsensical string, however, if we pipe it back to <code class="language-plaintext highlighter-rouge">base64</code> this time with <code class="language-plaintext highlighter-rouge">-d</code> to decode we get the original text back.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>echo -n "alpha" | base64 -i - | base64 -i - -D
</code></pre></div></div>
<p>Note: In the examples above I use <code class="language-plaintext highlighter-rouge">-n</code> flag on the <code class="language-plaintext highlighter-rouge">echo</code> command to make sure no newline character is appended.</p>
<p>A more involved example, however, should probably include a file.</p>
<ol>
<li>
<p>Taking inspiration from my very exiting example above create a file:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>echo 'A simple test. EXAMPLE=alpha' > /tmp/example
cat /tmp/example
</code></pre></div> </div>
</li>
<li>
<p>Use <code class="language-plaintext highlighter-rouge">base64</code> encode the file to into an environment variable.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>export EXAMPLE=$(base64 -i /tmp/example)
env | grep EXAMPLE
</code></pre></div> </div>
</li>
<li>
<p>Use <code class="language-plaintext highlighter-rouge">base64</code> to recreate the file.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>echo $EXAMPLE | base64 -i - -D > /tmp/example2
cat /tmp/example2
</code></pre></div> </div>
</li>
<li>
<p>(optional) You could instead decode the variable directly from your code,
for example python.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">base64</span>
<span class="n">base64</span><span class="o">.</span><span class="n">b64decode</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">'EXAMPLE'</span><span class="p">))</span>
</code></pre></div> </div>
</li>
</ol>
<p>This post wasn’t so much about where to keep your secrets, but more about once you have removed the things you want to remove, how do you get them back in during a continuous delivery pipeline.</p>I first came across `envsubst` [here][first-envsubst] whist doing some research. It seemed to me like neat way of keeping configuration items out of my source code.Removing older versions on Google App Engine2017-05-04T00:00:00+00:002017-05-04T00:00:00+00:00http://almcc.me/blog/2017/05/04/removing-older-versions-on-google-app-engine<p>Google App Engine holds onto older versions of your app after you push a new
deployment, however there are limits (20) to how many you can keep. If you
hit this limit any new pushes will fail to deploy. I wanted my Continuous
Deployment pipeline to keep the last 3 versions of a service. The following
bash script will do this for me:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="nv">VERSIONS</span><span class="o">=</span><span class="si">$(</span>gcloud app versions list <span class="nt">--service</span> <span class="nv">$1</span> <span class="nt">--sort-by</span> <span class="s1">'~version'</span> <span class="nt">--format</span> <span class="s1">'value(version.id)'</span><span class="si">)</span>
<span class="nv">COUNT</span><span class="o">=</span>0
<span class="nb">echo</span> <span class="s2">"Keeping the </span><span class="nv">$2</span><span class="s2"> latest versions of the </span><span class="nv">$1</span><span class="s2"> service"</span>
<span class="k">for </span>VERSION <span class="k">in</span> <span class="nv">$VERSIONS</span>
<span class="k">do</span>
<span class="o">((</span>COUNT++<span class="o">))</span>
<span class="k">if</span> <span class="o">[</span> <span class="nv">$COUNT</span> <span class="nt">-gt</span> <span class="nv">$2</span> <span class="o">]</span>
<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"Going to delete version </span><span class="nv">$VERSION</span><span class="s2"> of the </span><span class="nv">$1</span><span class="s2"> service."</span>
gcloud app versions delete <span class="nv">$VERSION</span> <span class="nt">--service</span> <span class="nv">$1</span> <span class="nt">-q</span>
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"Going to keep version </span><span class="nv">$VERSION</span><span class="s2"> of the </span><span class="nv">$1</span><span class="s2"> service."</span>
<span class="k">fi
done</span>
</code></pre></div></div>
<p>You can call it like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash delete-older-gcloud-app-versions.sh default 3
</code></pre></div></div>
<p>The first argument is the service name and the the next argument is the number
of versions to keep.</p>Google App Engine holds onto older versions of your app after you push a new deployment, however there are limits (20) to how many you can keep.Cleaning up old git branches2017-05-02T00:00:00+00:002017-05-02T00:00:00+00:00http://almcc.me/blog/2017/05/02/cleaning-up-old-git-branches<p>When you are using feature branching and constantly and merging often,
your local git tree can start to fill up with lots of old branches that
have been long since merged. Deleting them can be a real chore, all thing
being relative here! I found this handy tactic on
<a href="http://stackoverflow.com/a/28464339">stack overflow</a>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git branch <span class="nt">--merged</span> <span class="o">></span>/tmp/merged-branches <span class="o">&&</span> <span class="se">\</span>
vi /tmp/merged-branches <span class="o">&&</span> <span class="se">\</span>
xargs git branch <span class="nt">-d</span> </tmp/merged-branches
</code></pre></div></div>
<p>What is happening here is that you listing all the branches that are
already merged into the current branch and saving that to a file. Opening
<code class="language-plaintext highlighter-rouge">vi</code> to allow you a chance to remove some branches from the pending
deletion. Finally, we are asking git to remove those branches. I find it’s
a nice balance between the ‘just do it’ mentality but also giving you
and opportunity to choose what branches get deleted.</p>
<p>It’s is quite likely though that master will be in the list of branches
returned by <code class="language-plaintext highlighter-rouge">git branch --merged</code> so I have made the following alteration</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git branch <span class="nt">--merged</span> | <span class="nb">grep</span> <span class="nt">-v</span> master <span class="o">></span>/tmp/merged-branches <span class="o">&&</span> <span class="se">\</span>
vi /tmp/merged-branches <span class="o">&&</span> <span class="se">\</span>
xargs git branch <span class="nt">-d</span> </tmp/merged-branches
</code></pre></div></div>
<p>I have used grep to capture the master branch and then inverted the match
so that I get everything other than the master branch.</p>
<p>If you find yourself in <code class="language-plaintext highlighter-rouge">vi</code> and you have decided you don’t want to delete
any branches then you could just delete all the lines (<code class="language-plaintext highlighter-rouge">dd</code> for each line),
or you could bail out with <code class="language-plaintext highlighter-rouge">:cq</code> which will force via to exit with a non-zero
exit code. This means that the command will not continue onto the delete step.</p>When you are using feature branching and constantly and merging often, your local git tree can start to fill up with lots of old branches that have been long since merged.Python logging decorator2015-11-13T00:00:00+00:002015-11-13T00:00:00+00:00http://almcc.me/blog/2015/11/13/python-logging-decorator<p>A python decorator can be a handy way to quickly get a grasp of what is
going on with your code. Below is a function decorator for logging when a
python function is called, what it has been called with, what it returned,
or what error it has raised.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">functools</span> <span class="kn">import</span> <span class="n">wraps</span>
<span class="k">def</span> <span class="nf">logged</span><span class="p">(</span><span class="n">log</span><span class="o">=</span><span class="s">'trace'</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">wrap</span><span class="p">(</span><span class="n">function</span><span class="p">):</span>
<span class="o">@</span><span class="n">wraps</span><span class="p">(</span><span class="n">function</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">wrapper</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="n">logger</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="n">log</span><span class="p">)</span>
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s">"Calling function '{}' with args={} kwargs={}"</span>
<span class="o">.</span><span class="nb">format</span><span class="p">(</span><span class="n">function</span><span class="o">.</span><span class="n">__name__</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">kwargs</span><span class="p">))</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">function</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">except</span> <span class="nb">Exception</span> <span class="k">as</span> <span class="n">error</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s">"Function '{}' raised {} with error '{}'"</span>
<span class="o">.</span><span class="nb">format</span><span class="p">(</span><span class="n">function</span><span class="o">.</span><span class="n">__name__</span><span class="p">,</span>
<span class="n">error</span><span class="o">.</span><span class="n">__class__</span><span class="o">.</span><span class="n">__name__</span><span class="p">,</span>
<span class="nb">str</span><span class="p">(</span><span class="n">error</span><span class="p">)))</span>
<span class="k">raise</span> <span class="n">error</span>
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s">"Function '{}' returned {}"</span>
<span class="o">.</span><span class="nb">format</span><span class="p">(</span><span class="n">function</span><span class="o">.</span><span class="n">__name__</span><span class="p">,</span>
<span class="n">response</span><span class="p">))</span>
<span class="k">return</span> <span class="n">response</span>
<span class="k">return</span> <span class="n">wrapper</span>
<span class="k">return</span> <span class="n">wrap</span>
</code></pre></div></div>
<p>To enable logging for a particular function simple do the following:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">@</span><span class="n">logged</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">add</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
<span class="k">return</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span>
</code></pre></div></div>
<p>You will also need to set up the python logger, a simple setup to log to
<code class="language-plaintext highlighter-rouge">stdout</code> would look like this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="n">logger</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">()</span>
<span class="n">logger</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">DEBUG</span><span class="p">)</span>
<span class="n">handler</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">StreamHandler</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="p">)</span>
<span class="n">handler</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">DEBUG</span><span class="p">)</span>
<span class="n">formatter</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">Formatter</span><span class="p">(</span><span class="s">'</span><span class="si">%(asctime)</span><span class="s">s - </span><span class="si">%(name)</span><span class="s">s - </span><span class="si">%(levelname)</span><span class="s">s - </span><span class="si">%(message)</span><span class="s">s'</span><span class="p">)</span>
<span class="n">handler</span><span class="o">.</span><span class="n">setFormatter</span><span class="p">(</span><span class="n">formatter</span><span class="p">)</span>
<span class="n">logger</span><span class="o">.</span><span class="n">addHandler</span><span class="p">(</span><span class="n">handler</span><span class="p">)</span>
</code></pre></div></div>
<p>Then output should look something like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2015-11-13 21:52:25,815 - trace - DEBUG - Calling function 'add' with args=(1, 2) kwargs={}
2015-11-13 21:52:25,815 - trace - DEBUG - Function 'add' returned 3
</code></pre></div></div>
<p>Leaving a lot of <code class="language-plaintext highlighter-rouge">logged</code> decorators in place could have a performance hit
but a nice solution might be to decide whether to wrap the function based
on the logging level at the time.</p>A python decorator can be a handy way to quickly get a grasp of what is going on with your code.