SQS (S3 Event) Lambda Trigger

I ran into a little issue today parsing a S3 SQS event that was sent to Lambda via a SQS trigger.  I assumed the incoming event to Lambda was 100% of type dict.  Given this, I assumed I could pull the bucket name and key using this syntax.

bucketname = event['Records'][0]['body']['Records'][0]['s3']['bucket']['name']
objectname = event['Records'][0]['body']['Records'][0]['s3']['object']['key']

As it turns out the incoming event is not 100% of type dict and I got the following error.  

string indices must be integers

The Records after the body ([‘Records’][0][‘body’]) are of type str.  Below is the updated code to pull the bucket name and key from the incoming event.

event_body_records_string = event['Records'][0]['body']
event_body_records_dict = json.loads(event_body_records_string)

bucketname = event_body_records_dict['Records'][0]['s3']['bucaket']['name']
objectname = event_body_records_dict['Records'][0]['s3']['object']['key']

Now everything works out great!!!


Python – Slate – Install – Failed

I tried to install the Slate python package via PIP today.  No good, I ran into the following error.

  • Command “python egg_info” failed with error code 1 in…

The work-around was to bypass and install directly from guthub.

AWS SQS – receive_message

When using the “receive_message” Python Boto function to pull message(s) from a SQS queue, you will always get a response back when the command completes.  However, how do you determine if the response you got back actually contains a valid message?

Quick trick:

response = sqs.receive_message
if 'Messages' in response:
    print("Message on the queue to process")
    print("No messages on the queue to process")

Thats about it!!


Blink(1) USB Light

I just got my Blink(1) in the mail today.  Its a cool indicator light that hooks into your USB port.


There is a ton of potential with this device.  You can utilize the Blink1Control2 application to setup event polling triggers or just play around with the colors. (e.g. Show red color when a specific email is received)

Or you can enable your device to listen on a specific port, which would allow you to update the device remotely via a curl command. (e.g. Jenkin job failure)  You can find the URL summary here.

Another option to using the light is to integrate it into a program, which would allow for more customizations.  Pretty easy to do with Python.  The Blink Python lib can be found here.

Python – Tesseract – OCR – IMAGE

You can do some pretty cool things with tesseract-ocr.  Using PyOCR, which is a wrapper for Tesseract, you can generate text from an image using Tesseract.

Example Image:


Example Output:


Example Code:

from wand.image import Image
from PIL import Image as PI
import pyocr
import io
import sys

tool = pyocr.get_available_tools()[0]
lang = tool.get_available_languages()[1]
txt_list = tool.image_to_string('/home/build/aws.jpg'),

outputFile = open('output.txt', 'w')
for item in txt_list:
 outputFile.write("%s" % item)

Another use case I was working on today was rendering the text in a PDF file using Tesseract.  I was converting the PDF to an image file first, then performing the above actions to read the text from the new image.

Here are a couple valuable resources I used to complete this little test.

  • Installing Tesseract on a RHEL system – (link)
  • Installing PyOCR and other image conversion tools – (link)

Getting all the prerequisites installed was by far the hardest part on this effort.

AWS Lambda and S3

Below are a couple of problems I ran into when writing a Python 2.7 Lambda function that created a file and then uploaded it to S3. (s3.upload_file)

  1. The file I was creating and writing to in the function was empty in S3 after the upload.
    • Turns out I needed the “( )” braces on the Python “close” command.  Silly issue, but took my like 20 minutes to figure out….
  2. In your Lambda function, you need to create your files under /tmp, which is your functions ephemeral storage.
    • fileName = ‘/tmp/’ + name


Cygwin and PIP Package Missing

I installed Cygwin today and forgot to install the PIP package for Python 2.7.


Looking online, I found that you could install the PIP package by re-running the Cygwin installation .exe.  However, re-running the .exe took a LOT longer than I expected.

It is 100% faster downloading and executing from the following location:

Execute “python -m pip –version” to verify your version!

Flask-RESTful – Basic Authentication

I’m continuing to develop the REST API that will be used with the API.AI Webhook.  I decided that some sort of authentication is needed.

I played around with adding Basic Authentication to my API as API.AI supports this.  Below are the steps I took to get my authentication setup using Flask.  (Recommend reading Flask-HTTPAuth documentation)

  1. Include the necessary package
    • Flask-HTTPAuth==2.3.0
    • flaskauth
  2. Add get_password callback function.
    1. @auth.get_password
      def get_password(username):
          if username == 'devopsunleasheduser':
              return 'devopsunleashedpassword'
          return None
  3. Add error_handler callback function (Note “jsonify()” will need jsonify package)
    1. @auth.error_handler
      def unauthorized():
          # return 403 instead of 401 to prevent browsers from displaying the default
          # auth dialog
          return make_response(jsonify({'message': 'Unauthorized access'}), 403)
  4. Add login_required decorator to both classes to verify authentication before returning any info.
    1. decorators = [auth.login_required]


Python, Flask, and Rest API

This blog post is a continuation of my Google Home\Google Action side project.  The first blog post is here.

The project keeps evolving the more I think about it.  The 2 diagrams below illustrate the workflows that need to be developed.

Populate the Database – Workflow 1


Google Home\Google Action – Workflow 2


So as you can see above, a critical piece of this puzzle is the REST API that needs to be created to allow us to interact with the data.

For the REST API, I have decided to use Python, Flask, and Flask-RESTful.  For this service I can host the API instance on Heroku if needed.  As for which database to use, I am thinking MongoDB, and if needed host it online with mLab.  Tools I will use to help get this API created are Postman, Curl, and PyCharm.  (Great resource)

First – Lets stub out the RestAPI code and the HTTP method “flask-restful” classes using PyCharm.


Below is a “rough” first draft of the python REST API.

from flask import Flask
from flask_restful import reqparse, abort, Api, Resource

app = Flask(__name__)
api = Api(app)

# Sample lunch menu data for testing
    '4202017': {'menu': 'Main entry is corn dogs'},
    '4212017': {'menu': 'Main entry is chicken nuggets'},
    '4222017': {'menu': 'Main entry is tacos'},

# Argument parser - must add all arguments explicitly here
parser = reqparse.RequestParser()

# Check if item exists and error if it doesn't
def error_if_item_doesnt_exist(id):
    if id not in LUNCHMENU:
        abort(404, message="Menu Item {} doesn't exist".format(id))

# Check if item exists and error if it does
def error_if_item_does_exist(id):
    if id in LUNCHMENU:
        abort(404, message="Menu Item {} exist".format(id))

# Define get, delete, and put method class
class MenuItem(Resource):
    def get(self, id):
        return LUNCHMENU[id]

    def delete(self, id):
        del LUNCHMENU[id]
        return '', 204

    def put(self, id):
        args = parser.parse_args()
        LUNCHMENU[id] = {'item': args['item']}
        return LUNCHMENU[id], 201

# Define list and post method class
class MenuList(Resource):
    def get(self):
        return LUNCHMENU

    def post(self):
        args = parser.parse_args()
        LUNCHMENU[args['date']] = {'item': args['item']}
        return LUNCHMENU[args['date']], 201

# Setup Api resource routing to classes here
api.add_resource(MenuList, '/menuitems')
api.add_resource(MenuItem, '/menuitems/<id>')

# If program is executed itself, then run
if __name__ == '__main__':

Next step will be to true up the code to accept\parse the incoming JSON and then send the correct JSON format back to the API.AI agent.

  • Request format –
  • Response format –