Image for post
Image for post
Photo by Hybrid on Unsplash

What is “Web scraping @edge” or “Intranet web scraping”? The ability to extract data from a private website, i.e. from an IoT resource or device that is not publicly reachable. Why should we care? To collect data available only privately and to be able to process them (on the cloud at scale) in order to produce information and insights.

The purpose of this project is the data acquisition from motion detection sensors usually installed for home alarm system. It’s a closed and proprietary system, for obvious security reasons. A Raspberry PI is used in the local network to scrape the UI of Paradox alarm control unit and to send collected data in (near) realtime to AWS Kinesis Data Firehose for subsequent processing. …


Image for post
Image for post
Photo by Djim Loic on Unsplash

As part of an anomaly detection project, I have recently been able to use two very interesting open source products: Prophet released by the Core Data Science team by Facebook and Metaflow, an excellent framework by Netflix. I used Prophet, in a Metaflow flow, to create forecast models of time series. I decided to write this post to share my experience with these two products, creating a small machine learning project.

A small project

Being able to predict the future trend of a time series is very useful in many applications, from the world of finance to sales. For example, we try to predict the direction of the stock market or the correct supply of resources. This post does not set such ambitious goals, but only wants to explore the possibilities offered by Prophet by creating a forecast model that determines the future trend of daily temperatures. To train the model, I used a dataset that collects the minimum daily temperatures over 10 years (1981–1990) in the city of Melbourne, Australia. …


Image for post
Image for post
Photo by Omid Kashmari on Unsplash

What are AWS Lambda layers? As we know, AWS Lambda functions allow to execute code in the cloud according to the serverless paradigm. Each serverless cloud application is normally characterized by multiple independent Lambda functions capable of responding to specific events (like rest API, scheduled, triggers). Each Lambda function is defined by its own deployment package which contains its source code and any requirements, such as additional libraries, dependencies and middleware.

In this type of architecture, the AWS Lambda layers allow to introduce the concept of code/dependency reusability, in order to share modules among different functions: the layers are simple packages that can be reused in Lambda and they actually extend the base runtime. …


After my first approach to AWS Amplify, I want to deal with the implementation of source code linters and end-to-end tests with Cypress, obviously automated in the AWS Amplify CI/CD pipeline. Let’s face up this new challenge!

Image for post
Image for post
Photo by Yancy Min on Unsplash

References: this is the link to the GitHub repository and this is the link to the web application related to this post.

Linter

The linter is a tool that analyzes the source code to flag programming errors, bugs, stylistic errors and suspicious constructs — wikipedia

Linter tools allows to increase the quality of the source code. Using these tools in a CI/CD pipeline also allows to deploy an application when its source code meets certain quality levels. …


Image for post
Image for post
Photo by Annie Spratt on Unsplash

During the past AWS Summit I attended an interesting session concerning AWS Amplify. It quickly convinced me to try this framework. A few days later I had the right opportunity: keep my son trained with the multiplication tables during summer. I decided to develop a simple web application with AWS Amplify in order to discover its strengths and weaknesses.

This is the GitHub repository hosting source code explained in this post and the deployed web application.

What’s Amplify?

AWS Amplify is a set of tools and services that allow a developer to build modern full stack applications using AWS cloud services. For example, having to create a React web application, Amplify allows us to manage the development and deployment of the frontend, backend services and the related CI/CD pipeline. For the same application it is possible to have multiple environments (for example dev, test, stage & production). Amplify also allows to integrate some AWS services very quickly into your frontend, writing very few lines of code: one example above all, authentication with AWS Cognito. …


Image for post
Image for post

Let’s see how it is possible to use Chromium and Selenium in an AWS Lambda function; first, some information for those unfamiliar with these two projects.

2020.09 update: please check project repository for a new version of this project, including Chromium ver. 86.0.4240.0 and Selenium ver. 3.14.

Chromium is the open source browser from which Google Chrome derives. Browsers share most of the code and functionality. However, they differ in terms of license, and Chromium does not support Flash, does not have an automatic update system, and does not collect usage and crash statistics. …


Develop, build, and test your Lambda function without it ever actually entering the AWS cloud!

Image for post
Image for post

Starting from the Python environment used for the development of a sample Lambda function, let’s see how to create a Continuous Integration pipeline that deals with testing it for each new commit. We are going to use Goss as a test tool and TravisCI for the implementation of the CI pipeline.

AWS Lambda in a Docker container

Let’s quickly see how to run an AWS Lambda function in a container based on LambdaCI Docker images.

We use the following Python function, which deals with processing messages queued on SQS and which requires a package (PILLOW) normally not installed in the default Lambda environment. …


Image for post
Image for post

I work on projects that are increasingly oriented towards the serverless paradigm and increasingly implemented on the AWS Lambda platform. Being able to offline develop an AWS Lambda function, comfortably in your favorite IDE, without having to upload the code to be able to test it, allows significantly speed up of activities and increased efficiency.

AWS Lambda environment in docker

That’s right! The solution that allows us to develop AWS Lambda code in offline mode is to use a docker image that replicates in an almost identical manner as the AWS live environment. The docker images available at DockerHub constitute a sandbox inside to perform its function, sure to find the same libraries, file structure and related permissions, environment variables, and production context. …


Image for post
Image for post
Photo by Jason D on Unsplash

Thanks to Certbot and to Electronic Frontier Foundation it is possible to provide a totally free SSL certificate to your website. Certbot is a command line tool to request a valid SSL certificate for your domain, following a process to verify the ownership. The tool can also deal with web server certificate installation and many other tasks (plugins available). This post is a guide about how to automatically request and renew your free SSL certificates with Certbot in AWS Lambda.

Why use Certbot in AWS Lambda?

I deal with several web applications using CloudFront for content distribution, associated with a source S3 bucket. So I decided to create a simple Lambda function that deals with obtaining SSL certificates with Certbot and periodically verifying their expiration date. …


Image for post
Image for post
Photo by Joshua Aragon on Unsplash

Automation is awesome!” — This is one of the slogans that I read more and more often on blogs and articles dealing with DevOps and CI/CD. I agree: the today possibilities to automate integration and deployment processes are incredible.

Suppose, for example, we need to automate the installation of Apache on Ubuntu. We write an Ansible role to perform this operation and we can reuse it as a building block of a larger automated process, such as deploying a complete solution.

One wonders: how to test the role during its development and subsequent updates? Having recently faced a similar challenge, I had the opportunity to create a CI pipeline using very powerful (and free!) tools. …

About

Vittorio Nardone

AWS Certified Solutions Architect — AI/ML enthusiast

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store