Part 1 of series "AWS cloud components -simplified - The Power of Lambda"

AWS S3 bucket:

This component can be used to receive files coming from external sources. The file is routed via API Gateway in order to balance load, firewall unprecedented request for security and serve as a proxy server.

AWS Lambda:

This will monitor S3 bucket in order to record landing of a file. It will then provision the event of incoming file in bucket to different tasks like reception and transformation.

An example:

- S3 bucket can be structured to contain sub-folders named pending, processing, received, renamed, transformed, validated, error, exception and archive.

- Incoming Files can be filtered and sent to Pending folder on file creation event (marks file arrival). This will also instantiate AWS lambda.

- AWS lambda can then store filename and status in the table with status=’Received’. It will also invoke EC2 container (reception task) for further processing.

- Reception task is a docker instance running on EC2. It will identify the file that is ready and read the file from S3 and process it.

- Post job in ACW to invoke ProcessorLambda.

- On successful processing, the last step is to post an ACW job to invoke SubmissionLambda

- ACW will trigger based on time event (will trigger on specific time as per configuration) and invoke ExceptionTask via ExceptionLambda

AWS Cloudwatch:

ACW will perform orchestration role. It will ensure start of different tasks based on completion of previous tasks.

AWS SNS:

SNS is primarily used as a notification service. It will serve as an interface to report errors during processing of a task. It will send a notification in order to log the error using already existent loggly service.

AWS EC2 cluster:

EC2 cluster will comprise of several dockerized tasks. These are reception, transformation, submission and exception.

AWS RDS:

We can make use of AWS RDS (postgres) in order to hold DB objects. Broadly it needs to hold DB objects to preserve any processing or transnational tasks.