Log Search

This project implements a distributed cloud pipeline to search for logs that are constantly being udated to S3 storage, and filter logs with specific characterteristics such as time intervals and Log Levels.

Running this project

This project consists of multiple sub-modules that make up the overall root module. There are 3 modules namely

1. GrpcRestLambda : A scala project that implements a lambda function which can handle requests in both GRPC as well as REST style.
2. GrpcProto: A scala module that consists of the protobuf file. Compiling this project generated the required classes from the protobuf.
3. AkkaService: An Akka service that implements two REST API calls that functions as a wrapper on the client side to re-directs any requests to the AWS API gateway.

Below are the steps that need to be followed in order to run the project

1. Open the command prompt/terminal at the root directory of the project and run the command sbt clean compile. This will compile all the modules in the project
2. If you need to compile a specific module run sbt <Module-name>/compile
3. In order to create a fat jar of the GrpcRestLambda module inorder to be able to deploy the module, execute sbt GrpcRestLambds/assembly
4. To run the Akka service in localhost, execute sbt AkkaService/run

Deploying GrpcRestLambda module to Aws Lambda

1. Copy the fat jar generated by executing sbt GrpcLambda/assembly into an s3 bucket.
2. Create a lambda function by selecting Java 8 as the runtime language.
3. Once the Lambda function is created, within the code tab select upload -> from Amazon s3 -> In the dialog window paste the s3 link to the jar file and submit.
4. Under Runtime setting make sure to specify the pacakge and the class that contains the handle request function that needs to be invoked by Lambda.
5. Make sure to create appropriate IAM roles to allow lambda access rights to the s3 buckets.

Creating AWS API Gateway for the Lambda function

1. From the AWS console, navigate into the API Gateway service.
2. Click on Create API -> Rest API
3. Provide a name for the API and click on create
4. Once create a directive and add a POST method to the api.
5. Under the POST-setup select integration type to be Lambda Function, check 'Use Lambda Proxy integration'.
6. Enter the URN of the lambda function in the field and click on create.
7. Next under action click on Deploy API to deploy the api publicly

Once the above steps are complete. A unique api url will be provided using which post calls can be made to the lambda function via the API gateway.

Test Cases

Test cases have been written to test several util function used by the GrpcRestLambda module.

1. To run tests open terminal in the project directory.
2. run sbt GrpcRestLambda/test.
3. check the status of all executed test cases.

Implementation Details

Video Presentation