Prerequisites

You should have node package manager, the docker engine, and docker compose installed.

Make sure you are using Node v14.xx.x.

If you do not have an ELK stack already, you can use this docker image to get started.

Additionally, your AWS account must have the following permissions. Best practice is to create a brand new IAM role with these permissions attached, and then to assign this role to your user.

"iam:PassRole",
"iam:DetachRolePolicy",
"iam:CreatePolicy",
"iam:CreateRole",
"iam:AttachRolePolicy",
"iam:DeletePolicy",
"iam:DeleteRole",
"kms:Decrypt",
"kms:CreateGrant",
"kms:RetireGrant",
"kms:Encrypt",
"lambda:CreateEventSourceMapping",
"lambda:CreateFunction",
"lambda:DeleteEventSourceMapping",
"lambda:DeleteFunction",
"s3:DeleteObject",
"s3:DeleteBucket",
"s3:CreateBucket",
"s3:ListBucket",
"s3:PutObject",
"s3:GetObject",
"sqs:SendMessage",
"sqs:GetQueueAttributes",
"sqs:DeleteQueue",
"sqs:CreateQueue"

Also, you must configure the http plug-in for logstash. Here is an example of a Logstash config file that includes configuration for the plugin:

input {
    http {
      port => destired_port // default port 8080
    }
}

filter {
  mutate {
    // removes axios artifacts
    remove_field => ["[event][original]", "[http]", "[user_agent]", "[url]"]
  }
}

output {
  stdout {
    codec => rubydebug
  }

  elasticsearch {
    hosts => "your_elasticsearch_primary_node"
    index => "your_desired_index_name"  //example: s3logs-%{+YYYY.MM.dd}
  }
}

Finally, your logs to be re-ingested must be stored in JSON format in an S3 Bucket. It is recommended that your S3 Bucket has only logs in it.

One Time Set-up for Arroyo

Clone the repo

git clone https://github.com/Team-Arroyo/arroyo-deployment.git

Change directories

cd arroyo-deployment

Install dependencies

npm install

Create .env file

touch .env

Using your favorite text editor, add the items listed below:

AWS_REGION=your-region
AWS_BUCKET_NAME=name-of-your-s3-bucket-containing-logs
AWS_ACCESS_KEY=your-access-key
AWS_SECRET_KEY=your-secret-key
LOGSTASH_HOST=your-logstash-host

To deploy AWS infrastructure

npm run deploy

To run Arroyo

See our Readme here for instructions on how to perform a bulk re-ingest or query based re-ingest using the browser based graphical user interface.

To start Arroyo for the first time

docker compose up

To restart Arroyo

docker compose start

To stop Arroyo

docker compose stop

To stop Arroyo and remove containers and docker network

docker compose down

Tear-down Arroyo

To destroy AWS infrastructure:

npm run destroy