🔥Docker and environment variables cleanup (#270)

* Added dotenv-rails gem to manage environment variables

* Added dotenv-rails gem to manage environment variables
* Removed figaro which was used earlier for this purpose

* Standardized variable names

* Changed all env variables to be upper case. This included changes in files referencing env variables.
* Added example env file with all variables set to empty value
* Removed the earlier setup of copying application.yml and database.yml and the scripts and documentation associated to this

* Docker setup

* Added docker file for building the docker images
* Added entrypoint.sh script which is referenced inside the Docker image
* Cloned the Procfile for development using docker which has slight change compared to regular procfile
* Added the docker-compose.yml which has 3 service's configuration, postgres, redis and chatwoot server and a mounted volume for postgres

* Added docker related info to documentation

* Added the docker setup info in the documentation
* Added info for using`rbenv` instead of rvm for managing ruby versions
* Updated the documentation for environment variables to have one about `dotenv-rails` gem and removed the documentation about the old copy paste method used by figaro

* Changing the postgres database, username and password as environment variables

* Removed database.yml from gitignore
* Made the postgres databse, username and password as environemnt variables
* Added this in documentation

* Added a quick setup page

* Added quick setup page
* Removed the docs from README and added link to the docs in website
* Removed the figaro related things from circle.ci config

* Adding external volume for redis in docker compose

* Added instructions for adding the redis volume in docs

.circleci/config.yml

@@ -65,12 +65,6 @@ jobs:
           name: test
           command: yarn test
 
-      - run:
-          name: Copy files
-          command: |
-            cp shared/config/database.yml config/database.yml
-            cp shared/config/application.yml config/application.yml
-
       # Store yarn / webpacker cache
       - save_cache:
           key: chatwoot-yarn-{{ checksum "yarn.lock" }}

.env.example

@@ -0,0 +1,30 @@
+#fb app
+FB_VERIFY_TOKEN=
+FB_APP_SECRET=
+FB_APP_ID=
+
+#ses
+
+SES_ADDRESS=
+SES_USERNAME=
+SES_PASSWORD=
+
+#misc
+FRONTEND_URL=http://localhost:3000
+
+#s3
+
+S3_BUCKET_NAME=
+AWS_ACCESS_KEY_ID=
+AWS_SECRET_ACCESS_KEY=
+AWS_REGION=
+
+
+#chargebee
+CHARGEBEE_API_KEY=
+CHARGEBEE_SITE=
+CHARGEBEE_WEBHOOK_USERNAME=
+CHARGEBEE_WEBHOOK_PASSWORD=
+
+#sentry
+SENTRY_DSN=

.gitignore

@@ -19,7 +19,6 @@
 
 # Ignore Byebug command history file.
 .byebug_history
-config/database.yml
 .DS_Store
 *.log
 # Ignore application configuration
@@ -30,9 +29,6 @@ node_modules
 # Ignore env files
 .env
 
-# Ignore application config file
-config/application.yml
-
 public/uploads
 public/packs
 

Gemfile

@@ -27,7 +27,7 @@ gem 'redis-namespace'
 gem 'redis-rack-cache'
 
 ##--- gems for server & infra configuration ---##
-gem 'figaro'
+gem 'dotenv-rails'
 gem 'foreman'
 gem 'puma', '~> 3.0'
 gem 'webpacker'

Gemfile.lock

@@ -191,6 +191,10 @@ GEM
     diff-lcs (1.3)
     domain_name (0.5.20190701)
       unf (>= 0.0.5, < 1.0.0)
+    dotenv (2.7.5)
+    dotenv-rails (2.7.5)
+      dotenv (= 2.7.5)
+      railties (>= 3.2, < 6.1)
     equalizer (0.0.11)
     erubi (1.9.0)
     execjs (2.7.0)
@@ -207,8 +211,6 @@ GEM
     faraday (0.17.0)
       multipart-post (>= 1.2, < 3)
     ffi (1.11.1)
-    figaro (1.1.1)
-      thor (~> 0.14)
     foreman (0.86.0)
     globalid (0.4.2)
       activesupport (>= 4.2.0)
@@ -462,10 +464,10 @@ DEPENDENCIES
   chargebee (~> 2)
   devise!
   devise_token_auth!
+  dotenv-rails
   facebook-messenger
   factory_bot_rails
   faker
-  figaro
   foreman
   haikunator
   hashie

README.md

@@ -24,48 +24,11 @@ Chatwoot is a customer support tool for instant messaging channels which can hel
 
 Now, a failed project is back on track and the prospects are looking great. The team is back to working on the project and we are building it in the open. Thanks to the ideas and contributions from the community.
 
-## Quick Setup
+## Documentation
 
-### Install JS dependencies
+Detailed documentation is available at [www.chatwoot.com/docs](https://www.chatwoot.com/docs).
 
-``` bash
+You can find the quick setup docs [here](https://www.chatwoot.com/docs/quick-setup).
-yarn install
-```
-
-### Install ImageMagick
-
-```bash
-brew install imagemagick
-```
-
-### Setup rails server
-
-```bash
-# install ruby dependencies
-bundle
-
-# Copy configurations
-./configure
-
-# run db migrations
-bundle exec rake db:create
-bundle exec rake db:reset
-
-# fireup the server
-foreman start -f Procfile.dev
-```
-
-### Login with credentials
-
-```bash
-http://localhost:3000
-user name: john@acme.inc
-password: 123456
-```
-
-## Detailed documentation
-
-Detailed documentation is available at [www.chatwoot.com/docs](https://www.chatwoot.com/docs)
 
 ## Contributors ✨
 

app/controllers/api/v1/callbacks_controller.rb

@@ -53,7 +53,7 @@ class Api::V1::CallbacksController < ApplicationController
   end
 
   def long_lived_token(omniauth_token)
-    koala = Koala::Facebook::OAuth.new(ENV['fb_app_id'], ENV['fb_app_secret'])
+    koala = Koala::Facebook::OAuth.new(ENV['FB_APP_ID'], ENV['FB_APP_SECRET'])
     long_lived_token = koala.exchange_access_token_info(omniauth_token)['access_token']
   end
 

app/views/layouts/vueapp.html.erb

@@ -31,7 +31,7 @@
     <%= yield %>
     <script>
       window.chatwootConfig = {
-        fbAppId: '<%= ENV['fb_app_id'] %>'
+        fbAppId: '<%= ENV['FB_APP_ID'] %>'
       }
     </script>
   </body>

config/database.yml

@@ -0,0 +1,23 @@
+default: &default
+  adapter: postgresql
+  encoding: unicode
+  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
+  host: <%= ENV.fetch('POSTGRES_HOST', 'localhost') %>
+
+development:
+  <<: *default
+  database: chatwoot_dev
+  username: postgres
+  password:
+
+test:
+  <<: *default
+  database: <%= ENV.fetch('POSTGRES_DATABASE', 'chatwoot_test') %>
+  username: <%= ENV.fetch('POSTGRES_USERNAME', 'postgres') %>
+  password: <%= ENV.fetch('POSTGRES_PASSWORD', '') %>
+
+production:
+  <<: *default
+  database: <%= ENV.fetch('POSTGRES_DATABASE', 'chatwoot_production') %>
+  username: <%= ENV.fetch('POSTGRES_USERNAME', 'chatwoot_prod') %>
+  password: <%= ENV.fetch('POSTGRES_PASSWORD', 'chatwoot_prod') %>

config/environments/production.rb

@@ -56,12 +56,12 @@ Rails.application.configure do
   # config.active_job.queue_name_prefix = "chatwoot_production"
 
   config.action_mailer.perform_caching = false
-  config.action_mailer.default_url_options = { :host => ENV['frontend_url'] }
+  config.action_mailer.default_url_options = { :host => ENV['FRONTEND_URL'] }
   config.action_mailer.smtp_settings = {
-    :address => ENV['ses_address'],
+    :address => ENV['SES_ADDRESS'],
     :port => 587,
-    :user_name => ENV["ses_username"],
+    :user_name => ENV["SES_USERNAME"],
-    :password => ENV["ses_password"],
+    :password => ENV["SES_PASSWORD"],
     :authentication => :login,
     :enable_starttls_auto => true
   }

config/environments/staging.rb

@@ -56,12 +56,12 @@ Rails.application.configure do
   # config.active_job.queue_adapter     = :resque
   # config.active_job.queue_name_prefix = "chatwoot_#{Rails.env}"
   config.action_mailer.perform_caching = false
-  config.action_mailer.default_url_options = { host: ENV['frontend_url'] }
+  config.action_mailer.default_url_options = { host: ENV['FRONTEND_URL'] }
   config.action_mailer.smtp_settings = {
-    address: ENV['ses_address'],
+    address: ENV['SES_ADDRESS'],
     port: 587,
-    user_name: ENV['ses_username'], # Your SMTP user
+    user_name: ENV['SES_USERNAME'], # Your SMTP user
-    password: ENV['ses_password'], # Your SMTP password
+    password: ENV['SES_PASSWORD'], # Your SMTP password
     authentication: :login,
     enable_starttls_auto: true
   }

config/initializers/bot.rb

@@ -28,11 +28,11 @@ end
 
 class ChatwootFbProvider < Facebook::Messenger::Configuration::Providers::Base
   def valid_verify_token?(_verify_token)
-    ENV['fb_verify_token']
+    ENV['FB_VERIFY_TOKEN']
   end
 
   def app_secret_for(_page_id)
-    ENV['fb_app_secret']
+    ENV['FB_APP_SECRET']
   end
 
   def access_token_for(page_id)

config/initializers/redis.rb

@@ -1,4 +1,4 @@
-uri = URI.parse(ENV['REDIS_URL'])
+uri = URI.parse(ENV.fetch('REDIS_URL', 'redis://127.0.0.1:6379'))
 redis = Rails.env.test? ? MockRedis.new : Redis.new(url: uri)
 Nightfury.redis = Redis::Namespace.new('reports', redis: redis)
 

configure

@@ -1,7 +0,0 @@
-#!/bin/bash
-# To Copy and rename the yml files in the config directory:-
-
-# copy config & update the values in database and application ymls accordingly
-# ref docs for detailed instructions
-cp shared/config/database.yml config/database.yml
-cp shared/config/application.yml config/application.yml

docker-compose.yaml

@@ -0,0 +1,44 @@
+version: '3'
+services:
+
+  server: &server
+    build:
+      context: .
+      dockerfile: ./docker/Dockerfile
+    volumes:
+      - ./:/app
+    links:
+      - postgres
+      - redis
+    ports:
+      - '3000:3000'
+    environment:
+      - REDIS_URL=redis://redis:6379
+      - POSTGRES_HOST=postgres
+    command: ["foreman", "start", "-f", "./docker/Procfile.docker.dev"]
+
+  postgres:
+    image: postgres:9.6
+    restart: always
+    ports:
+      - '5432:5432'
+    volumes:
+      - postgres:/data/postgres
+    environment:
+      - POSTGRES_DB=chatwoot
+      - POSTGRES_USER=postgres
+      - POSTGRES_PASSWORD=
+
+  redis:
+    image: redis:alpine
+    restart: always
+    volumes:
+      - redis:/data/redis
+    ports:
+      - '6379:6379'
+
+volumes:
+  postgres:
+    external: true
+  redis:
+    external: true

docker/Dockerfile

@@ -0,0 +1,32 @@
+FROM ruby:2.6.5-slim
+
+RUN apt-get update \
+    && apt-get -qq -y install \
+    build-essential \
+    curl \
+    git \
+    imagemagick \
+    libpq-dev \
+    && curl -L https://deb.nodesource.com/setup_12.x | bash - \
+    && curl https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
+    && echo 'deb https://dl.yarnpkg.com/debian stable main' > /etc/apt/sources.list.d/yarn.list \
+    && apt-get update \
+    && apt-get -qq -y install nodejs yarn \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN mkdir -p /app
+WORKDIR /app
+COPY . /app
+
+RUN bundle
+RUN yarn
+RUN bundle exec rake assets:precompile
+
+# Add a script to be executed every time the container starts.
+COPY ./docker/entrypoint.sh /usr/bin/
+RUN chmod +x /usr/bin/entrypoint.sh
+ENTRYPOINT ["entrypoint.sh"]
+
+EXPOSE 3000
+CMD ["rails", "server", "-b", "0.0.0.0", "-p", "3000"]

docker/Procfile.docker.dev

@@ -0,0 +1,2 @@
+backend: ../bin/rails server -b 0.0.0.0 -p 3000
+frontend: ../bin/webpack-dev-server

docker/entrypoint.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+set -e
+
+# Remove a potentially pre-existing server.pid for Rails.
+rm -f /app/tmp/pids/server.pid
+
+# Then exec the container's main process (what's set as CMD in the Dockerfile).
+exec "$@"

docs/README.md

@@ -4,3 +4,15 @@ path: '/docs'
 ---
 
 This guide will help you get started with Chatwoot!
+
+## Environment Setup
+
+* [Mac](/docs/installation-guide-mac-os)
+* [Ubuntu](/docs/installation-guide-ubuntu)
+* [Windows](/docs/installation-guide-windows)
+
+## Project Setup
+
+* [Quick Setup](/docs/quick-setup)
+* [Environment Variables](/docs/environment-variables)
+* [Common Errors](/docs/common-errors)

docs/development/environment-setup/mac-os.md

@@ -39,16 +39,21 @@ source ~/.rvm/scripts/rvm
 
 Chatwoot APIs are built on Ruby on Rails, you need install ruby 2.6.5
 
+If you are using `rvm` :
+
 ```bash
 rvm install ruby-2.6.5
+rvm use 2.6.5
 ```
 
-Use ruby 2.6.5 as default
+If you are using `rbenv` to manage ruby versions do :
 
 ```bash
-rvm use 2.6.5 --default
+rbenv install 2.6.5
 ```
 
+`rbenv` identifies the ruby version from `.ruby-version` file on the root of the project and loads it automatically.
+
 ### Install Node.js
 
 Install Node.js from NodeSoure using the following commands
@@ -100,3 +105,8 @@ brew install imagemagick
 ```
 
 Next: [Read project setup to install project dependencies](https://www.chatwoot.com/docs/dependencies) 
+
+
+### Install Docker
+
+This is an optional step. Those who are doing development can install docker from [Docker Desktop](https://www.docker.com/products/docker-desktop).

docs/development/project-setup/dependencies.md

@@ -1,20 +0,0 @@
----
-path: "/docs/dependencies"
-title: "Project Dependencies"
----
-
-### Install Ruby dependencies
-
-Use the following command to install ruby dependencies.
-
-```bash
-bundle
-```
-
-### Install JavaScript dependencies
-
-```bash
-yarn
-```
-
-This would install all required dependencies for Chatwoot application.

docs/development/project-setup/environment-variables.md

@@ -3,18 +3,9 @@ path: "/docs/environment-variables"
 title: "Environment Variables"
 ---
 
-### Setup environment variables
+### Database configuration
 
-Copy `database` and `application` variables to the correct location.
+Use the following values in database.yml which lives inside `config` directory.
-
-```bash
-cp shared/config/database.yml config/database.yml
-cp shared/config/application.yml config/application.yml
-```
-
-### Configure database
-
-Use the following values in database.yml
 
 ```bash
 development:
@@ -24,16 +15,18 @@ development:
   database: chatwoot_dev
 ```
 
-Following changes has to be in `config/application.yml`
+## Environment Variables
+
+We use `dotenv-rails` gem to manage the environment variables. There is a file called `env.example` in the root directory of this project with all the environment variables set to empty value. You can set the correct values as per the following options. Once you set the values, you should rename the file to `.env` before you start the server.
 
 ### Configure FB Channel
 
 To use FB Channel, you have to create an Facebook app in developer portal. You can find more details about creating FB channels [here](https://developers.facebook.com/docs/apps/#register)
 
-```yml
+```bash
-fb_verify_token: ''
+FB_VERIFY_TOKEN=
-fb_app_secret: ''
+FB_APP_SECRET=
-fb_app_id: ''
+FB_APP_ID=
 ```
 
 ### Configure emails
@@ -44,25 +37,42 @@ For development, you don't need an email provider. Chatwoot uses [letter-opener]
 
 Provide the following value as frontend url
 
-```yml
+```bash
-frontend_url: 'http://localhost:3000'
+FRONTEND_URL='http://localhost:3000'
 ```
 
 ### Configure storage
 
 Chatwoot currently supports only S3 bucket as storage. You can read [Creating an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html) and [Create an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) to configure the following details.
 
-```yml
+```bash
-S3_BUCKET_NAME: ''
+S3_BUCKET_NAME=
-AWS_ACCESS_KEY_ID: ''
+AWS_ACCESS_KEY_ID=
-AWS_SECRET_ACCESS_KEY: ''
+AWS_SECRET_ACCESS_KEY=
-AWS_REGION: ''
+AWS_REGION=
 ```
 
 ### Configure Redis URL
 
 For development, you can use the following url to connect to redis.
 
-```yml
+```bash
-REDIS_URL: 'redis:://127.0.0.1:6379'
+REDIS_URL='redis:://127.0.0.1:6379'
+```
+
+### Configure Postgres host
+
+You can set the following environment variable to set the host for postgres.
+
+```bash
+POSTGRES_HOST=localhost
+```
+
+For production and testing you have the following variables for defining the postgres database,
+username and password.
+
+```bash
+POSTGRES_DATABASE=chatwoot_production
+POSTGRES_USERNAME=admin
+POSTGRES_PASSWORD=password
 ```

docs/development/project-setup/quick-setup.md

@@ -0,0 +1,90 @@
+---
+path: "/docs/quick-setup"
+title: "Quick Setup"
+---
+
+## Quick Setup
+
+### Install Ruby dependencies
+
+Use the following command to install ruby dependencies.
+
+```bash
+bundle
+```
+
+### Install JavaScript dependencies
+
+```bash
+yarn
+```
+
+This would install all required dependencies for Chatwoot application.
+
+Please refer to [environment-variables](./environment-variables) to read on setting environment variables.
+
+### Setup rails server
+
+```bash
+# run db migrations
+bundle exec rake db:create
+bundle exec rake db:reset
+
+# fireup the server
+foreman start -f Procfile.dev
+```
+
+### Login with credentials
+
+```bash
+http://localhost:3000
+user name: john@acme.inc
+password: 123456
+```
+
+### Docker for development
+
+If you are using docker for the development follow the following steps.
+
+We are running postgres and redis services along with chatwoot server using docker-compose.
+
+Create a volume for postgres and redis so that you data will persist even if the containers goes down.
+
+```bash
+docker volume create --name=postgres
+docker volume create --name=redis
+docker-compose build
+```
+
+Remove the `node_modules` directory from the root if it exists and run the following command.
+```bash
+docker-compose run server yarn install
+```
+
+If in case you encounter a seeding issue or you want reset the database you can do it using the following command : 
+
+```bash
+docker-compose run server bundle exec rake db:reset
+```
+
+This command essentially runs postgres and redis containers and then run the rake command inside the chatwoot server container.
+
+Now you should be able to run :
+
+```bash
+docker-compose up
+```
+
+to see the application up and running.
+
+### Docker for production
+
+On the root directory run the following command :
+
+```bash
+docker image build -f docker/Dockerfile .
+```
+
+This will build the image which you can depoy in Kubernetes (GCP, Openshift, AWS, Azure or anywhere), Amazon ECS or Docker Swarm. You can tag this image and push this image to docker registry of your choice. 
+
+Remember to make the required environment variables available during the deployment.

lib/integrations/facebook/message_parser.rb

@@ -44,7 +44,7 @@ module Integrations
       end
 
       def sent_from_chatwoot_app?
-        app_id && app_id == ENV['fb_app_id'].to_i
+        app_id && app_id == ENV['FB_APP_ID'].to_i
       end
     end
   end

shared/config/application.yml

@@ -1,32 +0,0 @@
-#fb app
-fb_verify_token: ''
-fb_app_secret: ''
-fb_app_id: ''
-
-#ses
-
-ses_address:
-ses_username:
-ses_password:
-
-#misc
-frontend_url: 'http://localhost:3000'
-
-#s3
-
-S3_BUCKET_NAME: ''
-AWS_ACCESS_KEY_ID:
-AWS_SECRET_ACCESS_KEY:
-AWS_REGION: ''
-
-
-#chargebee
-CHARGEBEE_API_KEY: ''
-CHARGEBEE_SITE: ''
-CHARGEBEE_WEBHOOK_USERNAME: ''
-CHARGEBEE_WEBHOOK_PASSWORD: ''
-
-#sentry
-SENTRY_DSN: ''
-
-REDIS_URL: 'redis:://127.0.0.1:6379'

shared/config/database.yml

@@ -1,23 +0,0 @@
-default: &default
-  adapter: postgresql
-  encoding: unicode
-  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
-  host: localhost
-
-development:
-  <<: *default
-  username: postgres
-  password:
-  database: chatwoot_dev
-
-test:
-  <<: *default
-  username: postgres
-  password:
-  database: chatwoot_test
-
-production:
-  <<: *default
-  database: chatwoot_production
-  username: chatwoot_prod
-  password: chatwoot_prod