Live Algo Trading on the Cloud – AWS

What does live algorithmic trading on the Cloud mean?

Live algorithmic trading on the Cloud means that your trading bots can use the cloud provider’s resources to run 24/7 while being easily maintainable.

What are the pros and cons of deploying your trading strategies to the Cloud?

The pros:

• Cloud services are easily accessible – they allow the user to access them at any time, from anywhere, and from almost any device.
  
• Can lower costs – cloud computing can lower the costs and provide you with a good connection and hardware that saves you money in the long run.
  
• Easy to maintain – you can upload your trading strategies to the cloud and get them running 24/7 with minimal effort.
  
• Is scalable – you can store almost limitless amounts of data in the cloud.
  
• Easy to set up – setting up your cloud server is quite easy.

The cons:

• Security risks – your data is being stored by a third-party provider and if a security breach happens your data is at high risk.
  
• Depends on the internet connection – even though this is the same case when running trading strategies by yourself, the cloud service provider can also suffer from downtime and other natural risks.
  
• Data confidentiality risk – your privacy is of utmost importance and when using a third-party cloud provider you can’t be too sure that you’re the only one accessing your data or being able to see it.

What is the Cloud Service?

A Cloud Service offers cloud computing as a service with an intent to provide affordable, easy and efficient access to various resources without the need to have your own hardware or infrastructure.

What is Cloud used for? 

Cloud services have many purposes for which they could be used for and here are some of the most common ones:

• Algorithmic trading – cloud services allow algorithmic traders to run multiple bots that trade the market.
  
• Big data storage – clouds offer an almost limitless storage capacity which is great when dealing with big data.
  
• Backup and recovery – clouds can be frequently backed up and you can easily recover data from them. They also offer good disaster recovery.
  
• Test and Development – cloud services are great when testing and developing your applications and trading strategies.
  
• Model training – when doing machine learning work, model training can take up to several months, and running it on a cloud can save you computing power and in some cases even money.

What cloud providers are good?

There are many cloud services and picking a quality one is important. In order to make your decisions process easier I will mention a few that are on top of their cloud game:

• Amazon Web Services (AWS)
• Microsoft Azure
• Google Cloud Platform
• Server Space
• IBM Cloud Services
• Oracle

What are Amazon Web Services (AWS)?

Amazon Web Services are cloud services that offer various features like computing power, database storage, content delivery, financial services, machine learning model training, and more.

Why should I use AWS?

• Easy to use
• Pay for the amount you use
• Affordable
• Trusted
• Has diverse services
• Offers unlimited server capacity

Why shouldn’t I use AWS?

• Resources limited by region
• Can be confusing for beginners
• Hard to move to another provider
• Has a steep learning curve
• Suffer from the cons of Cloud services

Kraken Bot

In this article, we will want to explore the AWS Lightsail cloud service that offers affordable servers on which we can run our trading strategies. The simple trading strategy that we will use will feature the Kraken exchange.

Before we set up a server and let the bot go crazy with trading, we need to explain how this strategy works and where you can learn how to code one.

The main idea behind the strategy will be to buy ETH when BTC moves 5% in the last 5 minutes. For this to work, we will pull BTC prices in 5 minute intervals and calculate the percentage difference.

If this difference is more or equal to 5% we will buy ETH, if the requirement isn’t reached we will continue to loop around. The strategy also features error management, logging, and environmental variables.

Error management is used to catch errors that might occur, logging is used so we can obtain data about our bot and what it’s been up to while running on the server and environmental variables allow us to interact with the bot.

How to sign up for an AWS server?

In order to sign up for an AWS server, you will first need to open an AWS account. So let us do that together. Go to the following link and in the upper right corner click the “Create an AWS Account” button.

How to pick a good AWS server location?

Picking a good AWS server location is important as you don’t want your trading strategies to lag. In order to make the right choice, we need to align the server location with the exchange location.

Sometimes there aren’t any servers in that particular exchange location, but our goal is to pick the closest one. There are several ways through which you can learn about the server location of your exchange or broker.

How to install Python and Git in an AWS Server?

Python and Git can be easily installed in an AWS server by utilizing the terminal commands. As our instance is based on Linux we will use the sudo yum install command that is the equivalent of the pip install command in Python.

In order to do this, we will open up our server instance by clicking the run button.

Now that we are in our terminal we will commence the installation of Python3 and Git. But first, let’s check if we have Python already installed in it.

python3 --version

Great! We have the latest version of Python3 installed. If you received that python3 wasn’t found you would need to write the following command:

sudo yum install python3

After that, you will agree for it to be downloaded and installed by passing a “y” letter when the terminal prompts you.

Now that Python is ready, it is time to turn our focus to Git. Git can be seen as a program that allows us to communicate with GitHub and send (push) or take (pull) code from it.

To install Git write the following command:

sudo yum install git

As you can see, on the last line the Terminal has prompted us to agree to the installation by passing the “y” letter. After you do that the installation will be complete and we will have our Git ready for use.

How to connect AWS to GitHub?

In order to connect your AWS instance to GitHub, you will need to utilize the SSH (Secure Shell) method and Git.

This process can be viewed as being consisted of 3 main steps:

1. Create your special “keys” (like the API and secret key)
2. Upload the public key to GitHub
3. Use the keys when interacting with GitHub

Let’s create the keys. Write the following command in your AWS instance terminal

ssh-keygen

Then give it a name and the passphrase can be left blank. I’ll name it “algotrading”.

Now that your key is created, you can find it by writing the following commands. The first one is ls that will list all the folders that are in your AWS instance.

here you can see our two keys. The “algotrading” key can be viewed as our secret while the “algotrading.pub” key is the public aka API key. For us to import the key to our GitHub we need to read it first.

To do this write the following command and copy the output:

cat algotrading.pub

Now it’s time to set this SSH key in our GitHub profile. Go over to your GitHub account -> Setting -> SSH and GPG keys.

When there, click on the green “New SSH key button”. Then give it a name and input the key. Then add the key and input your password when prompted. If you’ve done everything correctly the SSH should be ready.

Now we will instantiate the private key.

eval "$(ssh-agent)"

ssh-add algotrading

That’s it, our AWS instance is ready. The two code snippets above need to be typed in your Terminal every time you run the AWS instance. Now, let’s test it out by pulling our trading strategy from a private repository.

Go over to your repository on GitHub, click on the green “Code” dropdown button, select SSH and copy the URL.

Now write the following function in your AWS instance:

git clone <SSH-URL-HERE>

yes

We successfully cloned out the GitHub repository to our AWS instance. In order to confirm this write ls

How to install Python packages in an AWS Server?

Python packages can be installed in an AWS server by using the sudo pip3 install command. This can be tiresome if your code uses many dependencies so I’ll show you how to automate this process.

But as our code only uses the Kraken API let’s install it in our AWS instance with the following command:

sudo pip3 install pykrakenapi

If you have many dependencies (Libraries) you can utilize the pipreqs library that will export all the libraries your script is using into a .txt file that can be read by our AWS instance.

You can use this library in two ways, the first way would be to install the pipreqs library on your local machine and generate a .txt file there, upload it to GitHub, and then pull it to your AWS instance.

The second way is to install the library in your AWS instance and generate the .txt file there. I’m more of a fan of the second way as it is more efficient, so let’s go with it.

Type the following command to install the pipreqs library:

sudo pip3 install pipreqs

Then list all of the documents that are located in our AWS instance with ls and hone into the file you want the dependencies to be generated from. In our case, it is the Kraken-AWS file and we write this:

cd Kraken-AWS

Then create the requirements file:

pipreqs

As you can see we have our requirements.txt file inside of the Kraken-AWS folder. Now that we have it you can install the requirements with the following command:

sudo pip3 install -r requirements.txt

That’s it! Notice: To go back in your file hierarchy just type the following line that will take you a step back

cd ..

How to use environment variables?

In order to use the environment variables with in you AWS instance you will need to use the export command and state the values for those variables.

Let’s do it with our AWS instance and see how the code runs. We’ll start with a fresh AWS instance connection and go through all the steps.

eval "$(ssh-agent)"

ssh-add algotrading

cd Kraken-AWS

Now we can set up the environmental variables by typing export followed by the variable name and the value that we will assign it. In our case we have three environmental variables which are the following:

track_ticker = os.environ.get('ticker_to_track')
trade_ticker = os.environ.get('ticker_to_trade')
logname = os.environ.get('my_log_name') # Name of the saved log file

No we will assign them and launch our script:

export ticker_to_track='BTCUSD'
export ticker_to_trade='ETHUSD'
export my_log_name='log_kraken'

We have basically stated that we want to track BTC and trade ETH (as per our trade logic) and store the longs in a file named ‘log_kraken’. Now we can run the strategy with the following command:

python3 module1.py

And our script is operational and running. You can confirm this by the logs that are appearing. But how do we export these logs so that they can be read later on?

How to read and download logs from an AWS instance?

To read the logs from your AWS instance you can use the cat command followed by the name of your log file. To download the logs, you can push them to your GitHub repo from the AWS server instance.

Let’s check out the logs that are situated in the log_kraken file and read them with the following command:

cat log_kraken

Now, we will push this log file to our GitHub. For this be sure to be in the right repo. Then type the following command that will prepare the changes to be added to our GitHub repo:

git add .

After that let’s confirm the changes by commiting them:

git commit -m "Add Log files"

Now that the log files are ready we can push them to our GitHub with this command:

git push <SSH-URL-HERE>

As you can see, the log files has been added to our GitHub repo.

How to keep your script running on an AWS Server?

To keep your script running on an AWS Server Instance you will need to use the screen command that keeps running until you delete it. To create the screen you write the following command:

screen -S <screen-name>

When inside of the screen you can simply run your strategy as we did before. When you run it the screen will be running even when you close the Terminal and your PC. To detach from the screen press CTRL + AD.

How to run multiple scripts on an AWS Server?

You can also open multiple screens by using the screen creation command and run a custom script in each of them. Your screens can even have the same name.

If you want to see all the screen that you have you can type this:

screen -ls

If you want to delete your screens you can either delete them one by one or all at once. Here are the commands where the first one does the former and the second one does the latter:

# Type this while in the screen
exit
# To delete all screens
pkill screen

How to make a trading Telegraph bot?

Now that your strategy is running live. You want to be notified of significant events.

If you don’t do this, you are blind to whatever happens to your strategy and money! You should always know what is happening to your trading robot.

The usual significant events are:

• On strategy start-up and shut down.
• On trade entry or exit.
• On current positions and prices on selected assets every X hours. This ensures we know that the strategy is still running.
• When available margin drops before a certain level
• When volatility rises above a certain threshold
• When prices hit a certain level
• When prices move more than X% in the last Y minutes/hours/days
• Every place in Aiur where we have a logging message.

To do this, we use Telegram Chat Bots.

Step 1:

Search for ‘BotFather’ in the chat search bar.

Open a chat with the ‘BotFather’ from within Telegram.

Step 2:

Type in the command /newbot

It will prompt you to enter a bot name and send you an access token. You need this access token to use your bot. Don’t lose it or give it away!

Step 3

Search for your newly created bot

Open a new chat with your newly created bot. Click start or type something (this is important for Step 4).

Step 4

Go to the following URL – https://api.telegram.org/botxxxxxxxxxxxxxxxx/getUpdates – replacing the XXX with your access token. Here you should see a JSON structure if you had started a chat with your bot.

Note down the id (not to be confused with update_id or message_id). See picture below. Grab the number in the red rectangle.

Test your bot

At this point, the bot is created and messages can be sent to it. Here is a code snippet to test if everything is working:

import requests
def send(text):
	token = 'your_access_token_goes_here'
  	your_id = 'your_id_goes_here'
  	params = {'chat_id': your_id, 'text': text, 'parse_mode': 'HTML'}
  	resp = requests.post('https://api.telegram.org/bot{}/sendMessage'.format(token), params)
  	resp.raise_for_status()
send('Profit: $1,000,000,000')

Example on a notebook:

Make your bot private.

This means that no one else can chat with your bots.

Normally, bots created by BotFather are accessible to all.