Implementing the up-trendline indicator with Python — From acquiring data to modeling an algorithm and implementing a solution — Part 1 consuming a REST API

This is the first article in a series of articles about working with stock price data and implementing the up-trendline indicator with Python. The complete series will describe in detail implementation of the technical indicator called up-trendline. This article will describe the part of the solution which consumes the REST API, which will give us the data we will need in subsequent articles.

Requirements — You should be familiar with the following topics:

  • Having worked with python, objects, loops and data types.
  • Have some understanding about pandas and the requests library.
  • Know what technical analysis is.

Looking to refresh your skills or get an introduction to above topics? I can highly recommend following resources:

A brief explanation of the Up-Trendline indicator

In technical analysis the most well known and simple technical indicator is the up-trendiness analysis. This indicator is essentially a straight line and therefore relatively easy to recognize when looking at a graph. The pattern is a straight line which must go through at least three points. The lines slope must be increasing and must not be broken by a point that may be further ahead on the graph.

An uptrend line indicates; stock price will continue to be above the trend line, hence it will continue to rise. If at any time stock price hits the trend line, and as a result crosses the trend line; it signals the stock price will have a declining trend. See the illustration below.

If you keen to learn more about the indicator and in general about technical analysis. You will find more information here.

This is what we will build throughout this article

First and foremost you will need to install Python on your machine. I suggest installing Anaconda; in my opinion the easiest way to get working locally with python. But in this article I shall not go into the detail with setting up the developing environment. However, the matter will be subject in an exclusive article about installing Anaconda and setting up a development environment.

The rest of this article will focus on:

  • how to connect to Alphavantage REST API.
  • processing of data and placing it in a pandas data frame.
  • working with pandas data frame in order to prepare it for actual analysis.

Connecting to Alphavantage

First you need to have an API Key. This can be obtained by visiting the following link: https://www.alphavantage.co/ selecting ‘GET YOUR FREE API KEY TODAY’.

Here I assume you have installed Anaconda and created an environment with Python 2.7.xx installed.

You also have to install the following libraries in the environment you are using:

  • json
  • requests
  • pandas

Next create the .py files. Create following .py files inside your project folder:

  • get_historical_data.py
  • get_main_df.py

The file ‘get_historical_data.py’

First you need to import all necessary libraries we will be using in file get_historical_data.py.

import requests
import pandas as pd

 

We are then interested in creating a new class which will contain all necessary methods needed to connect to the REST API and retrieve the necessary data.

import requests
import pandas as pd


class AlphaVantage:

 

We will name the class AlphaVantage. This class needs a __init__ method, which will be called upon instantiation of the class. The __init__ method should look as follows:

import requests
import pandas as pd


class AlphaVantage:
    
    def __init__(self, symbol_code, interval="60min", outputsize="compact"):
        self.base_url = "https://www.alphavantage.co"
        self.default_endpoint = "/query"
        self.api_key = "<enter your API Key here>"
        self.symbol_code = symbol_code
        self.interval = interval
        self.outputsize = outputsize

 

Now I shall go through each line of code line by line. We already have created a class by defining it inline 5. In line 7 we define the __init__ method. This method will take three arguments where two of them will have a default value, meaning they will be optional when calling the class. In the __init__method there are six variables defined on between the lines 8 to 13.

  • self.base_url contains the value "https//www.alphavantage.co". This value will be used each time we need to build an url for calling the REST API.
  • self.default_endpoint contains the value "/query". This value is one of the endpoints of the REST API we will call multiple times. Therefore we want to put it in the __init__ method and thus avoid defining it multiple times.
  • self.api_key here you want to enter the API Key you retrieved from Alphavantage.
  • self.symbol_code the value for this variable is set by the parameter symbol_code, which we will be parsing when instantiating this class.
  • self.interval is set by the parameter interval, which has a default value set to "60min". This variable will later be used as a parameter when calling one of the endpoints provided by the REST API.
  • self.outputsize is a variable set by the parameter outputsize, which also has a default value set to "compact". This variable will also be used later as a parameter when calling one of the REST API endpoints.

Up to this point we have included all necessary libraries, defined a class and created our __init__ method within the class. Now we shall create an additional method called intraday inside the class AlphaVantage.

import requests
import pandas as pd


class AlphaVantage:

    .
    .
    .

    def intraday(self):
        parameters = {
            "function": "TIME_SERIES_INTRADAY",
            "symbol": self.symbol_code,
            "interval": self.interval,
            "outputsize": self.outputsize,
            "datatype": "json",
            "apikey": self.api_key
        }

        response_intraday = requests.get("{0}{1}".format(self.base_url, self.default_endpoint), params=parameters, stream=True).json()

        return response_intraday

 

The intraday method does not take any arguments except self. Ff you want to understand what self does in Python I suggest reading this article.

The intraday methods purpose is to call a specific endpoint with some parameters and return the response as JSON. For the purpose of calling the REST API we use the requests library, more specific we will use the requests.get() method. This method takes a number of parameters. We will provide the following:

  • The url we want to call. We build the url in line 21 specifically the part "{0}{1}".format(self.base_url, self.default_endpoint). As you can see we use some of the variables defined in the __init__ method.
  • We also have to provide parameters the REST API endpoint requires. For this we will build a dict called parameters, with some keys and values. The dict with its keys’ and values is defined in line 12 to 19. The keys’ and values in the dict, each represent a parameter the REST API endpoint requires in order to respond with the desired output.

Let’s go through some of the parameters defined in our dict called parameters.

  • "symbol" this keys’ value is set to variable self.symbol. We need to provide the endpoint with this parameter in order to specify for which company we want to see the stock price.
  • "interval" this keys’ value is set to the variable self.interval which has the value "60min". This means we want intraday data by the hour, resulting in 24 data points for each day.
  • "outputsize" this keys’ value is set to the variable self.outputsize which has the value "compact". In short this means we will receive the latest 100 data-points for any given "symbol" we make a request for.

If you want to learn more about this specific endpoint and in general about the Alphavantage REST API. I suggest taking a look at the documentation.

So now we have defined the method intraday, where we call a specific endpoint in the Alphavantage REST API. Furthermore, we have also defined all necessary parameters in a dict and provided the dict as a parameter to requests.get() method in line 21. Finally the last part of our method, line 23, returns the response we receive from the REST API we are calling.

Now we shell implement another method called daily in our class AlphaVantage.

import requests
import pandas as pd


class AlphaVantage:
    
    .
    .
    .

    def daily(self):
        parameters = {
            "function": "TIME_SERIES_DAILY",
            "symbol": self.symbol_code,
            "outputsize": self.outputsize,
            "datatype": "json",
            "apikey": self.api_key
        }

        response_daily = requests.get("{0}{1}".format(self.base_url, self.default_endpoint), params=parameters, stream=True).json()

        return response_daily

 

This method is very similar to the previous one. The daily methods’ purpose is to return stock price data on a daily basis instead of intraday. For that purpose there are two differences in this method compared to the previous. In line 13 the value for key "function" is now set to "TIME_SERIES_DAILY”, and we no longer have key "interval". Everything else stays the same and therefore no additional explanation is needed for this method.

The final get_historical_data.py file should now look as follows:

import requests
import pandas as pd


class AlphaVantage:

    def __init__(self, symbol_code, interval="60min", outputsize="compact"):
        self.base_url = "https://www.alphavantage.co"
        self.default_endpoint = "/query"
        self.api_key = "<enter your API Key here>"
        self.symbol_code = symbol_code
        self.interval = interval
        self.outputsize = outputsize

    def intraday(self):
        parameters = {
            "function": "TIME_SERIES_INTRADAY",
            "symbol": self.symbol_code,
            "interval": self.interval,
            "outputsize": self.outputsize,
            "datatype": "json",
            "apikey": self.api_key
        }

        response_intraday = requests.get("{0}{1}".format(self.base_url, self.default_endpoint), params=parameters, stream=True).json()

        return response_intraday

    def daily(self):
        parameters = {
            "function": "TIME_SERIES_DAILY",
            "symbol": self.symbol_code,
            "outputsize": self.outputsize,
            "datatype": "json",
            "apikey": self.api_key
        }

        response_daily = requests.get("{0}{1}".format(self.base_url, self.default_endpoint), params=parameters, stream=True).json()

        return response_daily

 

The file ‘get_main_df.py’

For this file you need to import the libraries pandas and json, in addition you also need to import the AlphaVantage class from file get_historical_data.py.

from get_historical_data import AlphaVantage
import pandas as pd
import json

 

We are now interested in creating a new function called get_main_data_frame. This function will at the moment contain all the logic we implement in the file get_main_df.py.

from get_historical_data import AlphaVantage
import pandas as pd
import json


def get_main_data_frame(symbol):

 

The first step we want to do in the function is call the AlphaVantage class in order to get some data we can then work with. We name this instance historical_data_daily.

from get_historical_data import AlphaVantage
import pandas as pd
import json


def get_main_data_frame(symbol):
    #
    # Get historical data as json
    #
    historical_data_daily = AlphaVantage(symbol).daily()

 

Notice that we also call the method daily() which is one of the methods implemented in class AlphaVantage. This means we will receive daily data and not intraday data.

The next code snippet contains a lot of code, however, the main focus is to iterate through the response received from calling the Alphavantage REST API.

from get_historical_data import AlphaVantage
import pandas as pd
import json


def get_main_data_frame(symbol):

    .
    .
    .

    #
    # Find and list all relevant keys in the original historical data response
    #
    list_keys = []
    list_historical_data__daily = []
    for key in historical_data_daily['Time Series (Daily)']:
        list_keys.append(key)

    for k in list_keys:
        #
        # Extract relevant data from original historical data response
        #
        data = historical_data_daily['Time Series (Daily)'][k]

        price_open = data['1. open']
        price_high = data['2. high']
        price_low = data['3. low']
        price_close = data['4. close']

        dict_data = dict([
            (u'date', k),
            (u'symbol', symbol),
            (u'price_open', price_open),
            (u'price_high', price_high),
            (u'price_low', price_low),
            (u'price_close', price_close),
        ])

        list_historical_data__daily.append(dict_data)

 

In line 15 and 16 two empty lists are defined. list_keys will contain all keys’ which are relevant for us and list_historical_data_daily will end up containing all data we want to work with. Here I am assuming that you have some basic knowledge working with dicts and JSON data.

Line 17 and 18 iterate through all keys’ found in key "Time Series (Daily)". Each key we find is appended to list_keys. In this context each key actually represents a date.

Lines 20 to 40 do couple of things. First we access the data in each key we have in list_keys and assign it to the variable data. Then four data points are identified price_openprice_highprice_low and price_close. All four variables will contain a numeric value representing a price for a given day.

Next a dict is defined with keys’. This will be the data we want to save for each day. The dict contains a date and a symbol, and the four price values. Finally the dict is appended to list_historical_data_daily.

The reminder of the code which can be seen in next code snippet is focusing on working with pandas in order to prepare the data for further analysis.

from get_historical_data import AlphaVantage
import pandas as pd
import json


def get_main_data_frame(symbol):

    .
    .
    .

    #
    # convert data to data frame
    #
    df = pd.read_json(json.dumps(list_historical_data__daily))

    #
    # prepare data, and add calculated fields to data frame
    #
    df['date_str'] = df['date']
    df.set_index('date', inplace=True)
    df.sort_index(inplace=True)

    df['price_close_lag'] = df['price_close'].shift(1)
    df['price_close_lead'] = df['price_close'].shift(-1)

    df.insert(0, 'date_id', range(1, 1 + len(df)))

    #
    # returns data frame
    # 
    return df

 

The first thing we want to do is take all our data stored in list_historical_data_daily and convert it to a pandas data frame. This is done in line 15.

  • Line 20 creates the field 'date_str'. This field is derived from ‘date’ field.
  • Line 21 sets field 'date' to be the new index of the data frame.
  • Line 22 sorts the index. The default sorting is ascending meaning the oldest date will be first.
  • Line 24 creates the field 'price_close_lag'. This field shows close price data from previous day. This will be useful when we in a subsequent article want to compare today’s price with the day before price, in order to determine the direction of the price.
  • Line 25 creates the field 'price_close_lag'. This field shows close price data from tomorrow. Like the previous field this field will also be of interest for later analysis in terms of comparing the prices.
  • Line 27 creates the field 'date_id'. This field increments an int value starting from 1. It is important that the pandas data frame is sorted ascending. Therefore, previously the index, which is a date value is sorted ascending.

Finally the data frame called df is returned.

The final get_main_file.py file should now look like following:

from get_historical_data import AlphaVantage
import pandas as pd
import json


def get_main_data_frame(symbol):
    #
    # Get historical data as json
    #
    historical_data_daily = AlphaVantage(symbol).daily()

    #
    # Find and list all relevant keys in the original historical data response
    #
    list_keys = []
    list_historical_data__daily = []
    for key in historical_data_daily['Time Series (Daily)']:
        list_keys.append(key)

    for k in list_keys:
        #
        # Extract relevant data from original historical data response
        #
        data = historical_data_daily['Time Series (Daily)'][k]

        price_open = data['1. open']
        price_high = data['2. high']
        price_low = data['3. low']
        price_close = data['4. close']

        dict_data = dict([
            (u'date', k),
            (u'symbol', symbol),
            (u'price_open', price_open),
            (u'price_high', price_high),
            (u'price_low', price_low),
            (u'price_close', price_close),
        ])

        list_historical_data__daily.append(dict_data)

    #
    # convert data to data frame
    #
    df = pd.read_json(json.dumps(list_historical_data__daily))

    #
    # prepare data, and add calculated fields to data frame
    #
    df['date_str'] = df['date']
    df.set_index('date', inplace=True)
    df.sort_index(inplace=True)

    df['price_close_lag'] = df['price_close'].shift(1)
    df['price_close_lead'] = df['price_close'].shift(-1)

    df.insert(0, 'date_id', range(1, 1 + len(df)))

    #
    # returns data frame
    # 
    return df

 

Hopefully, this article can give some ideas on how to start working with stock data, where to get data, and how to start preparing it in order to apply algorithms.

The next article PART 2 will focus on implementing the technical indicator up-trendline. I hope you are up for it, as it will be a bit more complicated but on the contrary more exciting.

Leave a Reply

Your email address will not be published. Required fields are marked *