A visit to config files.

Well we have some rudimentary tests, with no test framework.  However before we do any more work we need a place to store all those application keys.

The problem is that I almost committed the application keys to my github account, and once uploaded they're there forever!  Of course I would generate new keys, but I could do it again.  So the best thing to do right away is to find a home for these keys that are located outside the git repo, or put in the gitignore file.

When one of my former coworkers, a dev saw how I was storing all these values he was very quick to mention that if this was a production web service all those values would be stored in environment variables.  Here's the thing... a web service is running 1 instance of a particular service.  The api tests were testing against to 10 microservices, and 4 environments (dev, qa, beta, prod).  That makes 40 sets of configuration settings we need to keep track of.  In our example we would at least store 4 keys, and the twitter api url.  That would start getting messy.  So I devised in my config system a way to encrypt access keys, with a master key being stored somewhere else.  Much easier to keep track of one key than 160!

We'll revisit the config system with encryption and all that later.  For now all I want is a config file, and a way to read it.  A way for specifying the config file name on the command line would be nice.

1. First I created a directory for my config files.  If you are testing multiple systems in multiple environments these things will get out of hand quick.
1. mkdir ConfigFiles
2. Then I made a config file.  I used configparser.  It's built in, and I enjoyed the nostalgia of writing .ini files again (And the multiple sections have been useful on many occasions). I started by making one for me, but here is the sample:



[common]  
consumer_key = **your key here**  
consumer_secret = **your key here**  
access_token = **your key here**  
access_token_secret = **your key here**  

[env-qa]  
twitter_api_base_url = https://api.twitter.com  



3. I then made  a Config class to read and store the data.

 
 import configparser  
 import os  
 import sys 

 
 class Config:  
   CONFIG_DIR = 'ConfigFiles'  
   COMMON_CONFIG_SECTION = 'common'  
   ENV_CONFIG_SECTION = 'env-qa'  

   config_file_name = None  

   consumer_key = None  
   consumer_secret = None  
   access_token = None  
   access_token_secret = None  

   twitter_api_base_url = None  

   def __init__(self, config_file_name):  
     try_path = os.path.join(self.current_dir, config_file_name)  
     if os.path.exists(try_path):  
       self.config_file_name = try_path  
     else:  
       self.config_file_name = os.path.join(self.current_dir, self.CONFIG_DIR, config_file_name)  
     if not os.path.exists(self.config_file_name):  
       print("\n\nFatal Error: Config file: {} not found.".format(self.config_file_name))  
       sys.exit()  

   def read_config(self):  
     config = configparser.ConfigParser()  
     config.read(self.config_file_name)  
     self.consumer_key = config[self.COMMON_CONFIG_SECTION]['consumer_key']  
     self.consumer_secret = config[self.COMMON_CONFIG_SECTION]['consumer_secret']  
     self.access_token = config[self.COMMON_CONFIG_SECTION]['access_token']  
     self.access_token_secret = config[self.COMMON_CONFIG_SECTION]['access_token_secret']  
     self.twitter_api_base_url = config[self.ENV_CONFIG_SECTION]['twitter_api_base_url']  
     print("Twitter api base url = {}".format(self.twitter_api_base_url))  

   @property  
   def current_dir(self):  
     return os.path.dirname(os.path.abspath(__file__))  



4. Did you know pytest will support additional command line options?  And it's pretty easy.  You need to make a conftest file (conftest is a special python file pytest can use.  It will run, and allow you to do lots of cool things, one of which is add command line options to your tests).  
1. put conftest in the root of your project.



 def pytest_addoption(parser):  
   parser.addoption("--config", action="store", default='local',  
            help="specify the config file to use for tests")  


Note that I believe the function name is important. Don't try to name it anything else.

5. Now we can modify test_twitter_crud.py to reflect using the config file
1. You will notice the constants at the top of the file are gone.  They have been moved into the config file.
2. We need to initialize the Config class by retrieving the config file name from the command line options, and then passing it into the Config class.
3. Notice where we were once using constants we are now using "self.config.<config setting>"
4. We also added the base url for twitter to our url calls to the api.  This looks a little awkward here, but again, we haven't really made a framework yet.  In the next steps this will look much cleaner.



 import random  
 import string  
 import urllib  
 from urllib.parse import urlencode  

 import oauth2  
 import pytest  

 from Config import Config  


 class TestTwitterCRUD:  
   config_file_name = None  
   config = None  

   def setup_class(self):  
     self.config_file_name = pytest.config.getoption('config')  
     print("\nconfig file is {}".format(self.config_file_name))  
     self.config = Config(self.config_file_name)  
     self.config.read_config()  

   def test_get_timeline(self):  
     home_timeline = self.oauth_req('{}/1.1/statuses/home_timeline.json'.format(self.config.twitter_api_base_url))  
     print(home_timeline)  
     assert home_timeline is not None  

   def test_post_timeline(self):  
     status = "Test Status {}".format(self.make_random_string(6))  
     payload = "status={}".format(urllib.parse.quote(status))  
     response = self.oauth_req('{}/1.1/statuses/update.json'.format(self.config.twitter_api_base_url),  
                  http_method="POST", post_body=payload)  
     print(response)  
     assert response is not None  
     assert status in str(response)  

   def oauth_req(self, url, http_method="GET", post_body="", http_headers=None):  
     consumer = oauth2.Consumer(key=self.config.consumer_key, secret=self.config.consumer_secret)  
     token = oauth2.Token(key=self.config.access_token, secret=self.config.access_token_secret)  
     client = oauth2.Client(consumer, token)  
     resp, content = client.request( url, method=http_method, body=post_body, headers=http_headers )  
     return content  

   def make_random_string(self, num_chars):  
     return ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(num_chars))  



6. Lastly, I tried to check in my files, and my config file with my keys are in the commit list!  edit your .gitignore file, and exclude your config file.  I named mine "sams_config", so excluding anything beginning with "sams" will save us from future trouble.  You can then create multiple config files, they just just need to start with your name.  Just be sure your "prefix" is unique enough to only catch your config file, not other things (if I had used just "sam" a file called "sample" would also be excluded)! 
1. Add this somewhere in your .gitignore: "**/<your config file prefix>*"
7. As your tests grow you will outgrow having one config file for each environment.  Especially if you have microservices, so you need to test many services.  At my last contract the "Config" code I felt was the most messy, simply because of the number of options I had, and there were different ways to run our tests depending if it was being run by the CI system or not.  Here are some ideas that will allow you to grow.
8. Keep command line options to a minimum.  Really just keep it down to specifying a config file, and maybe test environment, like dev, qa, stage, prod.
9. When I had a lot of microservices to test I wound up modifying ALL my config files to add a new service url.  What I wound up doing is creating a config file with sections for each environment being tested, including local.  I then had settings for each service url in each section per ernvironment.  I was then only editing 1 file.
10. If your urls have a standard naming convention, like say <service>-<environment>.yourdomain.com, you may be able to programmaticly derive the service url.
11. I also made a config file with just api keys.  Often the api keys would vary by environment as well.  As mentioned we'll cover ways to encrypt api keys and other data in a later post.
12. Have settings that don't change, but still need to be in a config file?  Create a common.ini file for them.
13. I also used my Config class to store some important constants in my test suite, like say the logger id used throughout the suite. 

The files for this post have been committed to: https://github.com/lightmanca/TheTestFrameworkBlog/tree/twitter_tests_with_config_file