The Problem
Elastic Beanstalk is a nice and easy way to get up and running with a scalable and highly available cloud platform but it has one very annoying down side. The total size of all your environment variables combined cannot exceed 4kb. I ran into this limit and had to find a solution.
Prior and Attempted Solutions
First we tried to minimize what we stored there. Then I started putting new configurations in the SSM Parameter Store and referencing them in Elastic Beanstalk with the following.
{{resolve:ssm:/EB/SFCS}}This was a good temporary fix as you could reduce the size of the variable down since we were only storing the Parameter Store key in Elastic Beanstalk. Eventually, however, we would run out of space with that approach as well.
I tried one approach utilizing Elastic Beanstalk platform hooks. I ran across a post that gave me reason to believe that I could create a post deploy platform hook that would append the parameter to the following file with the expectation that it would from there be available as an environment variable. This, however, did not work for me.
/opt/elasticbeanstalk/deployment/env My Current Solution Overview
I ended up writing code to pull SSM Parameter Store variables directly into my Laravel app’s config. I created a Configuration Helper class that is responsible for calling SSM Parameter Store directly by specifying a Parameter Path thereby pulling in an array of parameters that I would later map into the Laravel config. The solution relies on a naming convention in SSM Parameter Store that must match up with the Laravel configuration path name.
I created a helper class that calls the SSM Parameter Store directly and maps the remote parameters to my Laravel parameters. I decided not to use this solution for secure string parameters for now for a couple reasons I’ll get to later.
This solution relies on a naming convention in the Parameter Store that must match up with the Laravel configuration path name.
SSM Example: In this example, ‘Direct’ is an arbitrary key word I use to denote the params that will be pulled into the app directly and ‘Stage’ is the environment.
/EB/Direct/Stage/Services/CrmService/ApiUrlLaravel Example: In Laravel, I would have a config in /config/services.php like the following
return [
'crm_service' => [
'api_url' => ''
]
];The config must be declared as a Laravel config, but not set. This way you have to be intentional about what SSM variables get pulled into your Laravel app.
Result: I can retrieve my config as follows.
$apiUrl = config('services.crm_service.api_url');The Helper Class
I’ll break down the parts of the Configuration Helper class I wrote. Below this section I have the complete class if you want to skip my explanations.
- First the construct gets and sets the environment variable and calls function set configs
public function __construct(){
$this->env = ucfirst(strtolower(env('APP_ENV')));
$this->setConfigs();
}- Next , the set configs method will first check to see if the configs have been stored in cache. If they are in cache, it returns them from cache, otherwise it retrieves them from SSM with getRemoteConfigs. Once the remoteConfigs property is set, it calls setLocalConfigs which will map the parameters returned from SSM with their local match.
public function setConfigs() {
$cacheKey = 'remoteConfigs';
$this->remoteConfigs = Cache::remember($cacheKey, now()->addMinutes(1), function(){
return $this->getRemoteConfigs();
});
// Now that the remote configs are loaded set the local configs (configs in Laravel)
$this->setLocalConfigs();
}- Method getRemoteConfigs uses the AWS SDK for PHP to retrieve the parameters by path. You must pass in parameter ‘Recursive’ = true if you want to get more than just the next directory level, and if you want to decrypt securely stored parameters from SSM, you can pass WithDecryption = true. I am not retrieving secure strings in this way however because I don’t want to store un-encrypted secrets in cache and because KMS decryption is not free, it would probably get expensive to not use caching.
protected function getRemoteConfigs()
{
$client = new SsmClient([
'version' => 'latest',
'region' => 'us-east-1'
]);
$vgParameters = $client->getParametersByPath([
'Path' => "/EB/Direct/{$this->env}/",
//'WithDecryption' => true,
'Recursive' => true
]);
return $vgParameters['Parameters'];
}- Finally, setLocalConfigs loops through the retrieved parameters, reformats them to what their Laravel config equivalent would be, and then looks to see if there is a config with that name.
config()->has($configString)will be true if there is a config with that name. Also, I found and added a helper function to convert the camel case format of my Parameter Store parameters to the underscore format of my Laravel config.
protected function setLocalConfigs()
{
$toSet = [];
foreach($this->remoteConfigs as $rc){
$name = $rc['Name'];
$value = $rc['Value'];
// construct what the config name should be following the convention
$nameParts = array_filter(explode('/', $name));
// Find the one after the environment
// take only the relevant parts
$ignoreParts = ['EB', 'Direct', $this->env];
$configStringParts = [];
foreach($nameParts as $part) {
if(in_array($part, $ignoreParts)) {
continue;
}
$formattedStringPart = $this->formatConfigStringPart($part);
$configStringParts[] = $formattedStringPart;
}
if(!empty($configStringParts)){
// check if there is a config with that name
// and if so, set it with the value
$configString = implode('.', $configStringParts);
if(config()->has($configString)){
$toSet[$configString] = $value;
}
}
// otherwise continue to the next remote parameter
}
if(!empty($toSet)){
config($toSet);
}
}Complete Solution
- Below is the complete ‘Configuration Helper’ that is responsible for: retrieving remote SSM parameters, mapping those parameters to Laravel configurations, using caching to reduce the number of calls to SSM. It is free to access SSM parameters, however it would add a small delay to the request time and slow things down a little.
<?php
namespace App\Helpers;
use Aws\AwsClient;
use Aws\Ssm\SsmClient;
use Illuminate\Support\Facades\Cache;
/**
* Only to be used with non-secure strings from SSM Parameter Store
* Using with secure strings will use KMS encryption services on every requests and could potentially be expensive
* AND do not want to store unsecured keys in cache
*
* Class ConfigurationHelper
* @package App\Helpers
*/
class ConfigurationHelper
{
protected $remoteConfigs;
protected $env;
public function __construct(){
$this->env = ucfirst(strtolower(env('APP_ENV')));
$this->setConfigs();
}
public function setConfigs() {
$cacheKey = 'remoteConfigs';
$this->remoteConfigs = Cache::remember($cacheKey, now()->addMinutes(1), function(){
return $this->getRemoteConfigs();
});
// Now that the remote configs are loaded set the local configs (configs in Laravel)
$this->setLocalConfigs();
}
protected function getRemoteConfigs()
{
$client = new SsmClient([
'version' => 'latest',
'region' => 'us-east-1'
]);
$vgParameters = $client->getParametersByPath([
'Path' => "/EB/Direct/{$this->env}/",
//'WithDecryption' => true,
'Recursive' => true
]);
return $vgParameters['Parameters'];
}
protected function setLocalConfigs()
{
$toSet = [];
foreach($this->remoteConfigs as $rc){
$name = $rc['Name'];
$value = $rc['Value'];
// construct what the config name should be following the convention
$nameParts = array_filter(explode('/', $name));
// Find the one after the environment
// take only the relevant parts
$ignoreParts = ['EB', 'Direct', $this->env];
$configStringParts = [];
foreach($nameParts as $part) {
if(in_array($part, $ignoreParts)) {
continue;
}
$formattedStringPart = $this->formatConfigStringPart($part);
$configStringParts[] = $formattedStringPart;
}
if(!empty($configStringParts)){
// check if there is a config with that name
// and if so, set it with the value
$configString = implode('.', $configStringParts);
if(config()->has($configString)){
$toSet[$configString] = $value;
}
}
// otherwise continue to the next remote parameter
}
if(!empty($toSet)){
config($toSet);
}
}
/**
* Convert from camel case / proper case to underscores
*
* @param $input
* @return string
*/
protected function formatConfigStringPart($input)
{
$pattern = '!([A-Z][A-Z0-9]*(?=$|[A-Z][a-z0-9])|[A-Za-z][a-z0-9]+)!';
preg_match_all($pattern, $input, $matches);
$ret = $matches[0];
foreach ($ret as &$match) {
$match = $match == strtoupper($match) ?
strtolower($match) :
lcfirst($match);
}
return implode('_', $ret);
}
}- Instantiate this helper early on in the bootstrap process so the configs will be available as early as possible. Since the construct handles everything, the class only needs to be instantiated for the parameters to be pulled in. I decided to call it from the AppServiceProvider.php which worked for me.
<?php
namespace App\Providers;
use App\Helpers\ConfigurationHelper;
use Illuminate\Support\ServiceProvider;
class AppServiceProvider extends ServiceProvider
{
/**
* Bootstrap any application services.
*
* @return void
*/
public function boot()
{
$this->app->singleton(ConfigurationHelper::class, function ($app) {
return new ConfigurationHelper();
});
$this->app->make(ConfigurationHelper::class);
}
/**
* Register any application services.
*
* @return void
*/
public function register()
{
}
}- Go into SSM parameter store and set a parameter that adheres to the naming convention
/EB/Direct/Stage/Services/CrmService/ApiUrl- Add the config to your Laravel config without a value. The following is in my /config/services.php
<?php
// /config/services.php
return [
'crm_service' => [
'api_url' => ''
]
];- Access my config like so
$apiUrl = config('services.crm_service.api_url');