Managing content delivery across multiple AWS Elemental MediaPackage origins

Traditional video on demand (VOD) workflows are often complex, with requirements to manage multiple content versions, packaging configurations and digital rights management (DRM) needs. These requirements often lead to complex transcoding pipelines, duplication of assets, and inflexible distribution configurations. Companies are starting to look to modernize their VOD workflows to increase agility, cost savings, and flexibility. At the same time, there are also opportunities to reduce complexity at the content delivery layer while still ensuring video reaches customers per their selected preferences, serve a fragmented device market, and improve monetization options.

AWS Elemental MediaPackage can be used as an origin for VOD libraries. It reduces the complexity of traditional VOD origin solutions by providing just-in-time packaging and integration with third-party DRM providers through the Secure Packager and Encoder Key Exchange (SPEKE) protocol.

In this post, we discuss some best practices for configuring MediaPackage to support large VOD catalogs with complex requirements. Circumstances may arise when you need to split your catalog over multiple MediaPackage VOD origins, which could potentially add complexity to the content delivery network (CDN)  configuration.

We also go over how to optimally configure MediaPackage and Amazon CloudFront to maintain simplicity in the configuration of your downstream clients and overall platform management.

Scenarios for multiple MediaPackage origins

Let’s examine two scenarios that can lead to a need to create multiple MediaPackage origins:

1. Service quotas

In situations where a large asset library (several thousands) needs to be ingested, we may need a pool of MediaPackage origins with similar configurations. The default soft limit for ‘Maximum Assets per Packaging Group’ is 10,000 assets per packaging group.

When a content library exceeds this limit, we suggest you create multiple Packaging Groups within MediaPackage, and ingest assets across the groups using an algorithm to balance the assets between the pool in a way that is meaningful for your use case.

You might start with a simple round-robin algorithm, ingesting each asset into the Packaging Group, which has the least number of assets ingested. Alternative approaches could be to base this routing decision on other business metadata such as cost center, content category, or specific packaging or DRM needs. What is important is to determine an algorithm that allows you to remain within service limits as your catalog grows over time.

To learn more about the service level quotas of MediaPackage applicable for VOD content refer to

https://docs.aws.amazon.com/mediapackage/latest/ug/limits-vod.html

2. DRM requirements

There could be situations when you need to encrypt assets with different DRM vendors, or where the requirements of packaging are specific to certain assets. This may be due to content rights requirements, product decisions or device compatibility. In these cases, it is a requirement to create a new MediaPackage packaging group for each set of distinct needs.

If your use case aligns with either of the preceding scenarios, you may find the resulting set of multiple MediaPackage origins presents a challenge when provisioning your CDN and configuring downstream clients. Let’s explore some options to address this when using Amazon CloudFront as your CDN.

A simple solution

When setting up a CDN in front of your MediaPackage origin, the simplest option is to deploy a distinct CloudFront distribution following a 1:1 mapping with each MediaPackage origin domain, which is obtained when you create the packaging group configuration. This configuration adds complexity to your downstream client applications, as assets may be delivered via multiple distinct domain names. So, if your configuration consists of more than a small number of origins, feel free to skip forward to the ‘A Unified Distribution’ section where we demonstrate how to serve content from a single, unified, CloudFront distribution.

Use the following steps to configure a distinct CloudFront distribution for each MediaPackage origin domain.

Step 1

Create the MediaPackage packaging group with the necessary configuration and note the ‘Domain name’ created. This is used as the origin for CloudFront distribution. The configuration will depend on the streaming formats you would like to support and is documented here https://docs.aws.amazon.com/mediapackage/latest/ug/pkg-group.html

 

Step 2

Create and configure a CloudFront distribution

1. Sign in to the AWS Management Console and open the CloudFront console at https://console.aws.amazon.com/cloudfront/v3/home

2. Choose Create Distribution.

3. Choose the settings for the distribution, including the following:

Origin Domain
The origin where your MediaPackage VOD content asset exists. This is the ‘Domain name’ we obtained from Step 1. Enter the value in this field excluding the protocol.

Protocol
Choose HTTPS only.

Viewer Protocol Policy
Choose Redirect HTTP to HTTPS.

Depending on your technical or business needs you can configure other settings. For a list of all the options for distributions and information about setting them, see Values That You Specify When You Create or Update a Distribution.

When you finish choosing the other distribution settings, choose Create Distribution.

4. If you need to support Microsoft Smooth Streaming let’s navigate to the CloudFront distribution we just created and navigate to the ‘Behaviors’ Then we choose ‘Create behavior’ and in the settings section, enter ‘Path Pattern’ with ‘index.ism/*’, select the origin domain obtained earlier under ‘Origin and origin groups’ and enable ‘Smooth Streaming’ under the ‘Additional settings’.

5. Optionally, you can enable CloudFront Origin Shield, which is an additional layer in the CloudFront caching infrastructure that helps to minimize your origin’s load, improve its availability, and reduce its operating costs. It is beneficial in this use case where MediaPackage provides just-in-time packaging for video streaming and involves an origination cost.

You enable Origin Shield by picking the AWS Region that has the least latency to your origin. To learn more on the benefits of Origin Shield feature, cost and the factors to consider while choosing the Origin Shield Region refer to https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/origin-shield.html

Once done it should look similar to the following:

Now we should be able to test the delivery of some assets through CloudFront using the default domain created by the distribution, which will be in the form ‘dxxxxxx.cloudfront.net’. For example, if the MediaPackage playback URL for your VOD asset is

https://b88ba461ce0ab366e731a60cf890ff91.egress.mediapackage-vod.eu-west-1.amazonaws.com/out/v1/28c17a078f2e425fad063cadac77db0e/b6e6022d40604f1188ebf6e35ca3c564/c817848e01aa480e8b59afa87efef9e2/index.m3u8

Your playback URL from CloudFront would be:

https://dxxxxxx.cloudfront.net/out/v1/28c17a078f2e425fad063cadac77db0e/b6e6022d40604f1188ebf6e35ca3c564/c817848e01aa480e8b59afa87efef9e2/index.m3u8

Though this approach is straight forward to set up, it presents complexity downstream in client applications, whereby content may be streamed from many domains, as well as adding complexity in the management of many CloudFront distributions.

A Unified Distribution

Alternatively, we could leverage CloudFront’s ability to support multiple origins, configuring a single CloudFront distribution with different cache behaviors corresponding to path patterns of each origin. The following is a simplified representation:

To achieve this, we need to identify the unique path pattern from the playback URL of one of the ingested assets from each MediaPackage packaging group.

For example, the following are HLS playback URLs of two assets,

https://b88ba461ce0ab366e731a60cf890ff91.egress.mediapackage-vod.eu-west-1.amazonaws.com/out/v1/28c17a078f2e425fad063cadac77db0e/b6e6022d40604f1188ebf6e35ca3c564/c817848e01aa480e8b59afa87efef9e2/index.m3u8

https://b88ba461ce0ab366e731a60cf890ff91.egress.mediapackage-vod.eu-west-1.amazonaws.com/out/v1/09c2c37a22a74808bc7701e888f38d4d/b6e6022d40604f1188ebf6e35ca3c564/c817848e01aa480e8b59afa87efef9e2/index.m3u8

If we generalize the URL pattern, it would be

https://packaging-group-id.egress.mediapackage-vod.region.amazonaws.com/out/v1/asset-guid/pkg-config-guid/manifest-guid/index.m3u8

So the common path pattern between all assets, which are ingested within a packaging config, can be generalized as

‘/out/v1/*/pkg-config-guid/*’

In the above example, this translates to:

‘/out/v1/*/b6e6022d40604f1188ebf6e35ca3c564/*’

These unique path patterns are what we need to define as behavior in CloudFront distribution.

So, for example, if we have 10 different packaging groups each with 3 packaging configurations for HLS, DASH, MSS then we will need 10 x 3 = 30 Cache Behavior and 10 Origin in CloudFront configuration. Each behavior will be mapped to one of the Origin serving a particular asset’s packaging format.

Note: In cases where you need more than 25 cache behavior within CloudFront you will need to submit a limit increase for ‘Cache behavior per distribution’ from the AWS Support center. More details on current CloudFront limits can be found at https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-limits.html

Automation (optional)

To ease the setup, we have published a solution to automate the deployment of this configuration. You can access the solution via this GitHub repository. The solution makes use of a sample AWS Lambda function that expects as input a CloudFront distribution ID (to which these definitions need to be added), an optional Origin Shield Region and a comma-separated list of packaging groups. The Lambda function infers the unique path patterns and origin definitions and updates them in the supplied CloudFront distribution. Follow the steps mentioned in the ‘Build’ section of the README file available in the GitHub repository before proceeding.

Pre-requisites

• Multiple packaging groups defined within AWS Elemental MediaPackage for VOD. Please note the packaging group IDs.
• At least one asset ingested into each of these packaging groups.
• A CloudFront distribution. Please note the CloudFront distribution ID.
• An optional AWS Region code, which needs to be defined as Origin Shield Region.

Test

1. Once you deploy the CloudFormation template, go to AWS Lambda console, and into the function name starting with ‘{STACK-NAME}-AEMPCloudFrontSync…’
2. Choose ‘Test’ tab, enter a name and following payload and choose ‘Save changes‘ to create a test event.

{

"DistributionId": "CLOUDFRONT DISTRIBUTION ID",

"PackagingGroups": "COMMA SEPARATE LIST OF VOD PACKAGING IDs",

"OriginShieldRegion": "AWS REGION CODE"

}

Example:

{

“DistributionId”: “E2UZBZ6X6T11VU”,

“PackagingGroups”: “VODWorkflow-packaging-group,me0010-vodjitp8-f7fb0b21d693-vod-package”,

“OriginShieldRegion”: “eu-west-1”

}

For a list of AWS Regions which support Origin Shield please refer to https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/origin-shield.html#choose-origin-shield-region

3. Press the ‘Test’ The function creates a unique list of path patterns associated with each MediaPackage origin for each packaging group and then creates the necessary origin and behavior on CloudFront (as shown in the following screen capture).

CloudFront Behavior:

Origin:

4. Once CloudFront distribution status is ‘Deployed’, review and test with some assets using the CloudFront domain name as described earlier.

Note: if there is already a definition for origin or cache behavior definition matching a path pattern, then the Lambda function does not modify it.

To summarize, we explored scenarios where you could potentially have a number of MediaPackage VOD origins. We looked into the details to understand the URL patterns and how it is designed for assets being served from MediaPackage VOD origin. Lastly, we covered ways to set up a single CloudFront distribution to cater to one or more MediaPackage VOD origins.

If you have questions, feedback, or would like to get involved in discussions with other community members, visit the AWS Developer Forums: Media Services.