Element451Integrations Guide

Technical Overview

Updated Article

Introduction#

For a functional overview of how the integration works, see Functional Overview.

The Element451 integration with CFNC is designed to be simple and automatic from a user perspective, while handling complex data processing behind the scenes.

Element451 has built a baseline integration that interacts with CFNC’s API endpoints and syncs their application data (including biographic/demographic, previous school history, emergency contacts, etc) as well as an application PDF and transcripts associated with the application/student. While a lot of data is brought in, there may be additional properties your integration is configured to bring in based upon your SOW.  Additionally, there can be multiple types of applications synced in from CFNC into Element depending upon a partner’s needs.  This guide assumes the traditional undergraduate application is being synced, but additional application types can be done and would follow the same pattern.

Scope of This Guide#

This section provides a high-level technical overview of what occurs after the integration process is by the various sync schedules (see Functional Overview).

Primary Tool#

Element451 uses Dagster to orchestrate all integration processes.

Dagster manages the following sync types:

  • CFNC Application Data → Element451
  • CFNC Application PDFs → Element451
  • CFNC Transcripts → Element451

How It Works#

  • Dagster runs a series of jobs and sensors to monitor and execute integration activity.
  • Within each sync process, multiple operations (ops) are performed to:
    • Extract data
    • Transform data
    • Send data to Element451

Each sync is composed of these coordinated steps to ensure data is processed and transmitted correctly across systems.

CFNC Application Data → Element451#

Overview#

The loading of application/demographic/contact data happens in two main stages:

  • Extract CFNC application data to a staging database table
  • Transform and load CFNC application data into Element

These two jobs are duplicated for each app type (so there are two jobs for traditional undegraduate applications, two jobs for CCP apps, and so on).

Extract CFNC Application → Staging#

By default, this job runs every hour and looks at the date of the last imported application and queries for any new applications from that datetime forward.  By default it grabs both submitted and started applications (but it can be configured to only pull submitted applications if desired).

It retrieves the applications from CFNC’s API using one of the following routes:

…and passes in the start/end date times (start date being what was defined above and the end date being the current date/time).

The raw data returned from these API fetches is stored in our database staging table.

Transform CFNC Application → Element451#

By default, this job runs 10 minutes after every hour (but can be configured as a sensor to trigger faster if the previous job is running on a faster schedule).  It’s first action is to retrieve the relevant data out of the staging table and it starts building it into an object that will eventually be tweaked for loading into Element.

After the relevant data has been extracted, a series of operations begin to manipulate the data into what Element needs.

Translate Application Data#

This operation translates many of the common application fields that are data source based in Element,  into Element friendly values.  These translations are done using translation tables that are setup as a part of the initial build process our developers run through.  The data that is translated here includes:

  • Citizenship Status
  • County
  • Gender
  • Major
  • Race
  • Student Type
  • Term

Translate Custom Data#

After standard data has been translated, any custom fields are translated.  These fields can vary by partner, but would typically include:

  • Academic Load
  • Educational Levels (for parents and student)
  • Employment Status

Prepare Common Import Parts#

This operation takes the data that has been extracted along with the translated information and begins putting it into a payload object for upload into Element.  It starts building a template of the data/values to be imported using Element API’s: /v2/users/import route.  Specifically this operations handles many of the common baseline fields in most Community College CFNC integrations.  This includes most demographic/contact information, application specific data, previous schools, and identifiers. It also adds milestones to be imported/set in Element at this stage.

Prepare Custom Import Parts#

The last preparation operation is to prepare any custom import parts.  This works very similar to the common import parts except it is appending other data that has been flagged as being custom/more unique per partner.  The specifics can vary by partner, but generally it includes things like:

  • Academic Load
  • Educational Goals
  • Educational Level
  • Employment Information
  • Military Information
  • Suspended/Expelled
  • etc

This data is appended to the payload template’s object/structure that was started in the previous Prepare Common Imports Part section.

Import into Element451#

After the above steps are done, the application data is then imported into Element using the /v2/users/import route in Element451’s API.

The success of the import (or any errors from any of the above steps) are logged in the integration log for review.

CFNC Application PDFs → Element451#

By default, this job runs 30 minutes after every hour.  It starts by querying all CFNC applications (of the CFNC app type the job is running for) imported in Element that are submitted and have a hidden internal property of “cfncapppreviewprocessed” not set to true.

Using that list of students/applications from Element, it will fetch the application previews from CFNC’s API using an Export Project ID (along with several other unique school identifers) from CrossConnect.  The API from CFNC returns several properties giving a count of the number of documents returned, and a download URL which is used to actually fetch the pdfs in zip archives.  From there, the pdfs are extracted from the zip archive and stored in a temporary folder (as well as renaming the file into a specific format that contains specific identifiers for an Element record).  At this stage, the hidden internal property “cfncapppreviewprocessed” is set to true.

Next, the PDFs in the temporary folder are uploaded to an S3 bucket. A scan is then performed on the files in the S3 queue folder to extract the user_id and registration_id from the renamed filename and verify that they match a valid Element user record with that specific application on file. If no matching Element application is found for the PDF, the file is moved to a different S3 failed folder so it does not break the import task

The valid pdfs are then queued to be imported from the S3 bucket into Element451 itself using Element API’s v2/importExport/tasks with a specific task ID that is built in every CFNC supported Element instance.

Once the task is completed, the files are removed from the S3 bucket and the temporary folder they were uploaded from.

CFNC Transcript PDFs → Element451#

By default, this job runs 30 minutes after every 3 hours.  It starts by checking whether the Element transcript import task is already running, and calculates a date range covering now back to 4 days ago. Using that date range, it queries CFNC’s transcript report API with a URL like:

  • transcripts/report/submitted?newOnly=false&page=1&pageSize=100&sortBy=DateCompleted&sortDirection=DESC&startDate={STARTDATE}&endDate={ENDDATE}

It pages through the report results and uses that data to build two groups of NC Student Numbers: one for students with Final transcripts and one for students with Current transcripts.

Next, it creates an individual processing task for each NC Student Number/transcript-type combination and runs those in parallel. For each task, it chooses the appropriate transcript project ID based on the app type and whether the transcript is Final or Current, then downloads the transcript into a temporary local folder.

After the first pass, a retry step runs for failed transcript downloads in the last 7 days (just to attempt to pull them again). Successful ones are extracted and stored in the same location as the earlier successes.

Then, the job scans the local temp folder for files with “Transcript” in the filename, filters them so only the latest version of each student’s Current and Final transcript is kept, and uploads those latest files to an S3 queue folder. At the same time, it also copies each uploaded file into a dated S3 history folder.

Finally, it starts the prebuilt Element transcript import task to import the queued transcripts from S3 into Element. Once the import task completes, the queued transcript files are removed from S3 and the local temporary folder is deleted.