Data Pipeline Trainings

As a data respondent, it is important to note the expectations and requirements of this role. 

First, determine which collection(s) fall within your responsibilities. Some data respondents may only have one collection to oversee, while others may be responsible for multiple collections. Either way, it is important to note the collection timeline(s) as listed on the specific collection website(s) and plan your schedule accordingly.  

Next, ensure you have the appropriate skillset for this role and request additional training if needed. The skills deemed necessary for this role include:

• Proficiency in Microsoft Excel
• Proficiency in the LEA’s Student Information System and/or Staff Information System as required to create, produce or extract data files required for the assigned data collection(s).
• Understanding of the privacy rules and legislation such as FERPA and Privacy Act of 1974.

Also attend all collection training and learning opportunities offered by CDE collection leads to best assist you with the process. Along with attending the trainings offered, review all website documentation provided and meet all expectations listed within the collection timeline. 

Resource: Collection Overview for Respondents

Resource: Data Pipeline Collection Dates (XLS)

To complete any Data Pipeline collection, you must be provided with the necessary role(s) within the Identity Management system (IdM) by your Local Access Manager (LAM). All roles are expected to meet the collection timeline requirements and provide accurate and complete information. 

If you are assigned the LEA Viewer role, you are responsible for reviewing the data reported to CDE and ensuring that the data is accurate. You cannot make updates to the data within the system, but you can view the data.

If you are assigned the LEA User role within IdM, you are responsible for creating the data file(s), uploading the file(s) into Data Pipeline, reviewing error reports to correct the data and repeating the process as needed until all errors are resolved, warnings are reviewed and data reported is determined to be complete and accurate. 

If you are assigned the LEA Approver role within IdM, you not only have the same responsibilities as the LEA User role, but you are also responsible for marking the data file as finalized in the Data Pipeline system.

There are four types of data collections:

Year-Round are files that are open throughout the school year and can be updated as needed. Our year-round collections consist of the Record Integration Tracking system (RITS), Educator Identification System (EDIS) and the Directory. RITS creates and updates the state assigned student identifiers (SASIDs); EDIS creates and updates the educator identifiers (EDIDs), and the Directory captures the Local Education Agency's (LEA's) general information and school information. Year-round files do not require the finalize submission process.

Interchanges are files uploaded by LEAs, such as districts, BOCES or administrative units. These files are placeholders of the data, which are fluid and updated by the LEAs often. Interchange files are the source information for the snapshots. There are generic business rules in place at the interchange level to ensure the data meets basic requirements such as having a valid code, not being blank, or requiring it to be reported as zero-filled. Interchange files do not require the finalize submission process.

Snapshots pull data from interchange files for reporting purposes. LEAs then create a snapshot that extracts the interchange data, which is a point in time and used for reports after the LEA approves the data. There are business rules in place in the snapshot that are more specific and can reference data from multiple interchange files. Snapshot collections require the finalize submission process.

Periodic collections are typically one-time per year collections that do not reference any interchange files. There are also periodic collections that have the possibility of referring to data reported in interchange file(s); such as the READ spring assessments.  Periodic collections require the finalize submission process.

All collections have resources posted on that collection’s website. These resources include collection timelines, file layouts, business rules, file upload templates, and training material.

Upon navigating to the Data Pipeline Home Website, you will notice a menu on the right-hand side of the screen. Data collections are organized by the type of collection: Interchanges, Snapshots, Year-Round Collections and Periodic Collections. Once a collection type and specific collection is selected, the webpage has a similar layout with the following sections: Overview, Deadlines, File Layout and Definitions, Business Rules and Additional Resources. 

Resource: File Layout Overview (PDF)

The overall reporting process can vary by collection type and collection, so it is best to review the resources on the specific collection website for the full list of steps and overall process. However, there are a few processes that are the same for all collections.

Troubleshooting in Data Pipeline

Files must have specific data fields in a specific order and format to upload successfully into Data Pipeline. Snapshots require certain conditions are met to successfully process or include records in the snapshot. Sometimes there is an issue with one of these processes that requires additional troubleshooting. While CDE staff are available to assist, there are some general tips and steps that can be taken initially by the data respondent.  

Syncplicity 

Syncplicity is a tool utilized for sharing personally identifiable information securely between CDE staff and data respondents 

Access Directions

1. Login to the Syncplicity website.
   • Your username will be the email address listed in IdM.
   • Your password is the password you created when you set up your Syncplicity account the first time a file was shared with you. 
2. Navigate to the folder for the desired collection.
   • Contact the specific Collection Lead if you do not have access to the necessary folder. Remember, each collection has an associated email address that can be found on the Contact Us webpage. 
3. Select the upload button and select a file to upload from your computer.
4. Email the collection lead notifying them a document has been uploaded to Syncplicity. The collection lead will review your request and respond.