File Upload | Best Practices for Uploading Data | Data Mapping | Field Mapping | Tables | Upload | Upload Policies

Save or Update Schema | View Upload Status | Service Refresh Details | Geofence Upload

Administrators, Editors, and Validators can upload data into DATAMARK VEP. The following sections will cover the Upload workflow, best practices, and how to review the upload status.

Note: Data that contains Contingent Values and Attribute Rules may be uploaded into VEP, however, these values and rules will not be applied to the data in VEP or any subsequent data downloads.

File Upload

1. Select Uploads from the Navigation Menu.

2. Choose Upload Data.

3. Use the Upload Type drop-down to choose:

• Data Upload - Add or update data in the full VEP dataset
• Detached Editing Upload - Add or update data to a locked-down portion of the VEP dataset. For more information on Uploading Detached Edit sessions, check out our Detached Editing article. 
• Exceptions Upload - Add or update the maintained validation exceptions in VEP. For more information on uploading exceptions, check out our article Marking Exceptions.

4. Choose a zipped file to upload. For best practices on zipping data, see below.

5. Select the File Type from the drop-down:

• File Geodatabase - Place all necessary datasets into a .gdb and zip the folder.
• Shapefile - Place all shapefiles into the same folder and zip. ALI and MSAG tables can be placed in the same folder. Ensure that the field names in each dataset are 10 characters or less.
• Table - Place Excel (.xls, xlsx) and/or CSV (.csv) files into a folder and zip the folder.
• Workspace XML - XML files must comply with Esri's XML Schema of the Geodatabase technical paper published in June 2008 and available on their website. Please refer to this document for all specifications around using XML for XML workspace documents, geodatabase RecordSet documents, and as part of an XML Data Changes document.

6. Optionally add a Note for future reference when viewing the upload on the View Upload Status page.

7. Click Next to proceed.

Continue to Data Mapping for the next steps of the Upload workflow.

Best Practices for Uploading Data

Use the following guidelines to minimize receiving an invalid upload type error:

• Zip data directly so that there are no file folders within the zipped folder.
• Remove the '.gdb' extension from the zipped folder name.
• Remove any spaces from the data file name.
• When uploading a file geodatabase, make sure that there are no files outside of the file geodatabase within the zipped folder.
• Choose the correct File Type based on the file you are uploading.
• To prevent corrupting the data, make sure that it is not open in an application when it is zipped.

Data Mapping

Did you know? All feature classes in the NENA NG9-1-1 data model are eligible for upload to VEP if enabled by the Administrator in the Layers and Attributes page.

1. Use the Field Mapping dropdown to choose:

Custom Field Mapping - Select this option to manually select how the data will be Data Mapped and Fields Mapped.

Saved Schema - Selecting this will automatically populate the Data Mapping and Field Mapping pages with pre-configured Native Schema settings. Only layers with the same name as the saved schema layers will be mapped.

• Select a Native Schema from the dropdown to use during the upload if the Saved Schema was chosen as the Field Mapping policy.

2. Use the Source Dataset dropdown to select the feature layer to map to the corresponding Target Schema.

3. Click Next to proceed.

Note: Each Feature Layer in the Target Dataset must be matched to a unique Source Dataset Layer. If a source feature layer needs to be mapped to multiple target datasets, they must be duplicated and uniquely named before data mapping.

The Source Dataset dropdown will display layers of the same Feature Class type as the Target Dataset layer. 

Field Mapping

Note: If a Saved Schema was applied to in the Data Mapping step, any identified fields from the schema will be auto-mapped on this page. VEP will only map fields with the same name as the previously saved schema fields.

Users may map additional fields or update the Saved Schema fields as they choose.

1. For each Target Field from the VEP NENA schema, use the dropdowns to map the appropriate Source Field found in the uploaded data. 

2. If a Source Field has the same name as the Target Field, click Auto Map to have VEP map these fields instantly.

Caution: If Auto Map is selected when a Saved Schema is applied, the schema will be removed and will need to be reapplied in the Data Mapping step.

3. Click Next once all the NENA schema Target Fields have been populated. If there are fields in the uploaded data that were not field mapped to a Target Field, VEP will offer to maintain them in a Related Table:

• If the unmapped fields match the names of fields found in an existing related table, the user may check the box to update the existing table. Use the dropdown to select which related table to update. Use the checkboxes to include or exclude what fields to update.
• Check Create a new related table to make a new related table from the unmapped fields. Edit the name if desired and use the checkboxes to include or exclude fields for the table.
• Click OK when finished.

Did you know? When downloading in the Native Schema, the related tables will automatically append to the associated feature class, resulting in one holistic dataset, just as uploaded. For more information on downloads, see our Standard Download article.

Note: If the external_nguid field is not field-mapped, VEP will populate value for this field.

If a data type mismatch occurs during the Upload process, VEP will attempt to detect and auto-correct the mismatch and provide the user with an option. We recommend making the field type change indicated in the message to make the data more NG9-1-1 ready.

Tables

1. Use the Source Dataset checkbox to select tables from the dataset to upload to VEP.

2. Click Next to begin Relationship Mapping the selected tables.

3. Use the following Relationship Mapping drop-downs to build a relationship between the uploaded tables and existing VEP data:

Target Table: Select a Feature Layer or existing table in VEP to create a relationship with the uploaded table.

Source Field: Designate a field from the uploaded table to link with a field from the Target Table.

Target Field: Select the field to be linked with the Source Field.

Relationship Type: Configure if the relationship between the Source Table record and the Target Table will be One-to-One or One-to-Many.

4. Click Next to proceed.

Note: Related tables are not automatically reflected in VEP after the upload is approved. An automatic Service Refresh will occur in VEP at 3 a.m. EST the following day, and the uploaded related tables will appear after this service restart. To manually start a Service Refresh, see Service Refresh Details below.

Note: Uploading additional tables is optional. To skip this step, or if there are no tables to upload, click Next to continue.

Caution: If a table was Data Mapped to the ALI or MSAG, do not select to upload it here. This will create a duplicated table in the data and cause errors during subsequent data downloads.

Note: To upload a standalone table, leave the Relationship Mapping dropdowns blank and click Next. 

To update an existing related table with schema changes, the original related table must be deleted before proceeding with the upload.

Upload

1. Select an Upload Policy Option:

- Apply to All Layers: Apply a single upload policy to all layers and tables being uploaded.

- Choose by Layer: Manually select an upload policy for each layer and table being uploaded.

2. Choose the upload policy to apply. For more information on Upload Policies, see below.

• Apply All Edits, Adds, & Deletes: Will apply edits, added features, and deletes to the existing feature class. All records in each of the feature classes and tables will need to be present in the .zip file uploaded.
• Replace All Data: Will delete all records for the existing feature class and replace them with the uploaded dataset. All records in each of the feature classes and tables will need to be present in the .zip file uploaded.
• Append Edits & Adds: Will apply edits and new features to the existing feature class and will not delete any records. This option will ignore deletes, so the entire dataset does not need to be present. 
• Upload New Table: This will add a new table and not affect any existing data. This policy is only available for new dynamic-related tables.
• Do Not Upload: Will not upload the feature class or table. This option is used for saving field mapping when adding a new schema.

3. Click Upload to be prompted to save or update a schema (For more information, see Save or Update Schema below).  Editor subscription users will be asked to name the upload. 

4. The Data Verification pop-up will appear to check the data for any discrepancies that could be harmful to the VEP dataset. 

Note: If an issue is found with the external_nguid sequence, the Data Verification step will provide information on the values that are out of sequence so the user may fix them before attempting another upload.

Once the Data Verification is completed, the user will be redirected to the View Upload Status page. (See below)

Upload Policies

The upload policy determines the write method that will be applied when GIS data updates are committed to the database. VEP uses four different upload policy types and it is important to understand the differences and how those will apply to data uploads so that the updates have the desired effect on any necessary data improvements and remediation efforts. The following Upload Policies table shows how each dataset requirement relates to different use cases: 

Save or Update Schema

At the end of the upload workflow, users will be prompted to save or update a Saved Schema with the current upload configuration. Click yes to access the Schema form:

1. To update an existing Saved Schema, use the Update Existing Schema dropdown to select the Saved Schema to replace.

2. To save a new schema, name the schema in the Save New Schema field

3. Click Save.

Did you know? You can use the Upload workflow to update or create a new Saved Schema without having to follow through with the upload. After saving the schema settings, you may exit the upload. The schema configuration will still be saved. 

View Upload Status

1. Select Upload from the Navigation Menu

2. Click View Upload Status.

3. Optionally query the Upload Requests by a date range.

4. View the status of pending or previous upload requests.

Note: For Validator subscriptions, this is the end of the Upload workflow. For Editor subscriptions, once the upload completes, an Admin will need to approve the upload for it to be posted to the data.
 
If the Admin rejects the uploaded version, it will display on the upload status page with the status "Rejected." No data will be migrated into the system and the version will be deleted. The user will need to perform the upload again with corrections, if necessary.

If a validation, download, database compression, or another upload is actively processing, the upload will display a Queued status. As soon as the active process completes, the queued upload will begin processing.

Service Refresh Details

1. Fill out the From and To fields to search a Map Service Refresh by a date range. 

2. If a service refresh has a Pending status, click the Refresh icon to trigger a manual Service Refresh.

Caution: Make sure there are no users editing in VEP before manually performing a Service Refresh. After the map refresh completes, all users should refresh their browser before resuming work in the web map.

Geofence Upload (Geofence Users)

For users that are assigned more than one Geofence Boundary, an additional drop-down will be displayed during the upload process called Geofence Boundary. For this drop-down, select which jurisdictions the uploaded data falls within. If multiple, select multiple jurisdictions. Proceed with the upload as normal.