(1) Identify and extract information from the forms provided

 (2) Data capture needs to be done on a smartphone for portability

 (3) AI & ML techniques are used to extract information and reduce human errors

 (4) Solution should be scalable, adding more types of forms for recognition

 (5) Solution should be integrated into Collin's existing infrastructure

1. The AI behind AIFRED contains three main models:
2.  

(1) Document Classification Model - Used to classify the input data form.

AIFRED adopts state-of-the-art deep learning models to identify the form type at the pre-processing stage.

(2) Template-based Key-Value Extraction Model - Used to extract information from ARC forms.

Two main tasks are handled by AIFRED's Template-based Key-Value Extraction model

- Construct Table Structure: Takes advantage of the border information to construct a table structure 

- Template Mapping: Utilize document domain-specific knowledge such as a pre-defined table template to extract key-value pairs 

(3)  Deep-learning key-Value Extraction model - Used to extract information from PO and AWB forms.

Two main tasks are handled by AIFRED's Deep-Learning Key-Value Extraction model

- Entity Extraction: Employ state-of-the-art deep learning models and utilize text, position, visual information to extract Key and Value entities

- Key-Value Linking: Use deep learning models to study position information and link related key and value entities and form key-value pairs 

1. Each model has been trained with thousands of real data to guarantee the desired performance.

AIFRED's mobile application ensures that the client has an intuitive interface to collaborate with the strong AI backend model. It aids in automating the manual process of extracting information from physical forms.

The application is built with Flutter for its cross-platform compatibility, fast performance and strong support for machine learning libraries.

1. AIFRED contains two main parts, Data Preparation and Data Processing& Storage. 
2.  

3. Data Preparation:

       (1) The user opens the mobile application to check the status of the backend servers. 

       (2) The user scans the form from the device camera or uploads the image from the mobile gallery.

 The mobile application automatically performs a perspective transformation on all the forms that are scanned.

       (3) When necessary, the user adjusts the bounding boxes of the perspective transformation to ensure that the form image is captured correctly.

1. Data Processing & Storage

      (1) Images will be uploaded to the AI server where our AI engine will perform form classification and information extraction, the extracted information will be sent back to the mobile app.

      (2) The AI engine returns alerts for fields that have low confidence predictions, the user will then be able to correct the data by consulting the original form.

      (3) After validating the data, the user then submits the extracted information with the necessary corrections to the storage server.

(1) The Document classification model achieves 98.1% accuracy

(2) AIFRED only needs 5s for processing each document 

(2) AIFRED outperforms Google's AI model by achieving 88.9% of Key-Value Extraction Accuracy for Forms with fixed template(ARC) 

(2) AIFRED achieves 48.8% of Key-Value Extraction Accuracy for forms without fixed template(PO, AWB) which is comparable with Google's AI model