Read Invoices Custom Forms

Extracts a custom form from a pdf, tiff, jpeg or png file.

For this extraction we use Microsoft Azure Form Recognizer api.

To be able to use this activity, we need an Api Key and an Endpoint URl.

You can use the existing keys from a Microsoft Azure Form Recognizer resource(if any available), or you can create a new one. To create a new resource, once you have the azure subscription, create a Form Recognizer resource in the Azure portal to get your key and endpoint. After it deploys, click Go to resource. You will need the key and endpoint from the resource you create to connect your workflow to the Form Recognizer API.

You can use the free pricing tier (F0) to try the service, and upgrade later to a paid tier for production.

Below is a screenshot of Form Recognizer resource. Just copy one of the keys to Api Key and the EndPoint to Endpoint URl.

Form Recognizer resource keys

For this custom form extraction we need to train the Form Recognizer model with labels using the sample labeling tool https://fott-preview.azurewebsites.net/. For more details on how to train a model please check https://docs.microsoft.com/en-us/azure/cognitive-services/form-recognizer/quickstarts/label-tool?tabs=v2-1#set-up-input-data.

Using Sample labeling tool, we can compose multiple models into 1. For instance, if we have invoices for multiple suppliers, we can build a model for each supplier and then compose all these models into 1. In this way we can handle multiple invoices as 1. For more details how to compose multiple models, please check https://docs.microsoft.com/en-us/azure/cognitive-services/form-recognizer/quickstarts/label-tool?tabs=v2-1#compose-trained-models.

Custom Forms designer

Designer Properties#

  1. Api Key The Microsoft Azure api key for Form recognizer. Please check above how to get this key.

  2. Endpoint Url The Microsoft Azure Endpoint for Form recognizer. Please check above how to get this key.

  3. File Path The path to the file for which we want to extract the tables, words or lines.

  4. Model id The Model id generated in the Sample labeling tool for a form or the composed model.

    Trained Model

  5. String Fields The string fields extracted from the document using the provided model id. Represents a dictionary of fields grouped by field name(Dictionary(string, string)). The field names are the names given to each Tag in the sample labeling tool. The filed name in the below example will be "orderDate".

    Tag Custom form

  6. Output Fields The output fields extracted from document using the provided model id. Represents a dictionary of fields grouped by field name(Dictionary(string, Field)). The field name is determined in the same as for the All String Fields

    public class Field
    {
    public string Name { get; }
    public object Value { get; }
    public FieldType Type { get; }
    public List<Field> ValueList { get; }
    public Dictionary<string, Field> ValueDictionary { get; }
    }
    public enum FieldType
    {
    String,
    Date,
    TimeSpan,
    PhoneNumber,
    Float,
    Integer,
    List,
    Dictionary,
    }

  7. Output Tables Extracted tables by page number represented as a Dictionary(int, List(DataTable)), where the key is the page number while the value is a list of tables for that page.

  8. Merged Tables All the extracted data tables merged under one table and represented as a DataTable. This is useful when having only one table.

Properties#

Custom Forms properties

Azure Properties#

  1. See Designer Properties above.
  2. Minimum Confidence Minimum confidence when accepting the parsed input content. By default, it is 0.5, but it can have any values between 0 and 1.

General Properties#

See General Properties.

Misc#

See Misc.

Out Error#

See Out Error.

Result#

  1. Document Lines Extracted lines by page and line number represented as a Dictionary(int, Dictionary(int, string)), where the key is the page number while the value is a dictionary from line number to corresponding string on that line.
  2. Merged Tables See Designer Properties above.
  3. Output Fields See Designer Properties above.
  4. Output Tables See Designer Properties above.
  5. Output Words Extracted words by page and line number represented as a Dictionary(int, Dictionary(int, List(string))), where the key is the page number while the value is a dictionary from line number to the list of words on that line.
  6. String Fields See Designer Properties above.

Example#

Custom Form

In this partial example, we extract orderDate and display it in a MessageBox.

In order to run this example, we need

  1. Set the Api Key, Endpoint URl and file path.
  2. Train a custom model using the sample labeling tool https://fott-preview.azurewebsites.net/. To train a model, we need at least 5 samples.
  3. Get the model id generated and assign that to Model id