Skip to main content

CLU Manager

The CLU Manager allows developers to train an Intent Classifier in Teneo using a service provided by Microsoft in Azure AI Services.

The sections on this page provide information about the various aspects of the CLU Manager:

Concept​

Conversational Language Understanding (CLU) is a service provided by Microsoft in the Azure AI Language layer in the Azure ecosystem. The service allows developers to train and query Natural Language Understanding (NLU) models using state-of-the-art, transformer-based, pre-trained language models. Furthermore, CLU provides multilingual capabilities.

CLU models are a way to perform Intent Classification in Teneo. Teneo allows you to integrate with CLU seamlessly and build, customize and test CLU models directly within Teneo Studio. When a CLU model is trained in Teneo Studio, the actual training takes place in the Azure AI Language layerr in the Azure ecosystem and when done, the model becomes available in the CLU Manager within Studio for the developer to assign it to the solution.

How this is done in practice:

  1. Classes and training examples are added in the Class management.
  2. The training is initiated in the CLU Manager; the CLU model starts training in Azure.
  3. The CLU model can now be assigned to a solution through the CLU Manager.

Depending on the language, within Teneo, inputs may be lowercased, but no further input processing is applied when using CLUs. Also, please note that there is a limit of maximum 10 CLU models per solution.

Models are displayed in the CLU Manager in the following cases:

  • The model has usages in the solution.
  • The model (either trained or training) can be assigned to the solution.
  • The model was used in a historical publication of the solution.
note

CLU settings must be defined in the Account tab of Teneo Studio to work with CLU.

Valid CLU Models​

A CLU model is valid and can be assigned to a solution if the training examples in the Class Manager have not changed. This is because the training examples must be exactly the same as when the CLU model was trained, as a CLU model is always trained on the latest solution revision.

If updates are performed to the training examples, then:

  • Any unassigned CLU model becomes unassignable.
  • A CLU model assigned to the latest solution revision is automatically unassigned and Studio defaults to the Learn classifier.

In both of the cases, a new CLU model must be trained based on the updated training examples in order to assign the new model to the solution. Users can also train a new model if they believe the other solutions contain updated training data but no model has been trained yet. Note that the model can be set to stable in projects which make use of the Version flag feature.

info

Changes in training data unassign the CLU model in the same solution, but don't affect other solutions directly.

In some cases, a model with old training data can be valid. For example, if the training data is modified but the model is used in the stable solution revision or if it has been published in one of the solution's publish environments, it will still be valid. In other words, the stable solution can still be using a previous model as long as the model was set to stable while it was in a valid state. Similarly, a previous model can be available in a published solution if the model was valid and assigned to the solution at the moment of publishing.

Reverting to Learn​

Teneo will default to Teneo Learn if a CLU cannot be used, as a Learn model is always sent along when publishing a solution. CLU is a cloud-based service and therefore there may be limitations regarding the number of deployed (i.e. runtime) models and number of API calls per minute. Also, as the deployed CLU model sits in the cloud, it might be impossible to query it in case of a problem in Azure or the network. For more information about Azure's service limits, please see the Azure documentation.

Additionally, Teneo Learn will be used in the following cases:

  • If no CLU model is assigned to the solution.
  • If updates to training examples invalidate the current CLU model.
  • If CLU is not available in the Studio account.
  • If no stable version of the Intent Model Assignment exists.
info

Teneo Studio displays warning messages when performing changes which will invalidate the currently assigned CLU model, and that Teneo Learn will be used as the intent classifier until a new CLU model is trained and assigned to the solution.

Localization Setup​

CLU models can be trained in one language (or even in various languages) and used to predict intents in another language, making CLU an interesting option for projects which make use of a Localization setup. This powerful feature can save a huge amount of time and effort; instead of building separate projects for every language, you can handle multilingual datasets in one project.

This is accomplished in Teneo through branching, where there is an option of creating a 'Master' version of a solution and its own 'Local' versions for each of the different languages needed. This way, Teneo allows to create local solutions with all of the parts of the master solution. If a master solution has a CLU model assigned and all classes are included, during branching the same model will already be assigned to the local solution.

In Localization setup, CLU models are still trained per solution and there is - as with other solutions - a limit of 10 CLU models per solution. However, solutions in Localization setups can use trained CLU models from any of the other solutions of the setup if the model is valid for that particular solution, i.e., if all solutions define exactly the same classes (based on the class Id).

Note that the Test Data Evaluation runs on the solution's test data examples and the test data examples are not shared across Master and Local solutions.

See Microsoft documentation for more information on the CLU multilingual option and read more about their supported languages.

Version Flags​

Teneo creates history when the assignment of the Intent Model changes, either when a new CLU model is assigned or when a CLU model is unassigned and Teneo reverts to Learn. You can access and restore previous versions of the CLU models in the version flags and use version flags to set a document as stable.

Intent Model Assignment​

The Intent Model Assignment is a versionable document which keeps track of the assigned CLU Intent Model history. A version of the Intent Model Assignment can be set as stable, but unlike other solution documents, doing so is not mandatory in a stable solution. You may want to set an Intent Model Assignment as stable when it has been quality assured, tested and is ready to be published. Read more about version flags here.

Only Intent Model Assignments that use CLU can be set as stable. If you want to use Learn, the Intent Model Assignment simply needs to be unset stable. When setting an Intent Model Assignment version as stable, it is possible that the CLU model referenced by that version no longer exists. In that case, Studio doesn't allow setting that particular version as stable and Teneo will handle creation of a correctly versioned stable Learn model.

The Intent Model Assignment can be set / unset as stable in the CLU Manager as described here.

Consistency​

Similarly to the latest solution version, where consistency between the training data and the assigned model is guaranteed by resetting the Intent Model Assignment from CLU to Learn if the training data changes, consistency must be guaranteed in the stable solution version.

Therefore, when assigning an Intent Model Assignment version that uses a certain CLU model...

  • All versions of the class documents that were used to train the model are automatically set as stable.
  • Class documents that were not used in the model are unset from stable.

Correspondingly, if CLU is used in the stable Intent Model Assignment, setting / unsetting class documents as stable will unset the stable Intent Model Assignment version.

How To​

note

The CLU Manager is only available in Teneo Studio Desktop. You can, however, modify the training examples in Teneo Studio Web as well.

Train​

To train a new CLU model based on changes performed in the Class Manager, follow the below steps:

  1. Open CLU Manager.
  2. Click Train in the top ribbon.
  3. Teneo will now start the CLU model training.
note

A new CLU model can only be trained if no previous model exists, if changes were performed to the training data since the last training and if no duplicated examples are detected.

The CLU model will be trained on the same training data as the native classifier. Our solution already has all of this training data ready, so let's get right into training the model.

  1. Open the CLU Manager in the ribbon bar.
  2. Click on 'Train'.
  3. Wait for the model to finish training. This can take a while, but you do not need to keep this window open until it is done; if you want, you can close it and come back later.
  4. Once the model is finished training, click on 'Assign' in the ribbon bar of the CLU Manager window. This will assign the model to the solution, ensuring that the engine will use this model for classification instead of Teneo's native classifier.

Cancel Training​

To cancel a training, simply click the Cancel button which appears in the CLU Models list for the running training.

Assign​

When a new CLU model has been trained, to assign the model to the latest solution revision, follow the below steps:

  1. Open CLU Manager.
  2. Select the new model in the CLU Models list.
  3. Click Assign in the top ribbon.
  4. Teneo confirms the assignment with a "Assignment process has finished" message. Click on Close and proceed to also close the CLU Manager.
info

A CLU model can only be assigned to the solution if the training data used to train the model is the same as the training data currently in the solution.

Unassign​

To unassign a CLU model, follow the below steps:

  1. Open CLU Manager.
  2. Select the currently assigned model in the CLU Models list.
  3. Click Unassign in the top ribbon.
  4. Teneo Studio prompts the user to confirm the action, optionally enter a comment (for the version history).
  5. Click Yes.
  6. Teneo confirms the unassignment with a "Unassignment process has finished" message, click Close and proceed to also close the CLU Manager.
info

When unassigning a CLU model, Teneo will automatically use the Learn Intent Classifier instead; a previously assigned model might still be used for the stable solution revision or in a published solution.

Delete​

To delete a CLU model, follow the below steps:

  1. Open CLU Manager
  2. Select the model to remove
  3. Click Delete.
  4. Teneo Studio prompts to confirm the deletion, click Yes.
  5. The model is now deleted and will disappear from the CLU Models list.

A solution can have a maximum of 10 CLU models, therefore inevitably at some point one or more models must be removed from the solution.

When deleting a CLU model in Teneo, please take into account the following:

  • CLU models used by latest or stable solution versions cannot be deleted.
  • CLU models used by an active publish target cannot be deleted.
  • When the model is deleted, it will no longer be available if old solution versions are re-published in the future.
  • Deleting a CLU model implies removing the CLU deployment and the CLU model from the CLU servers.
note

Models can only be deleted from the solution which trained it. However, it is not possible to remove models if other solutions still use them. Similarly, it is not possible to remove solutions if the solution contains models that are still in use by other solutions.

Close​

To close the CLU Manager, either:

  • Click the X in the top, right corner of the CLU Manager window.
  • Click the Close button available in the top ribbon.
  • In the CLU Manager backstage, click Close in the left-side menu.

Set Stable Version​

The Intent model assignment can be set to stable as follows:

First, at least one CLU model must be trained and assigned to the latest solution!

  1. In the top ribbon of the CLU Manager, click Intent model assignment.
  2. In the History, select the wanted model assignment version; note that only versions with "Intent model assignment type CLU" can be set as stable.
  3. Click Set Stable in the upper, right corner of the History view.

Also see History and Version Flag.

Connect CLU to a Solution​

Adjust the CLU settings in Teneo to allow access to CLU. This way, your solution will be able to use the CLU native intent classifier instead of Teneo's native classifier and classes in the Class Manager.

  1. Navigate to the 'Account' section in the sidebar. Here you can see information about and settings regarding your CLU setup.
  2. If you have not established a CLU connection, click 'Add' and enter your resource name and API key. If you wish to change the configuration, click 'Edit'.
  3. Hit 'Save'.
note

Please note that you can only configure one CLU model connection per account. However, it is possible to train up to 10 different intent classifiers using the CLU model.

UI​

CLU Manager Window​

The CLU Manager window is accessed by clicking the CLU Manager button in the top ribbon of Teneo Studio and consists of two tabs: Home and CLU.

info

The CLU Manager button is only visible when CLU Settings are defined in the Account tab.

clu-manager-window

The CLU tab consists of a left-side menu with the following:

  • Close allows to close the CLU Manager.

The Home tab contains the following:

  • Top ribbon providing buttons to
    • Close the window.
    • Train a CLU model.
    • Delete an existing CLU model.
    • Assign a CLU model to the solution.
    • Unassign a CLU mode from a solution.
  • The CLU Models list displays the available CLU models and previously trained models, providing information related to:
    • The training date (Started) and the duration of the training (Duration)
    • The user who started the training
    • The expiration date of the CLU model; read more in the Azure documentation.
    • Status (when applicable)
    • Stable/Latest version of a CLU model (when applicable)
    • Published events related to a specific CLU model (when applicable)

Assigned Intent Model​

In Teneo Studio Web, information about the currently assigned intent model is available in the upper, right corner of the Tryout.

web-tryout-assigned-model

Although a CLU model can be used in Tryout in Teneo Studio Web, currently the actual assignment of the CLU model can only be done in Teneo Studio Desktop.

Training Status​

Information related to the CLU model training is displayed in the CLU Manager where a new training is listed at the top of the CLU Models list and the status available in the right-hand side.

Training in progress

The status message may be one of the following:

  • Importing in progress
  • Training is queued
    • Models are usually queued on the CLU server before before the training actually starts.
  • Training in progress
    • Displayed while the training is running with a progress bar, the duration (until now) and the expected remaining time.
  • Cancelling
    • Displayed if the user clicks the Cancel button.
  • Training cancelled
    • Displayed when a training is successfully cancelled.
  • Trained CLU model available
    • Successful training of CLU model. The model can now be assigned and used in the solution.

Information related to the CLU model trainings is also displayed in the Status bar located in the lower part of the Teneo Studio window. This allows developers to work on other solution documents while the CLU model is being trained.

CLU model training messages

Duplicated Training Examples​

CLU does not allow for duplicated training data examples. If duplicated examples are detected by Teneo, these are raised as improvement suggestions in the backstage of Studio as either Class has training examples duplicated in multiple classes or Class has duplicated training data.

In case of duplicated examples across classes, the Train button is deactivated in the CLU Manager and an informative message displayed at the top of the CLU Models list to draw the developers attention to these as they need to be removed manually. Duplicated examples within the same class are automatically removed by Teneo before sending the data to CLU.

Duplicated examples detected

info

The detection of duplicated training data examples is case insensitive.

Practical Examples​

Multilingual CLU​

note

This practical example assumes that the user has already connected and trained a CLU in an existing solution.

You can easily use the same CLU model in different solutions, even across languages. When you branch a local solution from master, the new solution automatically uses the CLU model.

After branching, you can start chatting with the bot in the local solution's language immediately and it will understand you; you only need to translate the answers into another language. It is also recommended to fine-tune the local solution by adding additional training data in the local language.

This simplified example will show how you to use CLU and localize classes. In order to localize the whole solution, you will also need to include other elements like Language Objects, and update the TLML Syntax. To see how to do so, please take a look at Localization setup.

Add local training data​

After branching, the model can be expanded with additional local language training examples and a new fine-tuned model can be trained. Adding more training data in German for the German solution boosts the model's performance.

  1. Open the Class Manager in the top ribbon bar.
  2. Select a Class, for example 'TELL_ME_ABOUT_YOUR_COFFEE', and click on 'Edit'.
  3. Under the existing training examples for this Class, add examples in German. They may look like the following ones:
Können Sie mir über eure Kaffeesorten erzählen
Verkauft ihr espresso
Welche Getränke haben Sie
  1. Hit 'Save'.

Retrain the model​

As the classes in the localized solution are not identical to those in the Master solution anymore, we will need to train a new CLU model.

  1. Open the CLU Manager in the top ribbon bar.
  2. Click on 'Train'.
  3. Once the model is done training, select it and click on 'Assign' to assign it for use in the solution.

The bot will now use the newly trained CLU model to handle conversation in German.

Localize outputs​

The bot can now handle German inputs easily, but the Flows taken from our Master solution are still in their original language. Let's localize Flow outputs to get the answers we want in German.

  1. Open a Flow.
  2. Replace the answer in all the output nodes with a German answer.
  3. Repeat this process for the remaining Flows to get German answers for all use cases.

The bot should now have German responses!