For a first impression on what INCEpTION is, you may want to watch our introduction video.

INCEpTION is a text-annotation environment useful for various kinds of annotation tasks on written text. Annotations are usually used for linguistic and/or machine learning concerns. INCEpTION is a web application in which several users can work on the same annotation project and it can contain several annotation projects at a time. It provides a recommender system to help you create annotations faster and easier. Beyond annotating, you can also create a corpus by searching an external document repository and adding documents. Moreover, you can use knowledge bases, e.g. for tasks like entity linking.

The following picture gives you a first impression on how annotated texts look like. In this example, text spans have been annotated as whether they refer to a person (PER), location (LOC), organization (ORG) or any other (OTH).

INCEpTION’s key features are: First, before you annotate, you need a corpus to be annotated (Corpus Creation). You might have one already and import it or create it in INCEpTION. Second, you might want to annotate the corpus (Annotation) and/or merge the annotations which different annotators made (Curation). Third, you might want to integrate external knowledge used for annotating (Knowledge Bases). You can do all three steps with INCEpTION or only one or two.
In addition, INCEpTION is extendable and adaptable to individual requirements. Often, it provides predefined elements, like knowledge bases, layers and tagsets to give you a starting point but you can also modify them or create your own from scratch. You may for example integrate a knowledge base of your choice; create and modify custom knowledge bases; create and modify custom layers and tagsets to annotate your individual task; build custom so-called recommenders which automatically suggest annotations to you so you will work quicker and easier; and much more.

Getting Started focuses on annotating. For details on any other topic like Corpus Creation or the like, see the main documentation part of this User Guide: Core Functionalities.

INCEpTION is still in development, so you are welcome to give us feedback and tell us your wishes and requirements.

Our main documentation consists of three distinct documents:

All materials, including this guide, are available via the INCEpTION homepage.

In order to run INCEpTION, you need to have Java installed in version 11 or higher. If you do not have Java installed yet, please install the latest Java version e.g. from AdoptOpenJDK.

In this section, we will download, open and log in to INCEpTION. After, we will download and import an Example Project:

Step 1 - Download: Download the .jar-file from our website by clicking on INCEpTION x.xx.x (executable JAR) (instead of “x.xx.x”, there will be the number of the last release). Wait a minute until it has been fully downloaded. That is, until the name of the downloaded folder ends on “.jar“, not on “.jar.part“ anymore.

Step 2 - Open: There are two ways to open the application: Either by double-clicking on it or via the terminal.

Step 2a - Open via double-click: Now, simply double-click on the downloaded .jar-file. After a moment, a splash screen will display. It shows that the application is loading.

Once the initialization is complete, a dialog appears. Here, you can open the application in your default browser or shut it down again:

Step 2b - Open via terminal: If you prefer the command line, you may enter this command instead of double-clicking. Make sure that instead of “x.xx.x”, you enter the version you downloaded:

In this case, no splash screen will appear. Just go to http://localhost:8080 in your browser.

Step 3 - Log in: The first time you start the application, you will be asked to set a password for the default admin user. You need to enter this password into two separate fields. Only if the same password has been entered into both fields, it will be accepted and saved. After the password has been set, you will be redirected to the regular login screen where you can log in using the username admin and the password you have just set.

You have finished the installation.

In order to understand what you read in this guide, it makes sense to have an annotation project to look at and click through. We created several example projects for you to play with. You find them in the section Example Projects on our website.

Step 1 - Download: For this guide, we use the Interactive Concept Linking project. Please download it from the Example Projects section on our website and save it without extracting it first. It consists of two documents about pets. The first one contains some annotations as an example, the second one is meant to be your playground. It has originally been created for concept linking annotation but in every project, you can create any kind of annotations. We will use it for Named Entity Recognition.

In this section we will see what elements each project has and where you can adjust these elements by examining the Project Settings. Note that you may have different projects in INCEpTION at the same time.

If you prefer to make some annotations first, you may go on with First Annotations with INCEpTION and return later.

Each project consists at least of the following elements. There are more optional elements such as tagsets, document repositories etc. but to get started, we will focus on the most important ones:

For a quick overview on the settings, you might want to watch our tutorial video Overview. As for all topics of Getting Started, you will find more details on each of them in the main documentation on INCEpTION’s Core Functionalities.

The Settings provide a tab for each of these elements. There are more tabs but we focus on the most important ones to get started. You reach the settings after logging in when you click on the name of a project and then on Settings on the left. If you have not imported the example project yet, we propose to follow the instruction in Download and import an Example Project first.

Here, we will find out what you can do in each project having a look at the Structure of an Annotation Project. Therefore, we examine the dashboard.

If you are in a project already, click on the dashboard button on the top to get there. If you just logged in, choose a project by clicking on its name. As you are a Project Manager (see User Rights), you see all of the following sub pages. For details on each section, check the section on Core Functionalities.

In this section, we will make our first annotations. If you have not downloaded and imported an example project yet, we recommend to return to Download and import an Example Project and do so first. In this section, no or little theory and background will be explained. In case you want to have some theory and background knowledge first, we recommend reading the section Structure of an Annotation Project.

Create your first annotations

This will lead you step by step. You also may want to watch our tutorial video „Overview“ on how to create annotations. We will create a Named Entity annotation which tells whether a mention is a person (PER), location (LOC), organization (ORG) or other (OTH):

Step 1 - Opening a Project: After logging in, what you see first is the Project overview. Here, you can see all the projects which you have access to. Right now, this will be only the example project. Choose the example project by clicking on its name and you will be on the Dashboard of this project.

Step 2 - Open the Annotation Page: In order to annotate, click on Annotation on the top left. You will be asked to open the document which you want to annotate. For this guide, choose pets1.tsv.

Step 3 - Create an Annotation: After opening the document, select Named entity from the Layer dropdown menu on the right side of the screen to create your first annotation. Then, use the mouse to select a word in the annotation area, e.g. in my home in line one. When you release the mouse button, the annotation will immediately be created and you can edit its details in the right sidebar (see next paragraph). These “details” are the features we mentioned before.

Note: All annotations will be saved automatically without clicking an extra save-button.

Congratulations, you have created your first annotation!

Now, let‘s examine the right panel to edit the details or to be precise: the features. You find the panel named Layer on top and Annotation below.

In the Layer-dropdown, you can choose the layer you want to annotate with as we just did. You always have to choose it before you make a new annotation. After an annotation has been created, its layer cannot be changed any more. In order to change it, you need to delete it, select the right layer and create a new annotation.

If you are not sure what layers are, check the box on Layers and Features in the section Project Settings. In order to learn how to adjust and create them for your purpose, see section Layers in the main documentation.

In the Annotation panel, you see the details of a selected annotation. They are called features.

It shows the layer the annotation is made in (field Layer; here: Named entity) and what part of the text has been annotated (field Text; here in my home). Below, you can see and modify what has been entered for each of the so-called Features. If you are not sure what features are, check the box on Layers and Features in the section Project Settings (Here: The layer Named entity (see the note box on Named Entity) has the features identifier and value. The identifier tells, to which entity in the knowledge base the annotated text refers to. For example, in case the home referred to here is a location the knowledge base knows, you can choose it in the dropdown of this field. The value tells if it is a Location (LOC) like here, a Person (PER), Organization (ORG) or any other (OTH).).
You may enter free text here or work with tagsets to have a well defined set of labels to enter so all of the users within one project will use the same labels. You can modify and create tagsets in the project settings. See section Tagsets in Getting Started or check the main documentation for Tagsets.

You have almost finished the Getting Started. One word can still be said about the Sidebars on the left. These offer access to various additional functionalities such as an annotation overview, search, recommenders, etc. Which functionalities are available to you is determined by the project settings. The sidebars can be opened by clicking on one of the sidebar icons and they can be closed by clicking on the arrow icon at the top.

There are several features you might want to check the main documentation for. Especially the Recommender section of the sidebar (the black speech bubble) is worth a look in case you use recommenders (see Recommenders in the section Project Settings). Amongst others, you will find their measures and learning behaviours here. Also note the Search in the sidebar (the magnifier glass): You can create or delete annotations on all or some of the search results.

To get familiar with INCEpTION, you may want to follow the instructions for other example-projects, read more in-depth explanations on its Core Functionalities or explore INCEpTION yourself, learning by doing.

One way or the other: Have fun exploring!

We hope the Getting Started helped you with your first steps in INCEpTION and gave you a general idea of how it works. For further reading and more details, we recommend the main documentation, starting right after this paragraph.

Do not hesitate to contact us if you struggle, have any questions or special requirements. We wish you success with your projects and you are welcome to let us know what you are working on.

At the top of the screen, there is always a menu bar visible which allows a quick navigation within the application. It offers the following items:

After logging in to INCEpTION, the first thing you see is the project overview. Here, you can see all the projects to which you have access. For every project, the roles you have are shown.

Using the filter toggle buttons, you can select which projects are listed depending on the role that you have in them. By default, all projects are visible.

Users with the role project creator can conveniently create new projects or import project archives on this page.

Users without a manager role can leave a project by clicking on the Leave Project button below the project name.

When uploading projects via this page, user roles for the project are not fully imported! If the importing user has the role project creator, then the manager role is added for the importing user. Otherwise, only the roles of the importing user are retained.

If the current instance has users with the same name as those who originally worked on the import project, the manager can add these users to the project and they can access their annotations. Otherwise, only the imported source documents are accessible.

Users with the role administrator who wish to import projects with all permissions and optionally create missing users have to do this through the Projects which can be access through the Administration link in the menu bar.

Once you have selected a project from the Project overview, you are taken to this project’s dashboard. Depending on the roles that a user has in the project, different functionalities can be accessed from here such as annotation, curation and project configuration. On the right-hand side of the page, some of the last activities of the user in this project are shown. The user can click on an activity to resume it e.g. if the user annotated a specific document, the annotation page will be opened on this document.

When navigating to the Annotation page, a dialogue opens that allows you to select the document you want to annotate. If you want to access this dialog later, use the Open button in the action bar.

The keyboard focus is automatically placed into the search field when the dialog opens. You can use it to conveniently search for documents by name. The table below is automatically filtered according to your input. If only one document is left, you can press ENTER to open it. Otherwise, you can click on a document in the table to open it. The Filter buttons allow to filter the table by document state.

Users that are managers can additionally open other users' documents to view their annotations but cannot change them. This is down via the User dropdown menu. The user’s own name is listed at the top and marked (me).

Sentence numbers on the left side of the annotation page show the exact sentence numbers in the document.

The arrow buttons first page, next page, previous page, last page, and go to page allow you to navigate accordingly.

The Prev. and Next buttons in the Document frame allow you to go to the previous or next document on your project list.

When an annotation is selected, there are additional arrow buttons in the right sidebar which can be used to navigate between annotations on the selected layer within the current document.

You can also use the following keyboard assignments in order to navigate only using your keyboard.

The Layer box in the right sidebar shows the presently active layer span layer. To create a span annotation, select a span of text or double click on a word.

If a relation layer is defined on top of a span layer, clicking on a corresponding span annotation and dragging the mouse creates a relation annotation.

Once an annotation has been created or if an annotation is selected, the Annotation box shows the features of the annotation.

The definition of layers is covered in Section Layers.

The annotation sidebar provides an overview over all annotations in the current document. It is located in the left sidebar panel.

The sidebar supports two modes of displaying annotations:

The undo/re-do buttons in the action bar allow to undo annotation actions or to re-do an an undone action.

This functionality is only available while working on a particular document. When switching to another document, the undo/redo history is reset.

Once the document is opened, a default of 5 sentences is loaded on the annotation page. The Settings button will allow you to specify the settings of the annotation layer.

The Editor setting can be used to switch between different modes of presentation. It is currently only available on the annotation page.

The Sidebar size controls the width of the sidebar containing the annotation detail editor and actions box. In particular on small screens, increasing this can be useful. The sidebar can be configured to take between 10% and 50% of the screen.

The Font zoom setting controls the font size in the annotation area. This setting may not apply to all editors.

The Page size controls how many sentences are visible in the annotation area. The more sentences are visible, the slower the user interface will react. This setting may not apply to all editors.

The Auto-scroll setting controls if the annotation view is centered on the sentence in which the last annotation was made. This can be useful to avoid manual navigation. This setting may not apply to all editors.

The Collapse arcs setting controls whether long ranging relations can be collapsed to save space on screen. This setting may not apply to all editors.

The Read-only palette controls the coloring of annotations on read-only layers. This setting overrides any per-layer preferences.

Annotations are always immediately persistent in the backend database. Thus, it is not necessary to save the annotations explicitly. Also, losing the connection through network issues or timeouts does not cause data loss. To obtain a local copy of the current document, click on export button. The following frame will appear:

Choose your preferred format. Please take note of the facts that the plain text format does not contain any annotations and that the files in the binary format need to be unpacked before further usage. For further information the supported formats, please consult the corresponding chapter Formats.

The document will be saved to your local disk, and can be re-imported via adding the document to a project by a project manager. Please export your data periodically, at least when finishing a document or not continuing annotations for an extended period of time.

The search module allows to search for words, passages and annotations made in the documents of a given project. Currently, the default search is provided by MTAS (Multi Tier Annotation Search), a Lucene/Solr based search and indexing mechanism (https://github.com/textexploration/mtas).

To perform a search, access the search sidebar located at the left of the screen, write a query and press the Search button. The results are shown below the query in a KWIC (keyword in context) style grouped by document. Clicking on a result will open the match in the main annotation editor.

The search only considers documents in the current project and only matches annotations made by the current user.

Clicking on the search settings button (cog wheel) shows additional options:

The statistics section provides useful statistics about the project. Currently, the statistics are provided by MTAS (Multi Tier Annotation Search), a Lucene/Solr based search and indexing mechanism (https://github.com/textexploration/mtas).

After configuring one or more recommender in the Project Settings, they can be used during annotation to generate predictions. In the annotation view, predictions are shown as grey bubbles. Predictions can be accepted by clicking once on them. In order to reject, use a double-click. For an example how recommendations look in action, please see the screenshot below.

Suggestions generated by a specific recommender can be deleted by removing the corresponding recommender in the Project Settings. Clicking Reset in the Workflow area will remove all predictions, however it will also remove all hand-made annotations.

Curation i.e. the process of combining finished annotated documents into a final curated document, can be done via the Curation Page in INCEpTION (see Curation) but also via the Curation Sidebar on the Annotation Page.

To start a curation session, you need to choose a curation target and press the start curation session buttion next to the curation target select box.

Once the session has started, annotations from annotators that have marked the document as finished will be visible. You can see a list of these annotators in the sidebar. If you want to see only annotations from specific annotators, you can enabled/disabled them as you like.

The user can copy annotations manually from other users into the curation document by clicking on them. The automatic merge can be triggered by clicking the Re-Merge button (sync icon). It will copy all annotations that all selected users agree on into the curation document.

Depending on the layer settings, once an annotation has been merged or an annotation has been manually created in the curation target document, annotations from annotators might be hidden. This happens for example when at a given position in the curation target document an annotation exists and stacking is not enabled on the respective annotation layer. If you want to temporarily see all annotations from the selected annotators, use the Show all curatable annotations checkbox.

The curation session remains active for a given project, even if you leave the annotation page and come back at a later time.

To stop a curation session, use the stop curation session button next to the curation target select box.

It is possible to start/stop a curation session via the URL. Adding the URL query parameter curationSession=on to an annotation page URL will start a curation session (if none is running) while curationSession=off will stop a running session. By default, the session is started using the curation document as the curation target. By setting the parameter curationTargetOwn=true, the curation target can be changed to the current users own document - if the user has the annotator role in addition to the curator role. This parameter only takes effect when curationSession=on is also set. Mind that curation sessions in a project run until terminated. If you want directly link to a document on the annotation page and ensure that no curations session is running, be sure to add the curationSession=off parameter. Example: http://localhost:8080/p/PROJECT/annotate/DOC-ID?curationSession=on&curationTargetOwn=true.

Active learning is a family of methods which seeks to optimize the learning rate of classification algorithms by soliciting labels from a human user in a particular order. This means that recommenders should be able to make better suggestions with fewer user interactions, allowing the user to perform quicker and more accurate annotations. Note that Active Learning only works if there are recommenders and if these recommenders actually generate recommendations which are usually shown as grey bubbles over the text.

Open the Active Learning sidebar on the left of the screen. You can choose from a list of all layers for which recommenders have been configured and then start an active learning session on that layer.

The system will start showing recommendations, one by one, according to the uncertainty sampling learning strategy. For every recommendation, it shows the related text, the suggested annotation, the score and a delta that represents the difference between the given score and the closest score calculated for another suggestion made by the same recommender to that text. Additionally, there is a field which shows the suggested label and which allows changing that label - i.e. to correct the suggestion provided by the system. The recommendation is also highlighted in the central annotation editor.

One can now Annotate, Reject or Skip this recommendation in the Active Learning sidebar:

When using the Annotate, Reject or Skip buttons, the system automatically jumps to the next suggestion for the user to inspect. However, at times it may be necessary to go back to a recently inspected suggestion in order to review it. The History panel shows the 50 most recent actions. Clicking on the text of an item loads it in the main annotation editor. It is also possible to delete items from the history, e.g. wrongly rejected items.

The history panel displays whether a given suggestion was accepted, corrected or rejected, but this information can only be indicative. It represents a snapshot of the moment where the user made the choice. As the recommender is continuously updated by the system, the suggestions constantly change. It may happen that a suggestion which is shown as rejected in the sidebar is at a later time not even generated anymore by the recommender. Thus, deleting an item from the history will not always cause the suggestion from which it was generated to reappear. Resetting a document also clears the Active Learning history.

INCEpTION allows the user to create annotations as usual in the main annotation editor panel, even when in an Active Learning session. However, there is only a limited interaction with actions performed in the main annotation editor. If a suggestion is accepted or rejected in the main annotation editor, this is recorded in the history. However, if a user manually creates an annotation which causes a suggestion to disappear by overlapping with it, the history does not record this as a correction. For example, if the system generates a suggestion for Paul. (including the final sentence punctuation) but the user manually creates an annotation only for Paul (without the punctuation), the system does not recognize it as a correction.

Accepting/correcting, rejecting and skipping a suggestion in the sidebar cause the main annotation editor to move to the next suggestion. However, when a suggestion is accepted or rejected via the main editor, the generated annotation is opened in the annotation detail editor panel on the right side and the editor does not move to the next suggestion. For actions made in the main editor, it is assumed that the user may want to perform additional actions (e.g. set features, create more annotations in the vicinity) - jumping to the next suggestion would interfere with such intentions. That said, the next suggestion is loaded in the active learning sidebar and the user can jump to it by clicking on the suggestion text in the sidebar.

When removing an accepted/corrected item from the history and the annotation which was generated from this item is still present (i.e. it has not been deleted by other means), the user is asked whether the associated annotation should also be deleted.

Suggestions that are skipped disappear at first. However, once all other suggestions have been processed, the system asks whether the skipped suggestions should now be reviewed. Accepting will remove all skipped items from the history (even those that might no longer be visible in the history because of its on-screen size limit).

Concept Linking is the task of identifying concept mentions in the text and linking them to their corresponding concepts in a knowledge base. Use cases of Concept Linking are commonly found in the area of biomedical text mining, e.g. to facilitate understanding of unexplained terminology or abbreviations in scientific literature by linking biological entities.

Linking text to images can be useful e.g. when dealing with OCRed data or with text describing images. To support such cases, INCEpTION supports image features. Image features can be to annotation layers just like and other type of features. When selecting an annotation containing an image feature, the a text field is used as the feature editor. Enter an image URL into this field in order to link the annotation to an image. It is presently not possible to upload images to INCEpTION - the image must be accessible via an URL, e.g. from an IIIF server.

Open the images sidebar to get an overview over all images linked to any of the annotations currently visible on screen.

When navigating to the Curation Page, the procedure for opening projects and documents is the same as in Annotation. The navigation within the document is also equivalent to Annotation.

The table reflects the state of the document. A document can be in-progress, finished, curation-in-progress or curation-finished.

On the left, there is a sidebar titled Units, an overview of the chosen document is displayed. Units are represented by their number inside the document. Click on a unit in order to select it and to to edit it in the central part of the page.

The units are shown using different colors that indicate their state. Since the calculation of the state can take significant time, it is not updated as changes are made in the main editor pane. To update the coloring of the unit overview, use the Refresh button. When switching between documents, the overview is automatically refreshed.

The center part of the annotation page is divided into the Annotation pane which is a full-scale annotation editor and contains the final data from the curation step.

Below it are multiple read-only panes containing the annotations from individual annotators. Clicking on an annotation in any of the annotator’s panes transfers the respective annotation to the Annotation pane. There is also a small state icon for each annotator. If you click on that icon, you can change the state, e.g. from finished back to in progress. Note if you do that, the respective annotators document will no longer be available for curation. When the last finished annotation for a document is reopened, you will be forced to leave curation.

When a document is opened for the first time in the curation page, the application analyzes agreements and disagreements between annotators. All annotations on which all annotators agree are automatically copied to the Annotation pane. Any annotations on which the annotators disagree are skipped.

The annotator’s panes are color-coded according to their relation with the contents of the Annotation pane and according to the agreement status. The colors largely match the colors also used in the status over in the left sidebar.

Left-click on an annotation in one of the lower panels to merge it. This action copies the annotation to the upper panel. The merged annotation will turn green in the lower panel from which it was selected. If other annotators had a conflicting opinion, these will turn red in the lower panels of the respective annotators.

Right-click on an annotation in the lower panels to bring up a menu with additional options.

INCEpTION supports several different strategies for pre-merging data from the annotators to the curated document. The default strategy is Merge completely agreeing non-stacked annotations, but this default can be changed by the project manager in the project settings. It is also possible to update the default settings from the Re-merge dialog on the curation page.

By default, the curator can see the annotators names on the curation page. However, in some cases, it may not be desirable for the curator to see the names. In this case, enable the option Anonymous curation in the project detail settings. Users with the curator role will then only see an anonymous label like Anonymized annotator 1 instead of the annotator names. Users who are project managers can still see the annotator names.

Use static assignment if annotators should be able to freely choose which documents they want to annotate in which order and/or if you want to precisely control which annotator should be able to access which document.

To enable the static assignment workload manager, go to the Workload tab in the project settings.

In this mode, the workload management page allows you to monitor the progress of your annotation project. It also allows you to change the status of the annotation and curation documents. This allows you:

Use dynamic assignment if you want to get your documents each annotated by a certain number of annotators and do not care about which annotator annotates which documents.

To enable the dynamic assignment workload manager, go to the Workload tab in the project settings.

When dynamic assignment is enabled, annotators can no longer actively choose which documents they want to annotate. Any functionality for opening a particular document or switching between documents on the annotation page are disabled. When the annotator opens the annotation page, a document is automatically selected for annotation. The only way to progress to the next document is by marking the current document as finished.

The dynamic workload management page gives the project manager a fast overview of all documents and users within the current project. Additionally, the automatic distribution of documents to annotators can be modified.

Therefore, it mainly consists of a substantial, but easy to understand table containing the data a project manager needs from their documents. This data is represented by a unique row for each individual document in the project. The following columns with the respective data are displayed:

You can also configure display and workload settings using the three buttons on the top left corner of the table: Filter, Annotators and Settings.

Finally, also a small quick filter is integrated to the workload page on the top right corner. Upon selecting different states, the table will be filtered towards these in real time. These states are the same as the ones represented in the first column State. As default, all states will be shown in the table.

Overall, the workload feature shall support annotation projects in their organization. Thanks to the table, the filtering and the options for the annotation workflow and the annotators, the project manager now has more flexibility and insight in his projects' progress. Also,the redesigned annotation flow ensures better results from the annotations, as the data will be better distributed throughout the project.

Click on an annotator badge in the annotators column to cycle through the annotation states. Right-click on the badge for additional actions such as the option to reset the annotations.

Several agreement measures are supported.

Coding measures are based on positions. I.e. two annotations are either at the same position or not. If they are, they can be compared - otherwise they cannot be compared. This makes coding measures unsuitable in cases where partial overlap of annotations needs to be considered, e.g. in the case of named entity annotations where it is common that annotators do not agree on the boundaries of the entity. In order to calculate the positions, all documents are scanned for annotations and annotations located at the same positions are collected in configuration sets. To determine if two annotations are at the same position, different approaches are used depending on the layer type. For a span layer, the begin and end offsets are used. For a relation layer, the begin and end offsets of the source and target annotation are used. Chains are currently not supported.

Unitizing measures basically work by internally concatenating all documents into a single long virtual document and then consider partial overlaps of annotations from different annotations. I.e. there is no averaging over documents. The partial overlap agreement is calculated based on character positions, not on token positions. So if one annotator annotates the blackboard and another annotator just blackboard, then the partial overlap is comparatively high because blackboard is a longish word. Relation and chain layers are presently not supported by the unitizing measures.

When working with coding measures, there is the concept of incomplete annotations. For a given position, the annotation is incomplete if at least one annotator has not provided a label. In the case of the pairwise comparisons that are used to generate the agreement table, this means that one annotator has produced a label and the other annotator has not. Due to the way that positions are generated, it also means that if one annotator annotates the blackboard and another annotator just blackboard, we are actually dealing with two positions (the blackboard, offsets 0-15 and blackboard, offsets 4-14), and both of them are incompletely annotated. Some measurs cannot deal with incomplete annotations because they require that every annotator has produced an annotation. In these cases, the incomplete annotations are excluded from the agreement calculation. The effect is that in the (the) blackboard example, there is actually no data to be compared. If we augment that example with some other word on which the annotators agree, then only this word is considered, meaning that we have a perfect agreement despite the annotators not having agreed on (the) blackboard. Thus, one should avoid measure that cannot deal with incomplete annotations such as Fleiss' kappa and Cohen’s kappa except for tasks such as part-of-speech tagging where it is known that positions are the same for all annotators and all annotators are required (not expected) to provide an annotation.

The agreement calculations considers an unset feature (with a null value) to be equivalent to a feature with the value of an empty string. Empty strings are considered valid labels and are not excluded from agreement calculation. Thus, an incomplete annotation is not one where the label is missing, but rather one where the entire annotation is missing.

In general, it is a good idea to use at least a measure that supports incomplete data (i.e. missing labels) or even a unitizing measure which is able to produce partial agreement scores.

Multiple interpretations in the form of stacked annotations are not supported in the agreement calculation! This also includes relations for which source or targets spans are stacked.

To calculate the pairwise agreement, the measure is applied pairs of documents, each document containing annotations from one annotator. If an annotator has not yet annotated a document, the original state of the document after the import is considered. To calculate the overall agreement between two annotators over all documents, the average of the per-document agreements is used.

The lower part of the agreement matrix displays how many configuration sets were used to calculate agreement and how many were found in total. The upper part of the agreement matrix displays the pairwise agreement scores.

Annotations for a given position are considered complete when both annotators have made an annotation. Unless the agreement measure supports null values (i.e. missing annotations), incomplete annotations are implicitly excluded from the agreement calculation. If the agreement measure does support incomplete annotations, then excluding them or not is the users' choice.

The knowledge base page provides a concept tree hierarchy with a list of instances and statements, together with the list of properties as shown in the figure below. For local knowledge bases, the user can edit the KB contents here, which includes adding, editing and deleting concepts, properties, statements and instances.

The knowledge base page provides the specific mentions of concepts and instances annotated in the text in the Mentions panel which integrates the knowledge base page with the annotated text.

The concept tree in this page is designed using the subClass relationship for the configured mapping. Each concept associates itself with a list of instances (in case it has one) on the Instance panel which appear when we click on a specific concept along with the Mentions of the concept in the annotated text. The click on a specific instance shows the panel for the list of statements associated with the instance along with Mentions of the instance in the annotated text. In the left bottom side of the page, it lists the set of properties from the knowledge base. Clicking on the property showcases the statements associated with the property such as labels, domains, ranges, etc.

In case the user has the privilege to edit the knowledge base, the user may add statements for concepts, instances and properties.

INCEpTION allows the user to edit local knowledge bases. This includes adding statements or subclassing concepts and their instances.

In order to create a statement for a particular knowledge base entity, the Create Statement can be used.

When creating a new statement about an instance, a list of available properties is shown. After selecting the property of choice, the object of the statement has to be specified. The possible properties for a given subject are restricted by domain the domain of property, i.e. the property born_in would need an instance of human as the subject.

The same is true for the object of a statement: After choosing the property for a concept, the object has to be specified. The possible objects are limited by the range of the property if given. Right now, four different editors are available to specify features for:

Concept features are features that allow referencing concepts in the knowledge base during annotation.

To create a new concept feature, a new feature has to be created under Projects Settings → Layers. The type of the new feature should be KB: Concept/Instance/Property. Features of this type also can be configured to either take only concepts, only instances, only properties or either (select any).

When creating a new annotation with this feature, then the user is offered a dropdown with possible entities from the knowledge base. This dropdown is then limited to only concepts or features or both when selecting the respective filter in the feature configuration.

The scope setting allows to limit linking candidates to a subtree of the knowledge base.

Here, you can import project archives such as the example projects provided on our website or projects exported from the Export tab.

When a user with the role project creator imports a project, that user automatically becomes a manager of the imported project. However, no permissions for the project are imported!

When a user with the role administrator imports a project, the user can choose whether to import the permissions and whether to automatically create users who have permissions on the imported project but so far do not exist. If this option to create missing users disabled, but the option to import permissions is enabled, then projects still maintain their association to users by name. If the respective user accounts are created manually after the import, the users will start showing up in the projects.

After clicking on the Users tab, you are displayed with a new pane in which you can add new users by clicking on the Add users text field. You get a dropdown list of enabled users in the system which can be added to the project. Any users which are already part of the project are not offered. As you type the dropdown list with the users is filtered to match your input. By clicking on a username or by pressing enter you can select the corresponding user. You can keep typing to add more users to the project. When you press the Add button the selected users are added to your project.

By default, the users are added to the project as annotators. If you want to assign additional roles, you can do so by clicking on the user and then on Permissions pane select the appropriate permissions.

After ticking the wished permissions, click on Save. To remove a user, remove all the permissions and then click on Save.

The documents in a project can be managed on the documents panel.

To upload one or more documents, use the folder icon in the Files to import field. A browser dialog will open which allows you to navigate to some folder on your computer, select files, and then upload them. Typically, you can select multiple files in this dialog either by holding the control key on your keyboard then then selecting them one-by-one with the mouse - or by clicking on the first file, then holding shift on the keyboard and then clicking on the last file - thereby selecting all files in between the two. Note that if you upload multiple files, they must all have the same format.

After selecting the files, use the Format dropdown to choose which format your files are in. A project can contain files in different formats.

Finally, use the Import button to upload the files and add them to the project.

To delete a document from the project, you have to click on it and then click on Delete in the right lower corner. Again, you can select multiple files for deletion using with the aid of the control or shift keys on the keyboard.

While it is possible to upload multiple documents at once, there are limits to how many documents can be uploaded in a single upload operation. For a start, it can take quite some time to upload thousands of documents. Also, the server configuration limits the individual file size and total batch size (the default limit is 100MB for both). Finally, browsers differ in their capability of dealing with large numbers of documents in an upload. In a test with 5000 documents of each ca. 2.5kb size including Chrome, Safari and Firebird, only Chrome (80.0.3987.122) completed the operation successfully. Safari (13.0.5) was only able to do upload about 3400 documents. Firebird (73.0.1) froze during the upload and was unable to deliver anything to the server. With a lower number of documents (e.g. 500), none of the browsers had any problems.

All annotations belong to an annotation layer. Each layer has a structural type that defines if it is a span, a relation, or a chain. It also defines how the annotations behave and what kind of features it carries.

Here the project manager can configure settings that affect the experience on the annotation page.

In the Projects Settings, switch to the Knowledge Bases tab, then click New… on the bottom and a dialogue box shows as in the figure below.

To create a local or remote knowledge base, one needs to choose Local or Remote for the type. For the reification, NONE is the default case, but to support qualifiers, one needs to choose WIKIDATA.

For the local KB, the user can optionally choose a RDF file from which to import the initial data. Alternatively, the user can skip the step to create an empty KB to create a knowledge base from scratch. It is also always possible to import data from an RDF file after the creation of a KB. It is also possible to multiple RDF files into the same KB, one after another.

For remote KBs, INCEpTION provides the user with some pre-configured knowledge base such as WikiData, British Museum, BabelNet, DBPediaa or Yago. The user can also set up a custom remote KB, in which case the user needs to provide the SPARQL endpoint URL for the knowledge base as in the figure below.

Recommenders provide annotation support by predicting potential labels. These can be either accepted or rejected by the user. A recommender learns from this interaction to further improve the quality of its predictions.

Recommenders are trained every time an annotation is created, updated or deleted. In order to determine whether the annotations are good enough, recommenders are evaluated on the annotation data. During recommender evaluation a score for each recommender is calculated and if this score does not meet the configured threshold, the recommender will not be used.

Recommenders can be configured in the Project Settings under the Recommenders tab. To create a new recommender, click Create. Then, the layer, feature and the classifier type has to be selected.

To manager the tagsets, click on the tab Tagsets in the project pane.

To edit one of the existing tagsets, select it by a click. Then, the tagset characteristics are displayed.

In the Frame Tagset details, you can change them, export a tagset, save the changes you made on it or delete it by clicking on Delete tagset. To change an individual tag, you select one in the list displayed in the frame Tags. You can then change its description or name or delete it by clicking Delete tag in Tag details. Please do not forget to save your changes by clicking on Save tag. To add a new tag, you have to click on Create tag in Tag details. Then you add the name and the description, which is optional. Again, do not forget to click Save tag or the new tag will not be created.

To create an own tagset, click on Create tagset and fill in the fields that will be displayed in the new frame. Only the first field is obligatory. Adding new tags works the same way as described for already existing tagsets. If you want to have a free annotation, as it could be used for lemma or meta information annotation, do not add any tags.

To export a tagset, choose the format of the export at the bottom of the frame and click Export tagset.

Here you can export the project for different purposes. Once an export process has been started, its progress can be see in the right sidebar. If an export takes very long, you can keep it running and check back regularly to see its state. You can even log out and log in later. Once the export is complete, you have 30 minutes to download it before it gets cleaned up automatically. Any user with project manager permissions can visit this page and view the exports. When a user cancels an export or downloads an export, it is removed from the list. If there are any messages, warnings or errors, you should inspect them before cancelling or downloading the export. While an export is running, only the latest messages is displayed and it can happen that messages are skipped. Once the export is complete (either successfully or failed), the full list of messages is accessible.

Project managers can generate invite links to their projects which allow users to easily join their project. For this, visit the Project Settings and click on Share Project. Clicking on Allow joining the project via a link will generate the invite link that can then be copied and given to users (e.g. via email).

The user can now follow the invite link by entering it into a browser. She might be prompted to log into INCEpTION and is then automatically added to the project with annotator rights and directed to the project dashboard page. She can now start annotating.

Project managers can create snapshots of all documents in the project as well as its layer configuration via the versioning panel. This is done via a git repository stored in the .inception folder. This git repository can also be used to push to a remote repository, e.g. saving on Github or Gitlab. We currently only support pushing via HTTPS.

If you want to roll back to an earlier version, then you need to manually check out the old version in the local or remote git repository, load the old layer configuration manually in the layer settings and replace source and annotation documents via the remote API (see the Admin Guide).

If document repositories have been configured in a project, the Search page becomes accessible through the project dashboard. On the top left of the search page you can select in a dropdown menu which document repository you want to query. All document repositories that were created in the project settings should be selectable here and are identified by their Name. The field next to it is the query text field in which the search queries are entered. After entering a query, search by pressing the Enter key or by a clicking on the Search button. The documents in the document repository which match the search query are returned as the search results and then shown as a table. The table displays 10 results at a time and more can be accessed through the paging controls which are located above the table. Depending on the repository, you may see a document title or ID, text snippets with highlights indicating matches of your query in the document, and a score which represents the relevance of the document to the query. If a document has not yet been imported into your project, there is an Import button which extracts the document from the repository and adds it to the project, thereby making it available for annotation. If the document has already been imported, there is an Open button instead. Clicking on the document title or ID opens a preview page where the document text can be viewed before importing it.

The external search functionality can be used in the sidebar of the annotation page as well and can be opened by clicking on the globe-logo on the sidebar at the left of the annotation page. It essentially offers the same functionality as the external search page accessible via the project dashboard. Being able to search directly from the annotation page may be more convenient though because the user does not have to keep switching between the search page and the annotation page. Additionally, clicking on a search result in the external search sidebar automatically imports the document into your project and opens it in the annotation view.

Document repositories can be added via the Document repository tab in the project settings.

To import a constraints file, go to Project and click on the particular project name. On the left side of the screen, a tab bar opens. Choose Constraints. You can now choose a constraint file by clicking on Choose Files. Then, click on Import. Upon import, the application checks if the constraints file is well formed. If they conform to the rules of writing constraints, the constraints are applied.

A constraint set consists of two components:

It is necessary to declare short names for all fully qualified names because only short names can be used when writing a constraint rule. Short names cannot contain any dots or special characters, only letters, numbers, and the underscore.

Scopes consist of a scope name and one or more rules that refer to a particular annotation layer and define restrictions for particular conditions. For example, it is possible to reorder the applicable tags for a POS layer, based on what kind of word the annotator is focusing on.

While scope names can be freely chosen, scope rules have a fixed structure. They consist of conditions and restrictions, separated by an arrow symbol (→). Conditions consist of a path and a value, separated by an equal sign (=). Values always have to be embraced by double-quotes. Multiple conditions in the same rule are connected via the &-operator, multiple restrictions in the same rule are connected via the |-operator.

Typically a rule’s syntax is

This leads to the following structure:

Both conditions and restrictions are composed of a path and a value. The latter is always enclosed in double quotes.

A condition is a way of defining whether a particular situation in INCEpTION is based on annotation layers and features in it. Conditions can be defined on features with string, integer or boolean values, but in any case, the value needs to be put into quotes (e.g. someBooleanFeature="true", someIntegerFeature="2").

A condition set consists of one or more conditions. They are connected with logical AND as follows.

A restriction set defines a set of restrictions which can be applied if a particular condition set is evaluated to true. As multiple restrictions inside one rule are interpreted as conjunctions, they are separated by the |-operator. Restrictions can only be defined on String-valued features that are associated with a tagset.

A path is composed of one or more steps, separated by a dot. A step consists of a feature selector and a type selector. Type selectors are only applicable while writing the condition part of a rule. They comprise a layer operator @ followed by the type (Lemma, POS, etc). Feature selectors consist of a feature name, e.g.

Navigation across layers is possible via

Hereby all annotations of type <shortLayerName> at the same position as the current context are found.

The constraint language supports block comments which start with / and end with /. These comments may span across multiple lines.

To specify a list of repairs or checks in the settings.properties file, use the following syntax:

To switch to the PDF editor for an opened document, click on Settings in the Document panel located at the top. In the section General Display Preferences select PDF for the Editor field. Save your settings. The PDF editor will open.

Once the editor is loaded you can see the opened PDF document. To navigate through the document you can hover your mouse over the PDF panel and use the mouse wheel for scrolling. After clicking in the PDF panel it is also possible to use the Up and Down keys of the keyboard. For a faster navigation the Page Up and Page Down keys can be used. In the PDF panel on the top left there are buttons for switching to the previous or next page in the document. Next to the buttons you can enter a page number to jump to a specific page.

In the top center of the PDF panel the zoom level of the document can be adjusted.

The button on the top right in the PDF panel opens a small menu which provides functionality to go to the first or last page. You can also use the Home and End keys instead. The menu also contains an option to enable the hand tool to navigate through the document via clicking and dragging the pages.

When moving through the document annotations will not show immediately. Once the movement stops for a short period of time the annotations for the previous, current and next page will be loaded. The loading process might take a few seconds.

To create a span annotation first select the desired layer for it. This can be done in the Layer box on the right sidebar. If another annotation is already selected press the Clear button on the right sidebar.

Once you have chosen a layer, select the desired text to create a span annotation. This can be done by clicking at the beginning of the text span, dragging until the end and then releasing the mouse button. In the upper right corner you can see a moving circle which indicates that the creation of the annotation is in process.

The creation might take a few seconds. Once finished the new span annotation will be rendered if it was created successfully. If it was not possible to create the annotation an error message is shown. After the span annotation is created it will be automatically selected in the Layer box.

To create a relation annotation click on the knob of a span annotation and drag and drop on another span annotation knob. In order to create a relation annotation between two spans an according layer must exist.

After releasing the mouse button the creation process starts which is indicated by a moving circle in the upper right corner. This might take a few seconds. Once finished the new relation annotation will be rendered if it was created successfully. If it was not possible to create the annotation an error message is shown. After the relation annotation is created it will be automatically selected in the Layer box.

Currently long distance relation annotations are not creatable as annotations are rendered dynamically when moving through the pages.

Span annotations can be selected by clicking on their span annotation knob.

To select relation annotations click on the relation arc.

To modify an existing span or relation annotation you first need to select it.

Once an annotation is selected it will be shown in the Layer box.

You can now edit the selected annotation.

First select the annotation that will be deleted.

The selected annotation will be shown in the Layer box.

To delete the annotation click on the Delete button on the right sidebar.

To accept and convert a recommendation to an actual span annotation select it.

The recommendation is converted to an actual annotation and is selected automatically.

To reject a recommendation quickly double click the span annotation knob.

The recommendation then will be removed from the editor.

INCEpTION uses the Java BreakIterator internally. Note that the linked file is part of a specific version of OpenJDK 11 and may change in other Java versions or for other Java vendors.

If you need to provide your own tokenization, then the best choice would be to use a format that supports it, e.g. XMI, WebAnno TSV or CoNLL formats.

The is no immediate support for discontinuous spans in INCEpTION.

However, you can emulate them using either a relations or link features.

You can define a relation layer on top of your span layer. When you have multiple spans that should be considered as one, you can use a relation to connect them.

Or you can add a Link: XXX feature to your span layer which either points to the same layer or which points to a new layer you might call e.g. Extension.

So when you have a discontinuous span, you could annotate the first span with your normal span layer and then add one or more links to the other spans.

INCEpTION is the successor of WebAnno and evolved from the WebAnno code base. Both INCEpTION and WebAnno are currently developed/maintained by the same team at the UKP Lab at the Technical University of Darmstadt.

INCEpTION has all the flexibility and many more exciting features including a completely new human-in-the-loop annotation assistance support, the ability to search texts and annotations, support for RDF/SPARQL knowledge bases for entity linking, and much more. And best: it can import your WebAnno annotation projects (Projects of type automation or correction are not supported).

A sentence-oriented presentation of the text using inter-linear annotations. This editor is useful for texts that have been externally segmented into sentences. It supports rendering span annotations, relations and link features. The editor uses a heavily improved version of the rendering engine of brat.

A line-oriented presentation of the text using inter-linear annotations. This editor is useful for texts formatted using line breaks. It supports rendering span annotations, relations and link features. The editor uses a heavily improved version of the rendering engine of brat.

A line-oriented presentation of the text using inter-linear annotations that also wraps lines longer than 120 characters. This editor is useful for texts using consecutive line breaks mainly to indicate paragraph boundaries but that do not use line breaks within paragraphs. It supports rendering span annotations, relations and link features. The editor uses a heavily improved version of the rendering engine of brat.

This editor renders documents imported using a XML/HTML-based format such as 🧪 MHTML (Web archive) or 🧪 HTML. It is build on top of Apache Annotator. It supports rendering span annotations but not relations or link features.

This editor renders documents imported using a XML/HTML-based format such as 🧪 MHTML (Web archive) or 🧪 HTML. It is build on top of AnnotatorJS. It supports rendering span annotations but not relations or link features.

This editor renders documents imported using a XML/HTML-based format such as 🧪 MHTML (Web archive) or 🧪 HTML. It is build on top of RecogitoJS. It supports rendering span annotations, relations and link features.

This editor allows annotating PDF documents. It is based on PDF.js (version 2.x) and on the INCEpTION PDF format support. The editor supports rendering span annotations, relations and link features. The editor uses a heavily improved version of the rendering engine of PDFAnno.