On 17 and 24 November 2020 the InvenioRDM project partners met in two telecons to discuss the Metadata Reference document.

These conversations were a follow-up to the community's review of the document. Feedback and questions were collected in a Google Doc prior to the conversations. The conversations helped to confirm many answers to questions about the data model that had been posed in the Doc, such as: whether ‘role’ is required for Creators (no - only for Contributors), are Resource Types customizable (yes), and which file is shown in the previewer when a record has multiple files (the first, unless otherwise specified). See the Doc for responses to additional questions.

Use Cases Requested

For some fields, further feedback is needed in order to make them as useful for the community as possible. In particular, your use cases are requested for the following scenarios:

1. Adding a Language tag to Titles
2. Records with more than one Publisher (making Publisher a repeatable field)
3. Having a built-in “Unknown” option for Contributor type and/or role
4. Having more than one Identifier per Affiliation

Please post your use cases back to the top of the original Google Doc or to the new Metadata Interest Group topic at: https://invenio-talk.web.cern.ch (more on this below).

Subjects

In addition, the topic of Subjects was brought up for further discussion. The Subjects example in the Metadata Reference document shows the incorporation of one controlled vocabulary, the Medical Subject Headings, into the schema.

However, there are many controlled vocabularies that we can potentially map to the Subjects field. Which will be most useful to incorporate for the wider community, and what are potential use cases?

This is one of the first issues we would like to discuss in the Metadata Interest Group, which is set to launch in January 2021. This group welcomes the contributions and insights of developers, users, administrators, decision-makers, and anyone else at participating organizations with metadata experience, interest, and insights. Through the end of 2020 we will work to publicize the group and recruit members. If you’re interested in joining, please either sign up here: https://invenio-talk.web.cern.ch/t/metadata-interest-group/144 or DM me on Discord (see the Welcome channel).

Community Outreach

We are committed to serving the entire InvenioRDM community as our family of adopters continues to grow and thrive. This is the first, we hope, of many blog posts that will address topics of interest to the community at large, including developers, administrators, curators, and decision-makers. If there is a topic you would like us to explore, or perhaps even contribute a post on yourself, we’d like to know. If the topic would be best covered at a community-focused telecon (also on the horizon for the coming months!), we’d like to know that as well. Please contribute any and all ideas at my contact listed above.

Take care and stay safe!

We’re excited to announce the InvenioRDM Alpha 13 (November release)!

What’s new?

Files support

It is now possible to upload files on the draft form and publish them.

File uploader

When creating a new draft, you can now also upload files that will be included in the published record. The file uploader supports:

• Drag-n-dropping files to be uploaded
• Concurrent uploads (by default up to 4)
• Setting the default preview file
• Completely disabling files to allow for metadata-only records

New REST API

Big part of our efforts went into designing a new files REST API that will allow InvenioRDM to easily support upload workflows depending on third-party storage systems (e.g S3).

You can check out the documentation of these endpoints in our newly added REST API reference.

Files listing and preview

Uploaded files are also listed and can be previewed on the record's landing page, using a variety of extensible previewers. We currently support preview of PDFs, images, ZIP files, CSV tables, and some text-based formats like XML, JSON, and Markdown.

By default, files are listed in alphabetical order and the first previewable file is shown in the preview box. You can specifically set which file you want to be by default previewed in the record's draft form.

Reorganized CLI commands

Thanks to the community's feedback, in this release we have also updated the existing CLI commands, with the goal of improving the development and operational workflows of an InvenioRDM instance. Our newly added CLI reference documentation lists and describes in detail the command groups, their sub-commands, and the options they support.

Make sure to go through our documentation, since many old commands have now new options and different default behavior (e.g. service setup will now also create demo records).

Feedback

As always, we welcome your feedback. When you provide feedback on Discourse your message should be pre-populated with the classic template (bugs, what worked well, what didn’t work well, wishes for documentation).

Here is the template to give feedback if it’s not automated:

## Bugs

## What worked well

## What didn't work well

## Wishes for documentation

Take care and stay safe!

We're happy to announce the InvenioRDM October release. Thank you to our team members for their efforts on this release.

In early October we held a virtual project workshop. As part of the workshop we have updated the roadmap. The roadmap is now publicly available in two versions, one simplified and one for project tracking. See below:

1- Simplified

2- GitHub for project tracking

What's new?

Metadata Schema

The entire metadata schema for bibliographic records has been updated, and is now ready for a thorough review by all project partners. The update includes changes to the JSONSchema, Elasticsearch mappings, the REST API data validation layer as well as addition of many fields to the deposit form.

See (https://inveniordm.docs.cern.ch/reference/metadata/) for for a full reference of the new metadata schema.

User Experience

We have done a lot of work on improving the UX of the primary pages such as the frontpage, search results and record landing page. Below you can see some screenshots of before/after:

Search results

The search results have been tightened up as well. Also, we’ve added support for nested facets, so you can expand broader categories into subcategories (e.g., Publication and Image).

Landing page

The landing page you’ll notice now has Edit and New version buttons (only the Edit button currently works). The right column has been tightened up, and most of the new metadata fields are now properly displayed on the landing page.

Human readable labels

In the facets you’ll notice we now have human readable labels. Instead of 'publication' it will say 'Publication'; instead of 'open' it will say 'Open Access'.

Similarly to search results and landing pages, you’ll now see the correct resource type as well as icons on Open Access.

Localization

Dates, like the publication date which supports Extended Date Time Format is now properly localized as well using the Unicode Common Locale Data Repository. For instance here an English and Turkish localization:

Affiliations

The affiliation display was redone:

Manage section

There’s a new manage section on the record landing page, which allows you to edit a record.

CLI Improvements

We have made it easier to get started with developing InvenioRDM. Partially to make our own developers' life easier, but also to make it easier to customize InvenioRDM.

Shells

You can now easily activate the Python virtualenv shell as well start a Python terminal from your instance using the following commands:

invenio-cli shell
invenio-cli pyshell

Watching assets

We have now simplified how you can change styling via automatic watching of file changes on assets. Previously you had to manually rebuild the assets. Now instead you can simply execute the following commands, and the styling will automatically rebuild once the file changes:

invenio-cli assets --force --development
invenio-cli assets watch

Develop an Invenio module

Developers often need to install the latest development versions of Invenio modules to work on them. This can now easily be done with:

invenio-cli ext module-install ~/src/invenio-app-rdm ~/src/invenio-rdm-records
invenio-cli assets --force --development

Develop a React module

The above works only for Python packages. If instead, you are working on one of our React libraries, you can now easily install and watch the module for changes as well:

invenio-cli assets watch-module --link ~/src/react-invenio-deposit

Install

If you previously installed InvenioRDM, make sure you have the latest Docker image of your choice according to the Python version:

docker pull inveniosoftware/centos7-python:3.6
docker pull inveniosoftware/centos8-python:3.7
docker pull inveniosoftware/centos8-python:3.8

To install:

pip install --upgrade invenio-cli
invenio-cli init rdm
cd my-site
invenio-cli containerize --pre
invenio-cli demo --containers

To stop the instance without destroying the records that were created, you can just run:

cd my-site
invenio-cli stop

To destroy the Python virtualenv, and remove the docker containers, you can now just run:

cd my-site
invenio-cli destroy

Feedback

As always, we welcome your feedback. When you provide feedback on Discourse your message should be pre-populated with the classic template (bugs, what worked well, what didn’t work well, wishes for documentation).

Here is the template to give feedback if it’s not automated:

## Bugs

## What worked well

## What didn't work well

## Wishes for documentation

Take care and stay safe!

We are glad to announce InvenioRDM Alpha 11 (September release)!

What's new?

We now have support for drafts, meaning you can now start a deposit, save it, and come back many days later to publish it.

This simple change marks a very big milestone for the backend of InvenioRDM. During the past months we have been revamping the entire data and control flow of the backend. The flow is what is supporting the core of InvenioRDM - submitting, editing and searching for records - and it determines what is possible and what's not possible in InvenioRDM. These changes enable the customizations needed by partners including customizations to permissions, publishing workflows, metadata fields, indexing and serializations just to name a few.

A high-level overview of the new flow is shown below:

This new flow is implemented in the following Invenio modules:

• Flask-Resources
• Invenio-Records-Resources
• Invenio-Drafts-Resources

Install (TL;DR)

If you previously installed InvenioRDM, make sure you have the latest Docker image of your choice according to the Python version:

docker pull inveniosoftware/centos7-python:3.6
docker pull inveniosoftware/centos8-python:3.7
docker pull inveniosoftware/centos8-python:3.8

To install:

pip install --upgrade invenio-cli
invenio-cli init rdm
cd my-site
invenio-cli containerize --pre
invenio-cli demo --containers

To destroy the Python virtualenv, and remove the docker containers, you can now just run:

cd my-site
invenio-cli destroy

It's the first feature by Rodrigo Almeida, the newest addition to our team!

We are happy to announce InvenioRDM Alpha 10 (August release)! Thank you to our team members for their efforts on this release.

What's new?

We made some limited changes this month. However, three key areas were tackled.

• The concurrent Invenio Sprint brought the final Semantic-UI updates across all modules! If something does not show up right in your default RDM instance, then it's a legitimate bug now.
• These UI updates complement the updates to Search. The record search page now uses the new API endpoint from the last release. Customization of the search results is back (with documentation forthcoming). Different search pages are easier to set up.
• More library improvements across the board were merged.

Update invenio-cli to version 0.16.0 and follow the updated documentation to get started.

Semantic UI transition wrap-up

One of the many pages that were transitioned to Semantic-UI

The transition to Semantic-UI is for all intents and purposes complete. Future additions will use this framework and have a consistent look with the rest of InvenioRDM. Close to 15 styling issues were closed in the process.

Search customization and transition to new API

The search page now uses the new API (/api/rdm-records endpoint) we introduced in the last release. The new endpoint supports pagination and sorting, but aggregations were disabled to focus on the transitioning only.

Aggregations will be re-enabled soon.

Custom styling of search results is possible again: documentation to explain the new way is pending however.

Library Improvements

Finally, various improvements and fixes were made across modules including:

• an overhaul to how links are generated in API responses,
• a way to hook callbacks into action endpoints,
• the convergence of validation for drafts and records, and
• the introduction of record versioning behind the scenes (which will be made more visible in coming releases).

Some of these changes are visible in the API responses:

{
    "links": {
        "self": "https://127.0.0.1/api/rdm-records/jnmmp-51n47",
        "self_html": "https://127.0.0.1/records/jnmmp-51n47",
        "files": "https://127.0.0.1/api/rdm-records/jnmmp-51n47/files",
        "edit": "https://127.0.0.1/api/rdm-records/jnmmp-51n47/draft"
    },
    "metadata": {
         "conceptrecid": "5fk5g-mq814",
         ...
    },
    ...
}

What do you need to do?

You can install the latest invenio-cli v0.16.0, create a new instance and see what it looks like. Make sure to follow the updated documentation - there are a couple of caveats to be made aware of.

Install (TL;DR)

If you previously installed InvenioRDM, make sure you have the latest Docker image of your choice according to the Python version:

docker pull inveniosoftware/centos7-python:3.6
docker pull inveniosoftware/centos8-python:3.7
docker pull inveniosoftware/centos8-python:3.8

To install:

pip install --upgrade invenio-cli
invenio-cli init rdm
cd my-site
invenio-cli containerize --pre
invenio-cli demo --containers

To destroy the Python virtualenv, and remove the docker containers run:

cd my-site
pipenv --rm
docker-compose -f docker-compose.full.yml down

Feedback

As always, we welcome your feedback. When you provide feedback on Discourse your message should be pre-populated with the classic template (bugs, what worked well, what didn't work well, wishes for documentation).

Here is the template to give feedback if it's not automated:

## Bugs

## What worked well

## What didn't work well

## Wishes for documentation

Take care and stay safe! The next release will be big.

We are happy to announce InvenioRDM Alpha 9 (July release)! Thank you to our team members for their efforts on this release.

What's new?

The July release adds translation support and starts integrating the major backend development of last release into InvenioRDM. In particular, draft functionality has been added at the API level and is used under the hood for deposits. To achieve this integration, the hard problems like responsibility separation, error handling, pagination, linking, and internal PID management were solved. Other improvements and more details follow.

Update invenio-cli to version 0.15.0 and follow the documentation to get started.

Module translation

Thanks to the efforts of TÜBITAK, InvenioRDM was set up for internationalization: all text has been registered for translation and the Transifex service has been enabled. The team is currently at work on a VSCode extension to help future translators. Turkish was added as a first alternative language! How to go about translating and using the various tools will be laid out next month once we have more experience under our belt!

Library improvements

Error handling, default MIME type headers, and uniform body + querystring deserialization have been added to Flask-Resources, our Invenio-agnostic Flask REST library.

Note that invenio-resources is now invenio-records-resources (it includes the former invenio-records-agent) and invenio-drafts-resources is the new package that houses the draft functionality.

Behind the scenes, persistent identifiers have also seen some work.

New API integration (including draft functionality)

The deposit page now uses the new API to create records by creating drafts and publishing them immediately. In the next release, we hope you will be able to save and publish records separately. That being said, you can already do so on the command-line! Make sure to follow the updated documentation to see creation, publication, retrieval, and search in action there.

In addition to these tent-pole features, Python 3.7 support and general containerization was further improved thanks to Cottage Labs through an updated base image.

Community domains vocabulary

The list of available domains/disciplines that can be added to a community is now based on a pre-defined (and configurable) vocabulary.

What do you need to do?

Follow the documentation site: https://inveniordm.docs.cern.ch/ and install the latest invenio-cli v0.15.0, create a new instance and see what it looks like!

Install (TL;DR)

If you previously installed InvenioRDM, make sure you have the latest Docker image of your choice according to the Python version:

docker pull inveniosoftware/centos7-python:3.6
docker pull inveniosoftware/centos8-python:3.7
docker pull inveniosoftware/centos8-python:3.8

To install:

pip install --upgrade invenio-cli
invenio-cli init rdm
cd my-site
invenio-cli containerize --pre
invenio-cli demo --containers

To destroy the Python virtualenv, and remove the docker containers run:

cd my-site
pipenv --rm
docker-compose -f docker-compose.full.yml down

Feedback

As always, we welcome your feedback. When you provide feedback on Discourse your message should be pre-populated with the classic template (bugs, what worked well, what didn't work well, wishes for documentation).

Here is the template to give feedback if it's not automated:

## Bugs

## What worked well

## What didn't work well

## Wishes for documentation

Take care and stay safe!

We are happy to announce InvenioRDM Alpha 8 (June release). Thank you to our team members for their efforts on this release.

What's new?

The June release is a major backend development release. We developed three new modules that lay the foundation for the new core data flow of Invenio(RDM). Our work goes far to provide a better developer experience. You will be able to work on and customize Invenio(RDM) in an easier and cleaner way.

The primary focus for the release was:

• Continue the migration from Bootstrap to SemanticUI
• Bug fixes and improvements on the frontend (mainly on deposit)
• Develop new core modules for Invenio(RDM) backend

UI Customization

You are able to customize your templates again. The JSX customizations are not functional yet though, which means the search page results are not customizable.

Visual appearance

We have migrated the file previewer, the OAuth login, and the settings pages to SemanticUI.

You might still notice some issues related to visual appearance (e.g. the name previewed file is not in a panel as before). We will be working on making all this perfect in the next sprint on SemanticUI (end of August). In the meantime, it is, at least, better.

Creators, Contributors and Affiliations Identifiers

The Deposit page now accepts ORCID iDs for people (creators and contributors) and ROR identifiers for organizations (creators, contributors and affiliations). An icon with a link to the respective entity on the record landing page reflects this connection.

Contributors are now optional on the deposit page as well.

Python versions

There are now Invenio images for Python 3.7 and Python 3.8! This means that you can use Python 3.6, 3.7 and 3.8 for local development and the full containerized mode.

Backend and REST APIs

We have put a significant amount of work in refactoring some of the core API flow of Invenio. This addresses many of the issues that Invenio was facing in this domain. As a result, three modules were created:

• https://github.com/inveniosoftware/flask-resources: A generic Flask module to implement REST APIs and make their customization easy.
• https://github.com/inveniosoftware/invenio-resources: A module customizing Flask-Resources for Invenio(RDM) APIs.
• https://github.com/ppanero/invenio-records-agent: An experimental module to bridge the representational and data access worlds in Invenio. This module hosts the business logic and might be merged back into invenio-resources.

What do you need to do?

Follow the documentation site: https://inveniordm.docs.cern.ch/ and install the latest invenio-cli v0.14.2+, create a new instance and see how it looks like!

Install (TL;DR)

If you previously installed InvenioRDM, make sure you have the latest Docker image of your choice according to the Python version:

docker pull inveniosoftware/centos7-python:3.6
docker pull inveniosoftware/centos8-python:3.7
docker pull inveniosoftware/centos8-python:3.8

To install:

pip install invenio-cli --upgrade
invenio-cli init rdm
cd my-site
invenio-cli containerize --pre
invenio-cli demo --containers

To destroy the Python virtualenv, and remove the docker containers run:

cd my-site
pipenv --rm
docker-compose -f docker-compose.full.yml down

Feedback

As always, we welcome your feedback. We are experimenting with Discourse's built in post template. This means when you provide feedback on Discourse your message should be pre-populated with the classic template:

## Bugs

## What worked well

## What didn't work well

## Wishes for documentation

Thank you for your patience and support. We are working hard all the time to improve things. Take care and stay safe!

We are happy to announce InvenioRDM Alpha 7 (May release). Thank you to our team members for their efforts on this significant release.

What's new?

The May release is a major integration release, with a few rough edges that still need polishing. We have integrated a lot of code that was developed in separate branches into the main InvenioRDM code base. We have done this now, to ensure we have enough time to iron out integration issues.

The primary focus for the release was:

• Migrate from Bootstrap to SemanticUI
• Deposit form frontend (JavaScript)
• Integrate new communities prototype.
• Upgrade to Invenio v3.3

Known issues (please read!)

As a rule of thumb, many features will be broken and not work. Thank you for your continued patience and feedback as we work to make improvements.

A lot of code has changed in this release, and many features have not yet undergone quality control.

UI Customizations not working

The existing method for customizing the UI will change slightly, We hope to provide documentation for this in the next release.

Visual appearance

You'll notice issues related to the visual appearance, that we have not yet had the time to refine. This is with regards to alignment, colors, behaviours, font-sizes, and user experience.

Data model and deposit form

The data model and deposit form is far from complete. We have not yet focused on implementing the right data model with the right fields. For instance the access right displayed in the deposit form will change, as will many of the other fields displayed.

Communities

Communities is in a very initial state and many things will change.

Previewer, OAuth login, Settings pages

The file previewer and OAuth login still need to be migrated to SemanticUI, and thus the modules are likely not to work properly in this release.

What do you need to do?

You can install the latest invenio-cli v0.14.x, create a new instance from the beginning and see how it looks like!

Install (TL;DR)

If you previously installed InvenioRDM, make sure you have the latest Docker image:

docker pull inveniosoftware/centos7-python:3.6

To install:

pip install invenio-cli --upgrade
invenio-cli init rdm
cd my-site
invenio-cli containerize --pre
invenio-cli demo --containers

To destroy the Python virtualenv, and remove the docker containers run:

cd my-site
pipenv --rm
docker-compose -f docker-compose.full.yml down

Communities

To see the communities click "Communities" menu, then "New community". You'll be prompted to log in. Simply create a new account, and log in with it.

Deposit form

To see the deposit click "Uploads" then "New upload"

Want to get involved?

Follow the documentation site: https://inveniordm.docs.cern.ch/ for an up-to-date step-by-step install and usage of InvenioRDM.

Give us feedback on Discourse: https://invenio-talk.web.cern.ch/t/inveniordm-alpha-7-may-release/109 for this release.

Thank you for your patience. We are working hard all the time to improve things. Take care and stay safe!

We are proud to announce the release of Invenio v3.3.0. With this release, Invenio v3.1.x reaches end of life, and will no longer be maintained.

Python compatibility

Invenio v3.3 supports Python versions 3.6 and 3.7. Python 2 support in Invenio ended on January 1st, 2020 with the official end of life for Python 2 on the same date.

Getting started

See our quick start guide.

Upgrading

See our upgrade guide.

Release notes

Please see the full release notes for details about minor changes, deprecations etc.

Highlights for Invenio v3.3

Python 3.7

Python 3.7 is now supported by Invenio!

Improved support for Single Page Applications

Invenio v3.3 improves the support for Single Page Applications (SPA) by adding REST APIs for account management operations such as login, logout, user registration, password change, email confirmation and more. The integrated OAuth client also adds a new REST API so that you can login via your OAuth providers such as GitHub, ORCID and Globus.

CSRF Protection

Invenio-REST adds a CSRF middleware, called CSRFProtectMiddlewate to protect API views against CSRF attacks. The CSRF checks can be skipped in REST API calls when using a personal OAuth API token.

Dependency management

We have revamped management of third-party dependencies. New releases of some of the many third-party packages Invenio depends on could often cause dependency conflicts. Invenio v3.3 introduces the concept of coordinator packages, who are responsible for properly specifying third-party dependencies.

The release notes contain a table with all the current coordinator modules.

Helm chart (BETA)

Helm-Invenio provides a helm chart to deploy an Invenio application along with all its required services: Redis, RabbitMQ, HAProxy, Nginx, Elasticsearch, Logstash and PostgreSQL.

This chart is currently in beta version. Its templates are currently specific for OpenShift. However, work is on-going to support bare Kubernetes deployments.

Maintenance policy

Invenio v3.3 will be supported with bug and security fixes until the release of Invenio v3.5 and minimum until 2021-05-18.

See our maintenance policy.

What's next?

In Invenio v3.4 we are planning to release two major new features:

• Theming support and a new Semantic UI theme.
• A statistics bundle that adds support for collecting COUNTER Research Data Usage Metrics compliant statistics.
• Index migration support for migrating between Elasticsearch clusters (aka zero down-time reindexing and index migration).