On September 28-29 the InvenioRDM project team gathered again (though sadly, again, without fondue) for the Virtual Project meeting. Over 50 partners and team members were present and a great deal of helpful feedback was shared as we worked to develop the roadmap for the next stage of the project. Thanks to everyone for your contributions to the discussion, they are much appreciated!

The InvenioRDM project team would not be so robust without the thoughtful contribution of each of its members. Thank you to all of you who were able to commit the requested 1.5 person months to the project for the next year, in some cases more. The work of each partner site results in unique and valuable contributions to the team. We’d like to use this post to highlight, in a summarized format, the contributions that each project partner highlighted for us at the meeting. Here’s the amazing compendium of work that you have contributed over the past year:

• BNL: Integrating the keycloak OAuth plugin module a part of the InvenioRDM framework.

• Caltech: DataCite 4.3 support and JSON REST API for DOI Registration. Registering very complete metadata records with DataCite.

• Data Futures GmbH: IIIF Presentation API record serializer, IIIF powered search view thumbnails, embedded Mirador 3.0 IIIF previewer.

• EkoKonnect: InvenioRDM part of Open Science cloud platform.

• Geo: Predefined queries for GEO Priority Areas, map preview/reference.

• HZDR: Documentation of deployment steps, testing.

• INFN: Community survey, requirements gathering, testing functionalities, participating in Metadata Working Group, commitment to REST API testing, writing of guides on how to integrate InvenioRDM APIs with programming languages used by research communities.

• JRC: Invenio-records-resources feature to support one of January Release functionalities.

• Northwestern: Various core contributions, including implementing OpenID Connect / OAuth at NU (includes SSO and MFA) and sharing local migration process and local implementation rollout.

• NYU Contributed usability study findings and presented to Telecon; proof of concept geo previewer / viewer for vector GIS data; working toward extending the concept of index sample metadata records via the command line tool.

• NII Testing based on local scenarios; knowledge sharing for InvenioRDM operation on Kubernetes (Monitoring, Logging and Backup strategies, etc.).

• TIND Simple k8s demo setup (helm-invenio).

• Tubitak Provided Turkish translations to InvenioRDM; will maintain and participate in release testing and Interest Group participation.

• TU Graz: Ongoing work: 18n of the python modules; i18n of the react modules; OAI-PMH, core sprint participation.

• TU Wien: During development sprints, helped implement Access Control (v1.0), Share-by-Link (v2.0), and Keycloak SSO (v4.0). Would like to contribute to Communities code in future.

• Universitat Hamburg: Contribution highlights: S3 plugin, initial SAML integration. Committed to: Help with testing, documentation, dissemination.

• Universitat Munster: Development of helm chart, to deploy InvenioRDM inside a Kubernetes infrastructure; implementation/documentation of Kubernetes based architecture (https://github.com/ulbmuenster/inveniordm-2-production); tests of REST API by automated (OAI-PMH based) import of datasets from the WWU document repository miami (https://miami.uni-muenster.de) into InvenioRDM.

• Universitat Tubingen: Contributions planned if production status reached: dissemination, acquiring funding for development needs in interdisciplinary research projects, participation in NFDI’s (nationwide infrastructural developments in domain-related RDM).

• Universitat Freiburg: Regular testing of new releases and features; automatic deployment & recovery strategies; basic UI customization.

• ZHB Luzern: Testing: usability of setup resources (e.g. invenio-cli) & customization and styling; evaluation of metadata standard.

Thanks again, everyone, for your contributions! Hope to see you in person at the next project meeting.

We're very happy to announce that InvenioRDM v6.0 LTS has been released!!!

Try It

Want to try InvenioRDM? Just head over to our demo site: https://inveniordm.web.cern.ch

If you want to install it, follow the installation instructions on https://inveniordm.docs.cern.ch/install/

Production ready

InvenioRDM v6.0 is the first release to be suitable for production services, and therefore the first to receive the Long-Term Support release label. This marks the achievement a major milestone for the InvenioRDM.

Features

Following is a high-level overview of features currently supported by InvenioRDM. This is just the beginning as we have a packed road map with exciting features ahead of us.

Records

• Any resource type InvenioRDM allows you to store publications, datasets, software, images, videos or any other resource type you may have thus can serve as a single repository for all your records.

• Any file format/size InvenioRDM accepts any file format in any size given that your underlying infrastructure can support it.

• Versioning support Records and files are all versioned with optimized storage for large files.

• DOI registration via DataCite InvenioRDM can register DOIs with DataCite for all records, and allows you to write plugins for other identifier schemes.

• DataCite-based metadata InvenioRDMs internal metadata is based on the DataCite Metadata Schema which is a simple yet powerful format for describing nearly any research output (paper, data, software, ...).

• Strong support for persistent identifiers Authors, affiliations, licenses, related papers/datasets etc can all be identified via persistent identifiers such as ORCID and RORs.

• Extended Date Time Format (EDTF) support Publication dates and other dates support the EDTF format for recording imprecise dates and date ranges such as 1939/1945.

• Previewers InvenioRDM comes with previewers for common files formats such as PDFs, images, CSV, Markdown, XML and JSON.

• Citation formatting. InvenioRDM can generate citations strings for your records using the Citation Style Language with support for more than 800+ journal citation styles.

• Record preview. Before you publish your record, you can see a preview of how it looks like.

• Metadata-only records Both records with or without associated files are supported.

• Identifier detection and validation. InvenioRDM comes with support for automatic detection and validation for a large number of persistent identifier schemes (i.e. less typing and clicking for end-users).

Search

• Faceted search. InvenioRDM supports fully customizable faceted search.

• Advanced query syntax. InvenioRDM has support for advanced querying via simple term search, phrase search, range search, regular expressions and custom ranking/sorting/

• Auto-complete as you type. InvenioRDM exposed advanced APIs for search-as-you-type scenarios.

Auth, permissions and security

• Login via institutional account. InvenioRDM makes it easy to integrate your institutional authentication provider such as e.g. Keycloak, OAuth or alternative use e.g. ORCID for login.

• Restricted records. InvenioRDM supports restricting access to files only or to the entire record.

• Share by link. Restricted records can be shared with peer-reviewers or your colleagues via secret links.

• Embargo support Restricted records can be embargoed so that they are automatically made publicly on a specific date so that you can comply with e.g. funders' Open Access mandates.

• Logged in devices. InvenioRDM allows users to see a list of currently logged in devices on their account.

Customizations

• Styling and theming InvenioRDM can be styled and themed to fit into your institutional visual identity.

• Custom vocabularies All vocabularies such as types for resources, dates, roles, relations, affiliations etc can be customized to your local instance.

• Subjects InvenioRDM can load external subjects vocabularies used for classifications such as Medial Subject Headings (MeSH) and many others.

• Permission system InvenioRDM supports advanced customizations to the permission system for e.g. IP-based access control.

APIs and interoperability

• REST API InvenioRDM exposes a strong versioned REST API for all operations on the repository, that allows you to build your own integrations on top of InvenioRDM.

• Export formats InvenioRDM supports exporting records metadata in multiple formats such as JSON, Citation Style Language JSON, DataCite JSON/XML, Dublin Core.

Infrastructure

• Large file support InvenioRDM supports uploading and handling TB-sized files and can manage from MBs to PBs of data as long as your underlying storage systems supports it.

• Multi-storage systems InvenioRDM allows you to integrate backend multiple storage systems in the same instance such as S3, XRootD and more.

• Deploy anywhere InvenioRDM is a Python application and you can deploy into your institutional infrastructure wheather it is on bare metal, VMs, containers, Kubernetes or OpenShift.

Partners

The development of InvenioRDM is the result of an Open Source project kicked of 2 years ago with a diverse set of partners from all over academia and research. Most of the development and testing work has been conducted during the pandemic making it extra challenging for the people involved but also in a fully online environment.

Just the start

This release is just the start. Our next major milestone is to bring Zenodo.org on top of InvenioRDM, which means that most larger features have been shipped along the way, and that the system have been fully tested against large-scale heavy production loads.

Summer 2021 is an exciting time in the development of InvenioRDM, as the team works towards the Long-Term Support (LTS) version. Here are just a few updates we have to share about recent work:

Contributions from the community

InvenioRDM partners are not only local implementers, but frequently contribute their coding expertise to the project. Read this recent news item about the efforts of TU Wien developer Maximilian Moser related to the share-by-link feature and the authentication modules.

Usability testing

One way that the entire community can contribute towards InvenioRDM development is to test the most recent version of the software either through your local implementation or at the CERN sandbox site. Report any bugs you find using this form. Timely and accurate bug reporting gives the entire project a boost!

Getting the word out

InvenioRDM will be featured in two sessions at Open Repositories 2021, June 7-10, “Poster Minute Madness” and the “Repository Rodeo”, both taking place on June 9. If you or someone you know would like an introduction to the software, encourage them to attend!

Are you presenting on InvenioRDM at an upcoming meeting? Please let us know using this form.

InvenioRDM v2.0

We're happy to announce InvenioRDM v2.0!

Try it

Want to try the new features in v2.0 - just head over to our demo site: https://inveniordm.web.cern.ch/

If you want to install it, follow the installation instructions on https://inveniordm.docs.cern.ch/install/

Release notes

See the full release notes and upgrade instructions.

What's new?

Versioning support

InvenioRDM now supports versioning for records. By default, InvenioRDM does not allow you to change the files after you have published a record. With the new versioning support, you can the files can now be changed however in new versions of the record.

Creating new versions

Now, a user can create a new version of the record, add or remove files and publish it by clicking the "New version" button.

One way to make new versions is from the record's landing page directly by clicking on the "New Version" button. Only allowed users can do so:

Another way is while editing a previously published record:

Search

Search results will by default only display the latest published version of a record. However, toggling the flip switch in the versions facet, allows you to view and search all versions.

Landing page

The landing page for previous versions of a record clearly displays that a newer version exists, with a link to it.

Also, in the right-hand column you have a link to view all published versions:

Permissions

We manage permissions at two levels when it comes to versioning.

Each specific version can have its own record/files visibility (e.g. one version can be restricted another not). Embargo dates also apply to specific versions.

Ownership of records, secret links (see below) and sharing of records is applied to all versions. So for instance, if you create a secret link to edit a record, that link will allow the holder to edit all versions of that record.

Get an access link (REST API only)

During this month, we have also added the capability in the REST API to create a shareable link (similar to the Share or Get a link feature on platforms like Google Docs, Dropbox, ownCloud and the like). This link is interchangeably referred as an "access link", a "secret link" or a "shareable link".

You are able to create links with different permission, for instance to allow

• Anyone with the link to view the metadata of a restricted record.
• Anyone to access the restricted files of a record with public metadata.
• Anyone with an account to edit a record.

Note: One of the current limitations (that will be resolved further down the road) is that each secret link can only grant one permission, and they're not hierarchical (e.g. the permission to read files does not entail the permission to read the metadata).

In the future, we plan to also allow sharing only a preview of an unsubmitted record. This is particular useful for instance for peer-review scenarios, where a user can prepare a dataset and provide a journal with a link to the unpublished dataset. The peer-reviewer can then anonymously access the dataset, and the researcher can later update the dataset based on feedback from the peer-reviewer.

We plan to have a first minimal user interface for allowing users to use this new feature in the April release (v3.0). Below you see some of the mockups:

Usage

A secret link can be created by sending a POST request to the record's access/links endpoint, e.g.:

curl -H "authorization: bearer <PERSONAL_TOKEN>" -H "content-type: application/json" -d '{"permission": "read"}' https://<HOSTNAME>/api/records/<RECID>/access/links

If everything went well, the (JSON) response should contain a token. To use the token, it has to be specified in the query string of a URL, e.g.:

https://<HOSTNAME>/records/<RECID>?token=<TOKEN>

This will grant the user the permission with which the secret link was created (even if they don't have an account).

Note: Optionally, an expiration date (UTC timestamp) can be specified when creating a new secret link:

{
  "permission": "read",
  "expires_at": "2021-12-31 23:55:00"
}

Upgrading

The last major new feature is that you are now able to upgrade an InvenioRDM v1.0 instance to v2.0.

Do remember to make a backup of the database and files before performing the upgrade!

Please checkout the full upgrade instructions.

REST API changes

The JSON serializations for records in the REST API have only changed slightly this month, mainly for the new versioning support:

• Added new top-level key versions.
• Added new top-level key parent.
• Moved the top-level key conceptid to parent.id
• Moved the access.owned_by to parent.access.owned_by

Minor changes

• Database connection pooling: We have improved the disconnect handling of the database connection pool, so that connections are automatically opened/closed every hour.
• Documentation: We have slightly reorganised the install/customize part of the documentation.
• Documentation: We have slightly reorganised the REST API part of the documentation.

Known issues

Versioning

• Deposit form: When you create a new version, the files of the previous version are not transferred to the new version so you have to reupload everything.

The following are all carry-overs from the previous release:

Landing page

These are all carry-overs from the previous release:

• Preview: The image previewer is not using IIIF and only previews images up to 500kb.

• Preview: If you e.g. upload a .JPG extension file, you can set the default preview for the file, but the list of files will not have a preview button.

• Preview: The caption of the preview box does not change when selecting another file for preview.

Deposit form

• Validation error messages: The form validation error messages are not displaying correctly. Labels are system names, and sometimes [Object (Object)] is displayed for errors.

• Missing files: If files are missing to be uploaded, the publish button is disabled, but there is no feedback to the end-user that files are missing.

• Performance: We have seen some slight performance issues when entering information into the form. We're looking into this, to make sure it is not noticeable.

• Affiliations vocabulary: We have not initialized the affiliation vocabulary, and thus there is no auto-completion for affiliations (creators and contributors).

• Publish button: It is not enabled when reloading a page with a valid record - you first have to save the record for the publish button to be enabled.

• Protection: The files restriction in the protection widget only become visible after uploading a file. It should be hidden only for metadata-only records.

• Languages: The language vocabulary search results are not ranked properly for the query. For instance searching for "english" will show less relevant results first.

• Creators: The list of creators in the deposit form does not display the role nor identifier if entered.

List uploads

• The title of a search result item is not clickable even though it looks clickable.

Backend

• Embargo records are not automatically published due to the missing background job.

• The REST API JSON serialization has timestamps that are not in UTC, nor do they have timezone information.

Installation

• Some versions of NPM 7 are known to cause issues -- NPM 6.14.5 is recommended.

Feedback

For this particular release we're especially interested in hearing about your challenges and success about upgrading from v1.0 to v2.0.

Please post a message on the #rdm-general Discord channel.

For other issues, please first check if you've hit a known issue:

• Bugs: First check for known issues above, then report them on the Invenio-App-RDM repository on GitHub.
• Feedback: Suggestions for improvements, results of user testing, etc. - please reach out to Lars directly (Discord chat or mail).

What's up next?

During the April iteration (deadline April 30th) we're moving most of the CERN teams on top of InvenioRDM. This means, we'll be around 12 senior and junior developers working on InvenioRDM.

This in itself, poses its own challenges in onboarding new team members and reaching high productivity fast.

We'll be running three teams, each with its own focus:

• Team Communites: The goal for this team is to integrate the creation of communities in InvenioRDM v3.0 (you will not yet be able to associate records with communities)
• Team PIDs: The goal of this team is to ensure that InvenioRDM v3.0 can register DOIs for uploads.
• Team QA: The goal for this team is to catch-up on left-overs from previous iterations, perform quality assurance and release management.

Credits

The development work in this release was done by:

• CERN (Lars & Zach)
• Northwestern University (Guillaume)
• TU Wien (Max)

We're happy to announce InvenioRDM v1.0!

VERY IMPORTANT, read the full release notes before you install and run the system! The v1.0 label does not mean the system is suitable for production use!

FAQ (do not skip!)

Can I use v1.0 in production?

No. Do not even try, or think it might work out!

When can I use InvenioRDM in production?

The first release of InvenioRDM that you can use in a production system will be the LTS (Long-Term Support) release due in end-July.

Why the v1.0 label?

We follow semantic versioning, and in the coming months we'll be releasing v2.0, v3.0, etc. We use the v1.0 label because we're now in a state where the project partners can start running demo systems to demonstrate the progress of the project to their stakeholders. Furthermore, the project partners will be able to start user testing more widely.

Can I upgrade from v1.0 to v2.0?

Read the upgrade contract below.

Upgrade contract (do not skip!)

The following "contract" is meant to align expectations on how and what you'll be able to upgrade from v1.0 to v2.0.

You MUST expect breaking changes to anything on subsequent releases until the LTS release! REST APIs, programmatic APIs, features, Jinja/React templates, data model, vocabularies, etc.

We will ONLY guarantee that you will be able to upgrade a database created with v1.0 to v2.0. With that, we mean that through a manual, labour intensive and offline process you'll be able to upgrade your database. Basically this boils down to, that we will document the steps you need to apply in order to move your data from v1.0 to v2.0 code. In no way do we promise it will be easy! You MAY need to apply manual changes to records. We of course plan to make this an easy and smooth process for the LTS release, but for now it's not.

What's new?

Now that you've hopefully read all the above and are very scared of installing InvenioRDM in production (you should be!), then let's share a bit of the excitement we have for the release this month.

We have reached a really major milestone for InvenioRDM. We now have a minimal working repository with a minimal acceptable user experience with a very minimal feature set. From now on, we are in a releaseable state! That means that in the coming months we'll be improving the UX, the feature set, the stability and production readiness, and thus we're in a very good position for accomplishing our production-ready LTS release in end-July.

Below you'll find the specific areas we've been working on this month.

Access control

The major new feature this month is the support for minimal access control.

The deposit form now has a new protection widget that allows you to deposit records in the following categories:

• Public records with public files
• Public records with restricted files (with or without embargo)
• Restricted records with restricted files (with or without embargo)
• Public metadata-only records
• Restricted metadata-only records (with or without embargo).

Below are some examples of how this now looks in the deposit form:

Restricted records and files

Restricted records and files are only visible for the owner (the user who deposited the material). This means that e.g. a restricted record will not show up in the search results of other users.

User without access (no files and no edit button visible):

User with access (files and edit button visible):

Note, that the message "Sorry, the files are restricted!" is a temporary message, until we have a solution for the access badge (see next section).

Limitation: Access status

We have temporarily removed the access status badge (see before/after screenshot below) until we have found a suitable solution for the badge. The current badge was not easily transferable to the new access control, and is mixing indications about if you can access a record, and/or if files are available.

Before:

After:

Limitation: Embargo feature

You can apply an embargo period to restricted files or records. Once an embargo period is over the record/files are supposed to be made automatically publicly accessible. In v1.0 however, we were not able to include the background job responsible for doing this automatically.

We'll ship this task in the March v2.0 release.

Limitation: Manage permissions

Currently, only the owner is able to see a restricted record/files. In future releases, we'll of course add the support for managing permissions, so that restricted records and files can be shared with colleagues, groups and entire institutions.

Access backend

Underneath the hood, we have prepared the entire access control system to be able to share access with other users, communities, IP networks, etc. Thus, the backend part for allowing management of permissions is almost done, while we are primarily missing the frontend that allows an end-user to interact with the permission system.

Minimal UX

Disabled unfinished features

We have disabled all unfinished features from the user interface. We have for instance disabled the communities module, the version flip-switch in search results, a button to the advanced search and so on.

The features will be added back, once they have reached sufficient quality to be running in a demo system.

Deposit form fields

We have also removed all deposit form fields from the user interface that were not ready. Each of these fields will be added back progressively as they are finished. Most notably we e.g. had to remove the subjects field for this release.

Field help text

We have added help text to as many fields as possible. Please provide feedback, if they are understandable.

Styling issues

We have fixed a large number of small styling and consistentcy issues. Still some remain, and you're feedback is very much appreciated on anything you find in this respect.

List uploads

The list uploads page now has a nice greeting for first time users:

Also, the overall styling of the page was improved:

Search guide

We have added a guide (taken from Zenodo) on how to perform advanced queries. See it on https://inveniordm.web.cern.ch/help/search.

Release v1.0

We have gone through a lot of quality checking to ensure that we'll be able to upgrade data created with the v1.0 release to the coming v2.0 release. This work has focused on ensuring, for example, proper validation and integrity of the data that makes it into the database.

We have also ensured that all database tables have proper upgrade recipes for our automatic schema upgrade system (Alembic).

Known issues

Landing page

• Preview: The image previewer is not using IIIF and only previews images up to 500kb.

• Preview: If you e.g. upload a .JPG extension file, you can set the default preview for the file, but the list of files will not have a preview button.

• Preview: The caption of the preview box does not change when selecting another file for preview.

Deposit form

• Validation error messages: The form validation error messages are not displaying correctly. Labels are system names, and sometimes [Object (Object)] is displayed for errors.

• Missing files: If files are missing to be uploaded, the publish button is disabled, but there no feedback to the end-user that files are missing.

• Performance: We have seen some slight performance issues when entering information into the form. We're looking into this, to make sure it is not noticeable.

• Affiliations vocabulary: We have not initialized the affiliation vocabulary, and thus there is no auto-completion for affiliations (creators and contributors).

• Publish button: It is not enabled when reloading a page with a valid record - you first have to save the record for the publish button to be enabled.

• Protection: The files restriction in the protection widget only become visible after uploading a file. It should be hidden only for metadata-only records.

• Languages: The language vocabulary search results are not ranked properly for the query. For instance searching for "english" will show less relevant results first.

• Creators: The list of creators in the deposit form does not display the role nor identifier if entered.

Search results

• HTML entities (e.g.  ) in the description fields are not removed from the descriptions displayed in search results, causing them to be printed verbatim.

List uploads

• Facet: The search facet distinguishing between published and drafts on the list uploads page is using system labels instead of human readable labels.

• The title of a search result item is not clickable even though it looks clickable.

Backend

• Embargo records are not automatically published due to the missing background job.

• The REST API JSON serialization has timestamps that are not in UTC, nor do they have timezone information.

Installation

• Some versions of NPM 7 are known to cause issues -- NPM 6.14.5 is recommended.

Install

If you previously installed InvenioRDM, make sure you have the latest Docker image of your choice according to the Python version:

docker pull inveniosoftware/centos7-python:3.6
docker pull inveniosoftware/centos8-python:3.7
docker pull inveniosoftware/centos8-python:3.8

To install:

pip install --upgrade invenio-cli
invenio-cli check-requirements
invenio-cli init rdm
cd my-site
invenio-cli containers start --lock --build --setup

This month's version of invenio-cli is v0.22.0.

To stop the instance without destroying the records that were created, you can just run:

cd my-site
invenio-cli containers stop

To destroy the Python virtualenv, and remove the docker containers, you can run:

cd my-site
invenio-cli destroy

Feedback

First, check if you've hit a known issue.

• Bugs: First check for known issues above, then report them on the Invenio-App-RDM repository on GitHub.
• Feedback: Suggestions for improvements, results of user testing, etc. - please reach out to Lars directly (chat or mail).

Credits

The development work in this release was done by:

• CERN (Lars & Zach)
• Northwestern University (Guillaume)
• TU Graz (Mojib)
• TU Wien (Max)

A special thanks goes to Max and Mojib as well as their institutions TU Graz and TU Wien for joining full-time during the February iteration. This made it possible to reach this month's milestone for the regular team from CERN & Northwestern University!

Take care and stay safe!

Congratulations! Your team has just joined the InvenioRDM open source project and is eager to contribute as much as possible. With a mixture of dedicated developers, resource curators, subject matter experts, and various leadership stakeholders, your team simply needs to know about the available resources and how to get started. Read on!

Developers

Whether you would like to install and customize InvenioRDM, contribute on the main project sprints, or take on development of a module locally to contribute back to the project, there are a few important points of entry to participation:

• Roadmap: Familiarize yourself with the overall project timeline and goals.

• Development Roadmap: Get to know the specific milestones

• Sprintboard: See the planning board for the current sprint

• Documentation: Read the official project documentation

• Discord: Ask questions in the project Discord (chat/video platform) and introduce yourself in either the general or newcomers-help channels.

• Discourse: Want to provide feedback, but in a less time-sensitive manner than a chat? Either join or start a thread in the Discourse discussion forum

• Development Setup: Install a development version

• Development Workflows: Learn about the development workflows used

Community Members

All stakeholders at your institution are welcome to contribute to InvenioRDM, whether their contribution is code-based or not. To create a tool to best serve the community, outlining stakeholder requirements will be vital:

• Demo site: Access the latest sandbox version of InvenioRDM here. Create an account to test record creation

• Discourse: Provide feedback on testing here. Use the Feedback category or start a new thread wherever you like

• Metadata Interest Group: This is the first community-focused group organized to help support the InvenioRDM development effort. This group meets about once/month and discusses issues related to the InvenioRDM Metadata Model. To join, either make a post on the Discourse thread or in this dedicated Discord channel

• Conference Calendar: Are you presenting on InvenioRDM at an upcoming conference? Make a note here so other interested members of the community can attend the session, or perhaps join your poster or presentation effort

• Welcome, Help and General Questions: Reach out to Community Manager Sara Gonzales via Discord for general questions or to join the Metadata Interest Group

• Schedule a Demo: Is your site, or a site you know, considering implementing InvenioRDM? Reach out to Community Manager Sara Gonzales via Discord to schedule a one-hour demo and information session about the repository

Everyone

• Project list-serv: Lars will have shared this email address with you upon joining the team. This is the frontline communication tool for the InvenioRDM community. Make this email address a safe sender to receive the latest updates on teleconferences (a.k.a. telecons) and community-based information such as invitations to the Metadata Interest Group meetings. Polls, forms, and additional community tools will be sent to this list-serv

• Telecons (see the list-serv for connection information): All partners of the InvenioRDM community are invited to bi-weekly telecons throughout the year. One telecon per month will focus on development updates, and the other will focus on community-based topics

• InvenioRDM Telecon Schedule

• Blog: You are reading it! The blog will also alternate between development-based and community-based topics. Would you like to contribute a blog post? Sign up here or contact Sara to get on the schedule

• Code of Conduct

• Governance

Welcome to the team, and happy collaborating!

2020 didn't stop us from shipping new releases. 2021 won't either. We’re excited to announce the InvenioRDM Alpha 15 (January release)! Thank you to our team members for their efforts on this release.

What’s new?

This month was primarily dedicated to improving the user experience of the deposit form.

We have made significant improvements to the overall user experience of the deposit form. That said, we still have many smaller UX improvements to make. This work will continue throughout February.

Creators/Contributors

The creators/contributors fields have gotten a complete makeover that significantly reduces the clutter. We have optimized the new creators field for adding basic information (name/affiliation) as well as linking the affiliation field to vocabulary records (e.g. ROR). We postponed the more advanced use case of allowing the user to edit all fields such as identifier scheme and identifiers for affiliations, as this created an overly complex form for the majority of use cases.

The creator field we imagine can be extended in the future with support for importing authors via predefined files (e.g. BibTex, etc.).

One particular feature to highlight is that the name identifiers field now does automatic detection of the identifier scheme. You can now paste an ORCID (e.g. an ORCID URL), and the backend will automatically detect it as an ORCID and normalise the stored value.

Coming Soon

We have decided to hide certain fields that we know need further work. You'll see these as "Coming soon". Also, we have temporarily removed the language field from titles and descriptions as we had last minute issues with these fields.

Minor UX Improvements

• Sticky side bar: The side bar with the save/publish buttons, is now sticky so it will follow you when you scroll down the form.

• Resource type: The drop-down was reduced from two drop-downs to a single drop-down to reduce the number of needed clicks to select a resource type.

• Delete/discard button: There's now a delete button that you can use to delete your draft or discard edits to a published record.

• Button consistency: We have applied consistent styling, wording and and naming to most buttons.

• Errors for inline fields are now consistently displayed.

• Brand color: We have now made it easier to customize the styling of the field groups via a common brand color. Previously, if your header was purple, the field groups would remain blue.

• File warning: The warning displayed under the files section was updated to match Semantic UI styling.

• Field help text: We've added an example of field help text to the Dates field, which will expand to all fields needing it.

• Array fields styling: dates, related identifiers and titles are now properly using the entire form grid from side to side.

• Consistent naming of related work.

• Title: The deposit form now has a title.

Other Changes

• Vocabularies: The vocabularies REST API received some significant backend changes, which means that we can now properly show translated text in auto-completion fields such as languages, license, affiliations, and subjects.

• Extended Date Time Format indexing: We now have support for searching EDTF date ranges properly (contributed by TU Wien).

• Geospatial indexing: We are now able to properly index almost any GeoJSON data type (points, boxes, polygons, etc.) so that they can be searched properly at a later point. They are indexed via their centroid which is computed using the Shapely library (contributed by Cottage Labs).

• Core API methods needed for reindexing of vocabularies was added as well (contributed by EU JRC).

• Prettyprint: If you explore the REST API in a browser, you can now add ?prettyprint=1 to the URL, and the JSON will be nicely formatted for you.

Known Issues

• Discarding a change to an edited record doesn't work.
• Licenses cannot be saved.
• Languages cannot be saved and only Danish and English are currently available.
• License data is missing.
• Error messages at the top of the form are not displayed correctly.

Install

If you previously installed InvenioRDM, make sure you have the latest Docker image of your choice according to the Python version:

docker pull inveniosoftware/centos7-python:3.6
docker pull inveniosoftware/centos8-python:3.7
docker pull inveniosoftware/centos8-python:3.8

To install:

pip install --upgrade invenio-cli
invenio-cli init rdm
cd my-site
invenio-cli containerize --pre
invenio-cli demo --containers

This month's version of invenio-cli is 0.21.0 .

To stop the instance without destroying the records that were created, you can just run:

cd my-site
invenio-cli stop

To destroy the Python virtualenv, and remove the docker containers, you can now just run:

cd my-site
invenio-cli destroy

Feedback

As always, we welcome your feedback. When you provide feedback on Discourse your message should be pre-populated with the classic template (bugs, what worked well, what didn’t work well, wishes for documentation).

Here is the template to give feedback if it’s not automated:

## Bugs

## What worked well

## What didn't work well

## Wishes for documentation

Take care and stay safe!

We’re excited to announce the InvenioRDM Alpha 14 (December release)! Thank you to our team members for their efforts on this release.

What’s new?

Vocabularies support

The focus of the December release was to add support for linking bibliographic records with authority records such as subjects, grants, licenses, languages and more.

This includes:

• Auto-complete on deposit fields
• Facet searches based on vocabulary values
• Support for internationalization on deposit, on facets, etc.
• REST APIs for vocabularies
• Linking of records in the backend.

Deposit form autocompletion

The most visible way to see the vocabularies in action is via the deposit form fields.

The below components were added in the deposit form and they support vocabulary autocompletion:

Languages

You can see the autocomplete search help:

And an example of i18n support, when the instance is in Turkish:

Licenses (preview)

We have added support for both standard and custom licenses. Note that we were not able to fully complete this work, and there are known bugs in the license widget. Most notably you'll see:

• Search field does not allow spaces
• Link not working
• UX issues such as over-long page
• Descriptions for licenses not available.

Subjects (preview)

The subjects field autocompletes from a subjects vocabulary that can host multiple different vocabularies at the same time.

The current version is not yet connected with the API, thus it has some demonstration values instead. Only the values you see in the image below can be selected. Don't save a record with subjects yet either.

Invenio v3.4

Thanks to the hard work of the December Invenio sprint, Invenio v3.4 was released (see more here). The release of Invenio v3.4 was an important step for InvenioRDM. We now depend on stable releases of Semantic UI.

Partial save and error reporting

This release includes support for partial validation on draft creation. It also displays partial saves in the uploads page, for example with "No title" if the title was not included in the draft.

For example, saving an empty draft will tell you precisely what is missing at the top, along with pointers per field (see title at the bottom of the image).

The former error space will update to good (green) once you save a draft that is publication-ready. The publish button will then be enabled.

Install

$ invenio-cli init rdm
$ cd my-site
$ invenio-cli packages lock --pre
$ invenio-cli install --pre
$ invenio-cli services setup
$ pipenv run invenio vocabularies import languages licenses
$ invenio-cli run

Feedback

As always, we welcome your feedback. When you provide feedback on Discourse your message should be pre-populated with the classic template (bugs, what worked well, what didn’t work well, wishes for documentation).

Here is the template to give feedback if it’s not automated:

## Bugs

(note, that there's known issues in this release for license and subjects fields)

## What worked well

## What didn't work well

## Wishes for documentation

Take care and stay safe!

We are proud to announce the release of Invenio v3.4.0.

Head over to our Getting started to see it in action.

Read the full release notes.

What's new in Invenio v3.4?

New UI Framework: Semantic UI

The largest new feature in Invenio v3.4 is the release of Semantic UI as the new UI framework. This works has been under way since we released version 3.0 with a deprecation notice on AngularJS 2.5 years ago.

Semantic UI is now the new default UI framework for Invenio, replacing the existing Bootstrap v3 framework. Most importantly, we have added support in Invenio v3.4 for supporting multiple UI frameworks concurrently. This means that Bootstrap 3 is still supported in v3.4 together with Semantic UI, and that we in the future can support new frameworks.

UI Frameworks evolve very rapidly and we expect that Invenio eventually will have to go through multiple UI frameworks over it’s life-time. Allowing support for multiple frameworks, isolate future changes, and even allow third-parties to develop custom themes for Invenio without impacting the business logic Invenio.

Technically the support for multiple UI frameworks works by having specialised Jinja template loading and themable webassets build systems.

Read more about the new system in User interface styling.

Reorganised documentation

We have reorganised the Invenio documentation, to make it easier for new developers to get started with Invenio.

In particular we now have a documentation overview page on https://inveniosoftware.org/documentation/ that gathers both Framework, RDM and ILS documentation under one umbrella.

We have also reorganised the main Invenio Framework documentation on https://invenio.readthedocs.io/en/latest/

Records improvements

Invenio v3.4 marks the start of some larger improvements we want to make to the overall data flow for records in Invenio. With Invenio v3.4 we are releasing new features for Invenio-Records. These improvements are not being used by other Invenio modules yet, but allows us to ensure the backward compatibility of the changes.

Overall, the changes made to Invenio-Records focuses on having records stored in separate database tables/Elasticsearch indexes and making a simplified programmatic API that is less error prone to use.

The following only describes the new featuers at a high-level.

Dumping/Loading support

Records now supports dumping and loading of records. This will eventually replace the using signals for customising the JSON being indexed, as well as standardising the record being loaded from Elasticsearch, database or other sources.

Type support

Currently, a record only supports native JSON data types. In particular, date/time data types are not supported. We have now added the possiblity to support that certain fields inside your JSON have custom data types (such as datetime).

Extensions

We have added support record extensions as a mechanism for replacing signals. Overall the signals were causing issues, because you often needed signals per type of record you were dealing with. With the extensions, you can now more easily hook into

System fields

A typical issue you would deal with in Invenio is to create a record and a persistent identifier. System fields allows you to easily compose a record type with new features, so that e.g. Record.create(...) will also automatically create a persistent identifer.

Backward compatibility

All changes are backward compatible thus existing source code and stored records will continue to work with the latest release.

Testing improvements

GitHub Actions instead of Travis CI

We have moved almost all our 100+ repositories from using Travis CI as the continues integration system to GitHub Actions. This was due to Travis CI imposing a migration from .org to .com and lowering the total number of resources availble for open source projects. Overall, this meant very long waiting times for pull-request builds as well as PyPI releases. We have therefore migration to GitHub Actions. We would like to thank Travis CI for the service they have provided through out the years to the open source community.

Reprodicible tests

We have developed a new tool Docker-Services-CLI, that helps bring up/down required services such as databases, Elasticsearch, etc for testing. This makes it easier to investigate specific test failures locally, as well as making it easier to migrate between continuous integration systems.

Centralised test dependencies and Pytest-Invenio

With the release of Pytest-Invenio v1.4 we have centralised test dependencies to a single module, and can more easily control breaking changes from third-party packages. Previously new version of isort and pytest have been causing wide-spread test failures due to minor changes.

Minor changes in v3.4

• Invenio-Accounts:

  • Password length validation during login was removed.
  • "Log in" instead of "Sign in" is now being used consistently throughout the application. Previously both version could be found.

• Invenio-App:

  • The /ping endpoint (used by e.g. load balancers for aliveness checks) now also supports HEAD and OPTIONS HTTP verbs as recommended by e.g. HAProxy documentation.

• Invenio-Celery

  • Only Celery to 4.4.x to 5.0.x releases are supported. Note, Celery have declared Celery 4.4.x as a Long Term Support release. Note, that Celery changed command line arguments from celery worker -A <app> to celery -A <app> worker between version 4 and 5.

• Invenio-DB

  • Now support SQLAlchemy <1.4 which include e.g. PostgreSQL 12 support.
  • Hides the database password from CLI output when running db init or db create.

• Invenio-Indexer

  • Elasticsearch delete requests now uses optimistic concurrency control by providing the the version and version_type in delete requests. The previous behavior can restored by calling RecordIndexer().delete(record, version=None, version_type=None) instead. This fixes an issue where deleting and recreating a document right after would fail.

• Invenio-JSONSchemas

  • Fixes an issue related to nested allOf being ignored.

• Invenio-OAuthClient

  • Added support for CERN OpenID contrib.

• Invenio-Records-REST

  • The header Cache-Control: 'no-cache' is now sent on successful HTTP 200 responses to ensure that browsers will not cache responses client side and ETag will be evaluated.

• Invenio-REST

  • Fixes a bug with CSRF checking when the endpoint did not exist.

Deprecations in v3.4

Bootstrap 3

Bootstrap v3 has reached end-of-life and will no longer receive further updates from their maintainers. Migrating to Bootstrap v4 or v5 is as difficult as upgrading to Semantic UI and there are no easy migration paths.

We will continue to support the Bootstrap v3 integration throughout the maintenance period of Invenio v3.4. Invenio v3.5 may move all Bootstrap 3 templates to a separate Invenio package, that you could install, however v3.5 will not add new Bootstrap 3 templates.

You should plan allocating time for a migration from Bootstrap 3 to Semantic UI during 2021

Features removed in v3.4

Elasticsearch v2 and v5 support

We have removed support for Elasticsearch v2 and v5 as announced in v3.3. Elasticsearch v2 and v5 have reached end-of-life and is no longer maintained with security fixes, thus you should strongly consider upgrading in case you have not already done so.

AMD/RequireJS

The old AMD/RequireJS assets build system, that was deprecated and replaced with Webpack in Invenio v3.1 has now been completely removed from the code base.

Maintenance policy

Invenio v3.4 will be supported with bug and security fixes until the release of Invenio v3.6 and minimum until 2021-12-17.

See our Maintenance Policy.