Invenio Blog

Follow news and updates on Invenio world

InvenioRDM June Release

Lisa O'Keefe, Pablo Panero Jul 1, 2020 Invenio

We are happy to announce InvenioRDM Alpha 8 (June release). Thank you to our team members for their efforts on this release.

What's new?

The June release is a major backend development release. We developed three new modules that lay the foundation for the new core data flow of Invenio(RDM). Our work goes far to provide a better developer experience. You will be able to work on and customize Invenio(RDM) in an easier and cleaner way.

The primary focus for the release was:

  • Continue the migration from Bootstrap to SemanticUI
  • Bug fixes and improvements on the frontend (mainly on deposit)
  • Develop new core modules for Invenio(RDM) backend

UI Customization

You are able to customize your templates again. The JSX customizations are not functional yet though, which means the search page results are not customizable.

Visual appearance

We have migrated the file previewer, the OAuth login, and the settings pages to SemanticUI.

You might still notice some issues related to visual appearance (e.g. the name previewed file is not in a panel as before). We will be working on making all this perfect in the next sprint on SemanticUI (end of August). In the meantime, it is, at least, better.

Creators, Contributors and Affiliations Identifiers

The Deposit page now accepts ORCID iDs for people (creators and contributors) and ROR identifiers for organizations (creators, contributors and affiliations). An icon with a link to the respective entity on the record landing page reflects this connection.

Contributors are now optional on the deposit page as well.

Python versions

There are now Invenio images for Python 3.7 and Python 3.8! This means that you can use Python 3.6, 3.7 and 3.8 for local development and the full containerized mode.

Backend and REST APIs

We have put a significant amount of work in refactoring some of the core API flow of Invenio. This addresses many of the issues that Invenio was facing in this domain. As a result, three modules were created:

What do you need to do?

Follow the documentation site: https://inveniordm.docs.cern.ch/ and install the latest invenio-cli v0.14.2+, create a new instance and see how it looks like!

Install (TL;DR)

If you previously installed InvenioRDM, make sure you have the latest Docker image of your choice according to the Python version:

docker pull inveniosoftware/centos7-python:3.6
docker pull inveniosoftware/centos8-python:3.7
docker pull inveniosoftware/centos8-python:3.8

To install:

pip install invenio-cli --upgrade
invenio-cli init rdm
cd my-site
invenio-cli containerize --pre
invenio-cli demo --containers

To destroy the Python virtualenv, and remove the docker containers run:

cd my-site
pipenv --rm
docker-compose -f docker-compose.full.yml down

Feedback

As always, we welcome your feedback. We are experimenting with Discourse's built in post template. This means when you provide feedback on Discourse your message should be pre-populated with the classic template:

## Bugs

## What worked well

## What didn't work well

## Wishes for documentation

Thank you for your patience and support. We are working hard all the time to improve things. Take care and stay safe!

Read More

InvenioRDM May Release

Lars Holm Nielsen, Lisa O'Keefe, Zacharias Zacharodimos Jun 5, 2020 Invenio

We are happy to announce InvenioRDM Alpha 7 (May release). Thank you to our team members for their efforts on this significant release.

What's new?

The May release is a major integration release, with a few rough edges that still need polishing. We have integrated a lot of code that was developed in separate branches into the main InvenioRDM code base. We have done this now, to ensure we have enough time to iron out integration issues.

The primary focus for the release was:

  • Migrate from Bootstrap to SemanticUI
  • Deposit form frontend (JavaScript)
  • Integrate new communities prototype.
  • Upgrade to Invenio v3.3

Known issues (please read!)

As a rule of thumb, many features will be broken and not work. Thank you for your continued patience and feedback as we work to make improvements.

A lot of code has changed in this release, and many features have not yet undergone quality control.

UI Customizations not working

The existing method for customizing the UI will change slightly, We hope to provide documentation for this in the next release.

Visual appearance

You'll notice issues related to the visual appearance, that we have not yet had the time to refine. This is with regards to alignment, colors, behaviours, font-sizes, and user experience.

Data model and deposit form

The data model and deposit form is far from complete. We have not yet focused on implementing the right data model with the right fields. For instance the access right displayed in the deposit form will change, as will many of the other fields displayed.

Communities

Communities is in a very initial state and many things will change.

Previewer, OAuth login, Settings pages

The file previewer and OAuth login still need to be migrated to SemanticUI, and thus the modules are likely not to work properly in this release.

What do you need to do?

You can install the latest invenio-cli v0.14.x, create a new instance from the beginning and see how it looks like!

Install (TL;DR)

If you previously installed InvenioRDM, make sure you have the latest Docker image:

docker pull inveniosoftware/centos7-python:3.6

To install:

pip install invenio-cli --upgrade
invenio-cli init rdm
cd my-site
invenio-cli containerize --pre
invenio-cli demo --containers

To destroy the Python virtualenv, and remove the docker containers run:

cd my-site
pipenv --rm
docker-compose -f docker-compose.full.yml down

Communities

To see the communities click "Communities" menu, then "New community". You'll be prompted to log in. Simply create a new account, and log in with it.

Deposit form

To see the deposit click "Uploads" then "New upload"

Want to get involved?

Follow the documentation site: https://inveniordm.docs.cern.ch/ for an up-to-date step-by-step install and usage of InvenioRDM.

Give us feedback on Discourse: https://invenio-talk.web.cern.ch/t/inveniordm-alpha-7-may-release/109 for this release.

Thank you for your patience. We are working hard all the time to improve things. Take care and stay safe!

Read More

Invenio v3.3 released

Pablo Panero May 20, 2020 Invenio

We are proud to announce the release of Invenio v3.3.0. With this release, Invenio v3.1.x reaches end of life, and will no longer be maintained.

Python compatibility

Invenio v3.3 supports Python versions 3.6 and 3.7. Python 2 support in Invenio ended on January 1st, 2020 with the official end of life for Python 2 on the same date.

Getting started

See our quick start guide.

Upgrading

See our upgrade guide.

Release notes

Please see the full release notes for details about minor changes, deprecations etc.

Highlights for Invenio v3.3

Python 3.7

Python 3.7 is now supported by Invenio!

Improved support for Single Page Applications

Invenio v3.3 improves the support for Single Page Applications (SPA) by adding REST APIs for account management operations such as login, logout, user registration, password change, email confirmation and more. The integrated OAuth client also adds a new REST API so that you can login via your OAuth providers such as GitHub, ORCID and Globus.

CSRF Protection

Invenio-REST adds a CSRF middleware, called CSRFProtectMiddlewate to protect API views against CSRF attacks. The CSRF checks can be skipped in REST API calls when using a personal OAuth API token.

Dependency management

We have revamped management of third-party dependencies. New releases of some of the many third-party packages Invenio depends on could often cause dependency conflicts. Invenio v3.3 introduces the concept of coordinator packages, who are responsible for properly specifying third-party dependencies.

The release notes contain a table with all the current coordinator modules.

Helm chart (BETA)

Helm-Invenio provides a helm chart to deploy an Invenio application along with all its required services: Redis, RabbitMQ, HAProxy, Nginx, Elasticsearch, Logstash and PostgreSQL.

This chart is currently in beta version. Its templates are currently specific for OpenShift. However, work is on-going to support bare Kubernetes deployments.

Maintenance policy

Invenio v3.3 will be supported with bug and security fixes until the release of Invenio v3.5 and minimum until 2021-05-18.

See our maintenance policy.

What's next?

In Invenio v3.4 we are planning to release two major new features:

  • Theming support and a new Semantic UI theme.
  • A statistics bundle that adds support for collecting COUNTER Research Data Usage Metrics compliant statistics.
  • Index migration support for migrating between Elasticsearch clusters (aka zero down-time reindexing and index migration).
Read More

InvenioRDM April Release

Lars Holm Nielsen, Lisa O'Keefe, Guillaume Viger May 5, 2020 Invenio

We are pleased to announce the InvenioRDM April release. The Deposit page has been our focus and that meant designing the frontend and the backend based on lessons learned from the beta invenio-deposit python module and from the invenio-records-js and invenio-files-js previous angularjs modules. We will be implementing most of these designs in May.

Thank you all for testing the last release and for your many contributions.

Some Highlights include:

Want to get involved?

Although in this release, we haven’t made significant visual changes, you are always welcomed to try new releases - invenio-cli v0.12.X, this time around- by following the documentation site: https://inveniordm.docs.cern.ch/ . Give us feedback on Discourse: https://invenio-talk.web.cern.ch/t/inveniordm-alpha-6-april-release/104 for this release.

As mentioned, apart from the /deposits/new proof of concept page you won’t see much other differences. Take it easy this time. We will be asking a lot of feedback from you next month!

Thank you for your interest, take care and stay safe!

Read More

Invenio v3.2 released

Lars Holm Nielsen Dec 20, 2019 Invenio

We are proud to announce the release of Invenio v3.2.0. With this release, Invenio v3.0.x reaches end of life, and will no longer be maintained.

Python compatibility

Invenio v3.2 supports Python 2.7 (until 2019-12-31), Python 3.5 and Python 3.6.

Getting started

See our quick start guide.

Upgrading

See our upgrade guide.

Release notes

Please see the full release notes for details about minor changes, deprecations etc.

Highlights for Invenio v3.2

Files bundle

We have released four new modules as part of the new Files bundle:

To understand more about how to use the new Files bundle see our integration guide on https://invenio.readthedocs.io/en/latest/tutorials/handling-files.html.

Elasticsearch v7 support

Invenio now supports Elasticsearch v7.

Marshmallow 3 compatibility

Invenio now support both Marshmallow v2 or v3. The support is done via a compatibility layer, that allows Invenio to work with either schemas from v2 or v3. This should allow users to upgrade to Invenio v3.2 without being forced to upgrade their Marshmallow schemas immediately to v3.

We advice all users to start planning an upgrade of their Marshmallow schemas from v2 to v3 as the upgrade is non-trivial and needs proper testing due to significant differences between Marshmallow v2 and v3.

Invenio will continue support for both Marshmallow v2 and v3 for a transition period to allow users to upgrade at their own pace. After the transition period Marshmallow v2 support will be deprecated and removed from Invenio.

For information about how to upgrade see https://invenio.readthedocs.io/en/latest/tutorials/upgrade-marshmallow.html.

Search index prefixing

Elasticsearch does not support the concept of virtual hosts and thus with previous versions of Invenio it was not possible to share an Elasticsearch cluster between multiple Invenio instances.

We have now added support for index/alias/template prefixing, so that all names can be prefixed with a string and thus allows multiple Invenio instances to share an Elasticsearch cluster. Note, that this is only name prefixing, thus technically two Invenio instances will can read each other indexes and thus must trust each other.

Read more about the new feature on https://invenio-search.readthedocs.io/en/latest/configuration.html#index-prefixing.

Maintenance policy

Invenio v3.2 will be supported with bug and security fixes until the release of Invenio v3.4 and minimum until 2020-12-20.

See our maintenance policy.

What's next?

In Invenio v3.3 we are planning to release the Statistics bundle including:

  • invenio-stats
    • Invenio module for statistical data processing and querying with support for collecting COUNTER Research Data Usage Metrics compliant statistics.
  • counter-robots
    • Library for COUNTER-compliant detection of machines and robots.

In addition to the Statistics bundle, we will also release final versions of the following two modules:

  • invenio-index-migrator
    • Elasticsearch index migrator for Invenio (aka zero down-time reindexing and index migration).
Read More

Introducing RDM, ILS and Framework

Lars Holm Nielsen Oct 15, 2019 Invenio

We're happy to announce a major overhaul of inveniosoftware.org. Some of the highlights of the new website include:

  • Rebranding of Invenio into three products: InvenioRDM, InvenioILS and Invenio Framework
  • A new forum
  • People in the community
  • Logo downloads

Framework, RDM and ILS

The primary reason for the website overhaul is a rebranding of Invenio into three different products:

Both, InvenioRDM and InvenioILS are applications built on top of the Invenio Framework. On each product page, you'll find a lot more information about the product as well as their current roadmaps.

Talk - Discourse forum

We are also launching a new forum for both users, administrators and developers, which replaces our current troubleshooting repository on GitHub and compliments the current chatrooms.

People and institutions in the community

We have also added a new people section to better showcase the persons who are making Invenio into a real community. Don't hesitate to email us if you'd like to be displayed on the list.

New logos

Last but not least, we've made some very minor modifications to the Invenio logo, as well as made a dedicated download page where you can get SVG versions of all logos.

Read More

Towards Invenio v3.2 and Elasticsearch 7 support

Lars Holm Nielsen Aug 7, 2019 Invenio

Two CERN sprint teams with a total of 14 developers have just each finished a two-week sprint:

  • Sprint team 1: Focused on the Invenio v3.2 release which will include the new Files bundle.
  • Sprint team 2: Focused on adding Elasticsearch 7 support as well prepare the Invenio v3.3 release which will include the new index migration utilities.

Highlights

A total of 23 new module releases where made during the sprint. The highlights from the two sprints include:

  • Elasticsearch 7 support.
  • Elasticsearch index prefixing and suffixing support for shared clusters (this is needed for the upcoming index migrator utilities).
  • Marshmallow v2 and v3 compatibility. Invenio is now able to accept both Marshmallow v2 and v3 schemas. In your Invenio instance you will need to pin the Marshmallow version that matches your schemas, and follow the upgrade guide provided by Marshmallow to upgrade your schemas.
  • Sentry support is now using the Sentry-Python library instead of Raven library (you can still switch back to Raven by setting SENTRY_SDK = False in your configuration).
  • Rate limiting now differentiates between guests and authenticated users, and allows for external modules to provide per user rate limits.
  • Improved HTML sanitisation support in several modules.
  • Improved support for client-side infinite scroll in the REST API.
  • Housekeeping: we have fixed a significant number of build failures as well as deprecation warnings from other libraries.

All of above highlights will be released together with Invenio v3.2. Most of the individual modules have already been released, however you are still on your own if you decide to go head with them prior to the Invenio v3.2 release (if you encounter problems, we are of course very interested in hearing about it, so that we can solve them before the v3.2 release).

Future plans

Invenio v3.2 and Files Bundle

The primary focus is still to release Invenio v3.2. The pending issues are limited to final testing and documentation.

Elasticsearch v2 and v5 deprecation

Invenio v3.3 will add index migration utilities that will allow Invenio users to upgrade their Elasticsearch clusters to supported Elasticsearch versions. In Invenio v3.4 we plan to then remove support for Elasticsearch v2 and v5.

Python v2 support ends January 1st 2020.

Python v2.7 will reach end of life on January 1st, 2020. Invenio will only support Python 2.7 until that date. From January 1st, 2020 we will remove Python 2.7 from our test matrixes, and thus new module releases after January 1st 2020 will very likely no longer work on Python 2.7.

We are already seeing a large number of our dependent Python libraries that have removed Python 2 support, and thus we will not be able to continue Python 2 support beyond January 1st, 2020.

Invenio v3.3 - Index migration and usage statistics

Invenio v3.3 is planned for release in late 2019 or early 2020. The primary focus for Invenio v3.3 will be adding support for Elasticsearch index migration as well as releasing the Statistics Bundle (COUNTER-compliant usage statistics). The Statistics Bundle includes the following modules:

  • Invenio-Stats
  • Invenio-Queues
  • COUNTER-Robots

Invenio v3.3 may also see the release of a new module, Invenio-Records-Permissions, which will significantly simplify the defining and managing access control for records.

Releases overview

  • invenio-access: v1.2.0
    • Removed DynamicPermission from Invenio-Access (deprecated since v1.0.0)
  • invenio-app: v1.2.0
    • Fixed issue with instance_path and static_folder being globals evaluated once which caused problems with fixtures in pytest-invenio.
    • Improved the rate limiting to differentiate between guests and authenticated users.
    • Added possibility for external modules to provide per user rate limits via the Flask g global request object.
    • Fixed deprecation warnings from Werkzeug.
  • invenio-assets: v1.1.3
    • Changed module to hide webpack warnings (primarily needed for the cookiecutter-invenio-instance to reduce output clutter).
  • invenio-base: v1.1.0
    • Added support for allowing instance_path and static_folder to be callables which are evaluated before being passed to the Flask application class (related to invenip-app fix ).
  • invenio-celery: v1.1.0
    • Fixed missing release on PyPI.
  • invenio-config: v1.0.2
    • Added ALLOWED_HTML_TAGS and ALLOWED_HTML_ATTRS default configuration for bleach HTML sanitisation library (values are used by Invenio-Records-REST, Invenio-Formatter and Invenio-Previewer).
  • invenio-db: v1.0.4
    • Added PostgreSQL v10 into the test matrix to ensure future compatibility.
  • invenio-formatter: v1.0.2
    • Added a new Jinja filter sanitize_html that uses the bleach library to sanitise data and to be used in combination with the safe template filter to prevent Cross-Side Scripting (XSS) vulnerabilities.
  • invenio-indexer: v1.1.0
    • Added Elasticsearch 7 support.
    • Added before_record_index.dynamic_connect() signal utility for connecting index receivers directly to specific indexes.
    • Fixed Elasticsearch index prefixing support.
  • invenio-logging: v1.2.0
    • Changed Sentry integration to use the sentry-python module instead of raven library. Raven library is still supported for backward compatibility.
  • invenio-oaiserver: v1.1.1
  • invenio-oauthclient: v1.1.3
    • Fixed deprecation warnings from Flask-OAuthlib
    • Fixed issue with the ?next parameter not supporting a query string.
  • invenio-records: v1.3.0
    • Removed the CLI (deprecated since v1.1.0)
  • invenio-records-rest: v1.5.0
    • Added Elasticsearch 7 support
    • Added CSV serialiser (for allowing CSV exports)
    • Added Marshmallow v3 support
    • Added “from” and “aggs” query parameters for better supporting client-side infinite scroll use cases.
    • Changed SanitizedHTML marshmallow field to use central configuration from Invenio-Config.
    • Fixed a deprecation warning.
    • Fixed Elasticsearch index prefixing support.
    • Fixed bug with browsers not respecting the content type when caching the REST API responses. (PENDING merge)
  • invenio-rest: v1.1.1
    • Added compatibility layer for marshmallow v2 and v3
  • invenio-search: v1.2.1
    • Added Elasticsearch v7 support
    • Fixed bug with Elasticsearch index prefixing support.
    • Added index suffixing and write aliases.
    • Deprecated Elasticsearch v5 support.
    • Changed default library used for making request to Elasticsearch from requests to urllib3 (default recommended library).
  • invenio-theme: v1.1.4
    • Added an error handler for HTTP 429 (rate limiting error).
  • xrootdpyfs v0.1.6
    • Fixed bug preventing previewing large ZIP files (2GB+).

Files Bundle

The Files Bundle also saw releases of the following modules:

  • invenio-files-rest: v1.0.1
  • invenio-records-files: v1.1.1
  • invenio-previewer: v1.0.1
  • invenio-iiif: v1.0.1
  • pytest-invenio v1.2.0

We don’t recommend upgrading to these versions until Invenio v3.2 have been released. In particular, we have made breaking changes to Invenio-Records-Files from the v1.0.0a11 to v1.0.0, that are likely to impact you if you depended on the unsupported alpha releases.

Stay tuned for the Invenio 3.2 release!

Read More

Invenio security releases - XSS and Host header injection

Lars Holm Nielsen Jul 15, 2019 Invenio

Two vulnerabilities have been identified in supported Invenio modules.

  • Invenio-Records (security advisory): A Cross-Site Scripting (XSS) vulnerability has been identified in Invenio-Records in the administration interface.
  • Invenio-App (security advisory): A Host header injection vulnerability has been identified in Invenio-App.

In addition, two XSS vulnerabilities have been discovered in unsupported Invenio modules:

  • Invenio-Previewer (security advisory): An XSS vulnerability affecting the JSON, Markdown and iPython Notebook previewers.
  • Invenio-Communities (security advisory): An XSS vulnerability affecting the Jinja templates.

The vulnerabilities were found after an XSS vulnerability was reported to Zenodo by Ciro Santilli. As a standard measure and after patching Zenodo, we reviewed the Invenio source code for potential similar issues to those identified in the Zenodo source code. This led to the discovery of three additional XSS vulnerabilities. The host header injection vulnerability was discovered after a standard vulnerability scan of another service running at CERN.

Releases

We have issued two new Invenio releases fixing these issues:

  • Invenio v3.0.2 and v3.1.1

The following individual modules fixing the vulnerabilities have been released:

  • Invenio-Records v1.0.2, v1.1.1 and v1.2.2
  • Invenio-App v1.0.6 and v1.1.1
  • Invenio-Previewer v1.0.0a12 (unsupported)
  • Invenio-Communities v1.0.0a20 (unsupported)

New security policy

We have taken the chance during handling of these vulnerabilities to also clearly define and document Invenio's security policy. Please have a look and let us know what you think.

Previously, we have sometimes privately notified potentially affected services about a security vulnerability. We have however decided to discontinue this practice, and instead, send out an advance notification to everyone about an upcoming security release including only the severity level of the issue. This allows everyone to plan ahead for the upcoming release and ensure they have staff available to handle the release. This partially smoothens the communication process but also ensures that everyone receives the same information in a scalable approach.

GitHub security advisors

As a new thing, we have also evaluated the new GitHub maintainer security advisories to handle the vulnerabilities. These advisories are reviewed by GitHub and should allow a security alert to be sent to affected repositories.

For more information

If you have any questions or comments about this security release:

Read More

Keeping up with Elasticsearch

Alex Ioannidis & Lars Holm Nielsen May 24, 2019 Invenio

In our latest Invenio sprint from May 13-24 we've been working on making Elasticsearch upgrades, migration and reindexing simpler and with near zero downtime. Read on, to learn more about how you keep up with Elasticsearch rapid release and deprecation cycles with Invenio and for a preview of what will come of new features in Invenio v3.2.

End of life

Once you start running a production Invenio instance, you'll quickly notice that your Elasticsearch cluster will become outdated pretty fast, due to Elasticsearch's rapid release and deprecation cycle. Invenio v3 development was started in 2016 against the newly released Elasticsearch v2. When Invenio v3 was finally released in June 2018, Elasticsearch has had two major releases (v5 and v6) and v2 had already reached end of life. As can be seen from Elastic Product End of Life Dates, you can expect a major Elasticsearch version to be supported for a maximum of 2-3 years. Thus, you should plan to regularly upgrade your Elasticsearch cluster in order to not run on unmaintained software.

Index upgrades with downtime

Elasticsearch allows an index created in v5 to be read in v6 which combined with rolling cluster upgrades allows you to upgrade a cluster relatively easy with no downtime. However, you are out of luck as soon as you want to upgrade again from v6 to v7 since v7 cannot read an index created in v5.

The solution is for every major Elasticsearch upgrade to create a new index and reindex content from your old index. This requires you to take offline your Invenio instance to avoid that new content is indexed during the reindexing process. For large-scale production systems with many millions of records, reindexing can take a long time, and thus might require several hours of downtime, which can quickly be a big issue these systems.

Fixing Elasticsearch mappings

A related problem you may hit when running an Invenio instance is that you would like to tweak the Elasticsearch mappings in order to improve search performance and precision. In Elasticsearch you cannot modify the mapping of an existing index, thus similar to above, you need to create a new index and reindex content from the old index.

Use cases

All in all, you'll find yourself in need of tools to manage indexes and performing reindexing in order to:

  • Upgrade an Elasticsearch cluster between major versions.
  • Migrate between two Elasticsearch clusters.
  • Change Elasticsearch mappings of existing indexes.

Near-zero downtime upgrade, migration and reindexing

To solve the above use cases, we've built a new Invenio module named Invenio-Index-Migrator. The module allows you to synchronize two Elasticsearch indexes (in-cluster or cross-cluster) over an extended time period, and when you are ready, you can roll over the new index. Depending on how you have deployed Invenio, the rollover can happen with either zero downtime or near zero downtime (<5 minutes compared to hours)

The new module supports migrating records but also any other index you may have in Elasticsearch such as event logging from usage statistics. The new module works by defining migration recipes and managing them via a CLI

Demo

Disclaimer: APIs etc may change slightly before the final release.

First, you initialize a migration recipe (here a recipe named records) which will create new indexes to hold the new data:

$ invenio index migration init "records"

Next, you kick-off the index migration:

$ invenio index migration run "records"

Index migration happens in two stages:

  • Bulk migration: First a snapshot of the current index is reindexing into the new index. Invenio-Index-Migrator is flexible an supports several methods. By default, we use the Elasticsearch Reindex API which in most cases are orders of magnitudes faster than any other method. Other methods allow you to use Invenio's default record indexing or simply implement your own.
  • Synchronization: Once the first snapshot has been fully indexed, a second job is started to keep the two indexes in sync. Essentially this works by indexing any modifications to the old index in the new one.

At any point during the migration, you can check the status:

$ invenio index migration status "records"

Once you're ready, you can roll over the new index:

$ invenio index migration rollover "records"

In an in-cluster migration, the rollover works by shuffling around index aliases (since Elasticsearch does not support renaming indexes). In a cross-cluster migration you update the Invenio configuration to point to your new Elasticsearch cluster.

In case something goes wrong during the index migration, you can of course also cancel the job:

$ invenio index migration cancel "records"

Bonus

During the sprint a number of other issues were addressed that will also be part of Invenio v3.2:

  • Elasticsearch v7 support was added to Invenio-Search, Invenio-Indexer and Invenio-Records-REST. Only Invenio-OAIServer needs to be upgraded to support Elasticsearch 7.
  • Search index prefixing allowing multiple Invenio instances to use the same Elasticsearch cluster. Elasticsearch by default does not support the concept of virtual hosts. With search indexing prefixing, it will be possible to use a single Elasticsearch cluster for multiple Invenio instances, given that the Invenio instances either trust each other, or you have a protection-layer like Elastic X-Pack or ReadonlyREST in front of Elasticsearch.

Next steps

The Invenio-Index-Migrator is only the first building blocks in order to make Elasticsearch upgrades, migration and reindexing simple. We hope to add assistant-like features that will make it even easier to keep with the latest Elasticsearch releases.

The Invenio-Index-Migrator will be finally released with Invenio v3.2 which is planned for July-September.

Read More