Invenio Blog

Follow news and updates on Invenio world

Invenio User Group Workshop 2019, June 10th

Lars Holm Nielsen May 3, 2019 Invenio

We would like to announce the 5th Invenio User Group Workshop, IUGW2019, to be held as part of Open Repositories 2019 (OR2019) on Monday, June 10th from 09:00-17:00 at the University of Hamburg (main building).

Registration (mandatory)

  1. First register to Open Repositories 2019 conference.

  2. Next, register to the Invenio User Group Workshop 2019.

Call for proposals

Invenio User Group Workshop (IUGW) is a biennial workshop where the Invenio repository community meet among current and future users and developers from around the world. The workshop consists of a series of tour de table service presentations and talks from attendees related to the Invenio digital repository framework.

Submit abstract

Deadline: May 22nd at 23:59 CEST

Acceptance: Notification on May 27th.

The Invenio User Group Workshop 2019 will address a wide range of topics related to the overall theme for Open Repositories 2019:

All the user needs

1. Understanding user needs and user experience

  • User research and engagement
  • User experience design for repository services
  • Better user experience through data and workflow integration
  • Improving repository user interfaces

2. Discovery, use and impact

  • Increasing content visibility in search engines and discovery systems
  • Open access discovery, research data discovery
  • Tools for researchers, interfaces for machines
  • The role of aggregation services
  • Measuring use and impact

3. Repositories – evolution or revolution?

  • Beyond the repository: using repository platforms for purposes not originally intended
  • Convergence with other types of systems (e.g. current research information systems, digital asset management systems, journal publishing platforms, library service platforms)
  • Interoperability vs integration: will repositories survive as stand-alone systems?
  • The developing role of repositories in the scholarly communications and research information systems ecosystem (e.g. the Next Generation Repositories vision)
  • New models for scholarly sharing (e.g. blockchain)
  • Data mining, artificial intelligence and machine learning

4. Supporting open scholarship and cultural heritage

  • Providing access to different types of materials (e.g. research data, scholarly articles, pre prints and overlay journals, open access monographs, theses and dissertations, educational resources, archival and cultural heritage materials, audiovisual materials, software, interactive publications and emerging formats)
  • Workflows and support services for the repository users
  • Training, communication and outreach
  • Long-term access and preservation
  • Repositories as digital humanities and open science platforms
  • Working with large and complex data sets

5. Open and sustainable

  • Service and business models that meet user needs
  • Local systems vs repository as a service
  • The expanding role of service providers in the repository landscape, pros and cons?
  • Sustainability of the open source community model
  • Securing long-term funding for open infrastructures
  • Open business models and open governance for open infrastructures

6. Policies, licensing and the law

  • Impact of GDPR (General Data Protection Regulation) and copyright laws
  • Publisher policies, embargoes and rights retention
  • Licenses, use and re-use of content
  • ‘Closed’ material in ‘open’ repositories
  • Compliance and impact of funder policies (e.g. Plan S) on repositories

7. How can metadata and standards help our users?

  • Development and standardisation of repository metadata
  • Data models and entities
  • Linked open data and repositories
  • Persistent identifiers (e.g. DOI, Handle, URN, ORCID, ISNI)
  • Open citations
  • International Image Interoperability Framework (IIIF)

8. Repositories and global knowledge

  • Integration with other open knowledge resources (e.g. Wikimedia and Wikidata)
  • National vs global solutions
  • Repository systems and language barriers
  • Repositories in the global south
  • User needs in developing countries

Invenio RDM: a turn-key open source research data management platform

Lars Holm Nielsen Apr 29, 2019 Invenio

CERN has partnered with 10 multidisciplinary institutions and companies to build a turn-key open source research data management platform called Invenio RDM, and grow a diverse community to sustain the platform.

The Invenio RDM project is funded by the CERN Knowledge Transfer Fund, as well as all the participating partners, including:

The project has an ambitious one year schedule in which it will deliver:

  • Invenio RDM - A research data management platform based on Zenodo and Invenio v3 Framework.
  • A community of public and private institutions to sustain Invenio RDM.
  • Minimum two existing repositories migrated to Invenio RDM, with Zenodo being one of them.

The key to successfully achieving the ambitious schedule is that Invenio RDM will be based on Zenodo that have already been successfully validated over the past 5 years.

Our vision in the next five-years, is to make Invenio RDM a world-leading extensible research data management platform used by research institutions all around the world and with businesses providing services, support and customizations on top of Invenio RDM.

What is a RDM platform?

An research data management (RDM) platform allows researchers to share and preserve scientific results. Researchers can share anything from publications, posters, presentations to datasets and software. Once a researcher have shared a result, they get a DOI (Digital Object Identifier), that allow them to properly cite their result.

There are three primary purposes of RDM is to

  • Disseminate and archive
  • Enable reproducibility
  • Enable reuse

Most importantly, research funding agencies all over the world have realised the huge potential economic and social benefits of RDM to society and are now demanding solutions.

Zenodo

CERN in partnership with OpenAIRE has built one such RDM service with European Union funding called Zenodo. Zenodo has been highly successful and has in its 5 years of existence become a world-leading general purpose research data repositories.

In fact, other institutions have already taken Zenodo source code (which is open source), and started building their own local RDM solutions on it. The goal of this project is to join our efforts to build a common RDM-platform from which both CERN, other institutions and private businesses can profit.

Multidisciplinary partners

The real strength of the project is that it brings together a suite of partners from multidisciplinary domains, that each bring unique knowledge and know-how from their specific domains that will be critical to the success of project:

  • Institutional partners:
    • University of Hamburg
    • University of Münster
    • Caltech Library
  • Health and medical science partners:
    • Northwestern University
  • Physics partners:
    • Helmholz Zentrum Dreseden Rossendorf
    • Brookhaven National Laboratory
    • TUBITAK - The Scientific and Technological Research Council of Turkey
  • Digital humanities partners:
    • Data Futures
  • Business partners:
    • TIND Technologies
  • Community partners:
    • OpenAIRE

Building Invenio RDM

Invenio RDM will include most of the features that Zenodo alreday include today such as e.g. DOI miniting capabilities, versioning support and COUNTER compliant usage statistics to name some few.

The work in transforming Zenodo into a general purpose RDM-platform will involve three key areas:

Core repository

The repository platform will at its core include an extensible metadata model based DataCite metadata schema with support for handling millions of records and peta bytes of data. The repository will further be aligned with the Next Generation Repositories (NGR) standard.

Packaging and distribution

Key for easy adoption of Invenio RDM is to ensure that it is a real turn-key solution requiring minimal experience in installing, operating and administering the platform, or in short getting users started in no time. Thus, a significant part of the efforts will go into simplifing the installation, improving the packaging and distribution, as well as providing excellent end-user documentation.

Customization and extendability

A key requirement for Invenio RDM is that it can easily be extended and customized just enough to adapt to each particular institution. This includes for instance defining authentication mechanisms (SAML/LDAP/OAuth), integrating with mutliple storage backend system, and most important of all to make these customizations easy.

Contact

For more information about the Invenio RDM project, please contact:

Lars Holm Nielsen

Invenio Product Manager

CERN IT Department

Email: info@inveniosoftware.org

Invenio Training Bootcamp 2019

Nicola Tarocco Mar 25, 2019 Invenio

The Invenio v3 Bootcamp was held from the 19th to the 21st of March, 2019 at CERN. The event reached the maximum number of 30 participants with attendees from all over the world (Czech Republic, Danmark, Germany, Finland, France, Japan, Norway, Spain, Switzerland, UK and USA). The bootcamp was targeting developers wanting to learn more about Invenio and to understand how to create or customize an Invenio repository.

Format

The three days of the bootcamp was organised with a succession of talks and practical hands-on sessions, presented by 6 Invenio experts. Participants were able to develop each of the functionality on their laptop with a constant assistant in case of need.

The objective was to discover the Invenio framework and acquire knowledge on how to build a new repository by progressively introducing new concepts.

The main topics included:

  • Getting started with Invenio
  • Tour of functionalities and infrastructure
  • Customizations of data models and look & feel
  • Deposit of new records and references between data models
  • Access control
  • Security
  • Deployment and application architecture

Try it yourself

We've made all the material publicly available for all those who couldn't participate in the event, so you can try out the same exercises at home:

If you run into troubles, we're happy to answer questions you may have.

Takeaways

Participants had the opportunity to experience how to work with Invenio thanks to the hands-on sessions and to understand how to apply each of the introduced functionalities to use cases of their organization.

The bootcamp attracted quite a lot of interest for Invenio and participants were very involved and curious: we were very glad to answer to all of the interesting questions that were asked.

We would like to thank everyone for their participation and help to have contributed​ to the success of the event.

Invenio User Group Workshop 2019 @ Open Repositories 2019, June 10th

Our next event is the Invenio User Group Workshop (IUGW) at the Open Repositories 2019 conference in Hamburg. The workshop will be held on June 10th with presentations from Invenio users from around the world.

The call for proposals will be announced early April, so stay tuned.

See you in Hamburg!

Invenio v3.1.0 released

Lars Holm Nielsen Mar 11, 2019 Invenio

We are proud to announce the release of Invenio v3.1.0.

Head over to our Getting started to see it in action.

Python compatibility

Invenio v3.1 supports Python 2.7 (until 2019-12-31), Python 3.5 and Python 3.6. We expect to add support for Python 3.7 in the near-term future once Celery v4.3 has been released.

What's new in Invenio v3.1?

Webpack build system

Invenio v3.1 comes with a new assets build system based on Webpack for building and packaging your JavaScript applications, stylesheets and much more. The system replaces the previous AMD/RequireJS based system which was deprecated in v3.0.

The old build system is still available to allow users to upgrade to Invenio v3.1 without first migrating to Webpack. The old build system will be removed in Invenio v3.3

For more information about the new build system, please see:

Simplified scaffolding

We have simplified the scaffolding of new Invenio instance by merging the data model template into the main Cookiecutter-Invenio-Instance.

The previous approach of two separate packages -- one for the application and one for the data model -- caused friction and confusion for new users and we, therefore, decided to merge both.

This also fits with our long-term goal, where we want to provide standard data models (such as DataCite, Dublin Core, MARC21) so that users don't have write their own data model.

Docker base image

We have released a new Docker image that can serve as base image for your Invenio instances. The image is based on CentOS 7 and comes with Python 3.6, Node.JS, NPM and some standard libraries often needed by Invenio.

See inveniosoftware/centos7-python on DockerHub.

Pipenv

In order to manage Python dependencies more reliable and securely for your Invenio instance we have moved to use Pipenv which also handles the virtualenv creation. This has all been integrated with the Getting started guide.

Documentation

New sections where added to the documentation specifically on:

  • Bundles
  • Requirements
  • Build a repository
  • Managing access
  • Secure your instance
  • Infrastructure architecture

See https://invenio.readthedocs.io.

Request tracing

Invenio v3.1 has added new features for improved request tracing to allow for better troubleshooting and auditing of problems. The feature allows logging a request id, session id and user id across multiple services such as Nginx and Invenio error logs. This enables e.g. system administrators to identify exactly which Nginx access log line caused a specific error logged by Invenio.

If combined with e.g. centralised log aggregation, this can be used for e.g. viewing requests by a user in real-time, request performance statistics and many other metrics. Please note that in order to be compliant with EU General Data Protection Regulation (GDPR), you must ensure that these logs are automatically deleted after 3 months (the same is the case if you only log an IP address).

  • Cookiecutter-Invenio-Instance:
    • Nginx configuration has been updated to automatically generate a random request id and add is as X-Request-ID header.
    • Nginx log format has been updated to log timing information, request id, session id and user id if provided by the application server in the X-Session-ID and X-User-ID HTTP headers. Nginx will remove both headers prior to sending the response to the client.
  • Invenio-App:
    • Extracts the X-Request-ID header (max 200 chars) if set in the HTTP request and makes it available on the Flask g object as g.request_id.
  • Invenio-Logging:
    • The request id is made available to all log handlers.
    • The Sentry log handler will add the request ID as a tag if available.
  • Invenio-Accounts
    • The X-Session-ID and X-User-ID HTTP headers will be added to the HTTP repsponse if the configuration variable ACCOUNTS_USERINFO_HEADERS is set to True. This makes the session and user id available to upstream servers like Nginx.

Minor changes in v3.1

Token expiration

The token expiration was changed from 5 days to 30 minutes for the password reset token and email confirmation token. Using the tokens will as a side-effect login in the user, which means that if the link is leaked (e.g. forwarded by the users themselves), then another person can use the link to access the account. Flask-Security v3.1.0 addresses this issue, but has not yet been released.

Globus.org OAuth Login

Invenio v3.1 now comes with support for login with your Globus.org account. The feature was contributed by University of Chicago.

See Invenio-OAuthClient for details.

Health-check view

A /ping view that can be enabled via the APP_HEALTH_BLUEPRINT_ENABLED configuration variable has been added to support load balancers like HAProxy to check if the application server is responsive.

Backwards incompatible changes

  • Pytest-Invenio: The celery_config fixture has been renamed to celery_config_ext due to naming conflict with fixture provided by Celery.

Deprecations in v3.1

Following list of features have been deprecated and will be removed in either Invenio v3.2 or Invenio v3.3:

Elasticsearch v2 support

Elasticsearch v2 support will be removed in Invenio v3.2. Elasticsearch v2 has reached end of life and no longer receives any bug or security fixes.

Both the support in Invenio-Search for creating indexes for v2 as well as any v2 Elasticsearch mappings in other Invenio modules will be removed.

AMD/RequireJS

Invenio's assets build system based on AMD/RequireJS will be removed in Invenio v3.3.

This involves e.g. the two CLI commands:

$ invenio npm
$ invenio assets build

Several Python modules in Invenio-Assets will be removed, including (but not limited to):

  • invenio_assets.npm
  • invenio_assets.filters
  • invenio_assets.glob
  • invenio_assets.proxies

Also, bundle definitions in other Invenio modules will be removed. These are usually located in bundles.py files, e.g.:

  • invenio_theme.bundles

Also, some static files will be removed from bundles, e.g.:

  • invenio_theme/static/js/*
  • invenio_theme/static/scss/*

DynamicPermission class

The invenio_access.DynamicPermission class will be removed in Invenio v3.2. It has been superseded by the invenio_access.Permission class. The Permission class by default deny an action in case no user/role is assigned. The DynamicPermission instead allowed an action if no user/role was assigned.

Records CLI

The following CLI commands will be removed in Invenio v3.2:

$ invenio records create
$ invenio records delete
$ invenio records patch

Please use the REST API instead to create, patch and delete records.

AngularJS (reminder from v3.0)

In Invenio v3.0 we deprecated the AngularJS 1.4 application Invenio-Search-JS as AngularJS by that time was already outdated. We have selected React and SemanticUI as the replacement framework for AngularJS.

The new Webpack build system released in Invenio v3.1 is part of the strategy to move from AngularJS to React (note however that you can use Webpack with your favorite framework, including AngularJS).

We have started the rewrite of Invenio-Search-JS and have already released the first version of React-SearckKit which eventually will replace Invenio-Search-JS.

Features removed in v3.1

These following already deprecated features have been removed in Invenio v3.1.

  • invenio_records.tasks was removed from the Invenio-Records module.

Maintenance policy

Invenio v3.1 will be supported with bug and security fixes until the release of Invenio v3.3 and minimum until 2020-03-31.

What's next?

We originally planned to release the Files and Statistics bundle in Invenio v3.1. We however decided it was more urgent to release the new Webpack build system in order to avoid too much code being written against the old build system.

In Invenio v3.2 we are planning to release the Files bundle including:

  • invenio-files-rest
    • Object storage REST API for Invenio with many supported backend storage protocols and file integrity checking.
  • invenio-iiif
    • International Image Interoperability Framework (IIIF) server for making thumbnails and zooming images.
  • invenio-previewer
    • Previewer for Markdown, JSON/XML, CSV, PDF, JPEG, PNG, TIFF, GIF and ZIP files.
  • invenio-records-files
    • Integration layer between object storage and records.

Invenio v3 Training Bootcamp

Lars Holm Nielsen Dec 21, 2018 Invenio

We are pleased to announce the first Invenio v3 Bootcamp taking place at CERN, 19-21 March 2019. The Bootcamp is intended as an introduction to developing digital repositories with the Invenio v3 framework.

Website

https://indico.cern.ch/e/invenio-bootcamp/

Planned topics

  • Creating your first Invenio instance.
  • Customizing the look and feel.
  • Working with data models.
  • Managing access to records.
  • Managing files.
  • Creating a new module from scratch.
  • Depositing records.
  • Securing your Invenio instance.
  • Deploying Invenio.

The topics will be covered through practical tutorials and presentations. Note, that the final programme is subject to change, based on inputs we receive from you in the registration form.

Who can register

  • All registrations are subject to our approval (i.e. don't book flights until we confirm your registration).
  • Intended audience:
    • Software developers with some prior Python experience.
  • Limited capacity:
    • We have limited capacity so we prioritise to have as many different institutions/companies represented as well as people with concrete projects for which they plan to use Invenio v3.

How to register

https://indico.cern.ch/event/773969/registrations/46902/

Cost

The Bootcamp itself is free of charge, but you will need to cover your own expenses during the Bootcamp including (but not limited) to lunches, dinners, coffee, accommodation, transport and social events.

Location

CERN, Geneva, Switzerland.

Invenio v3.0.0 Released

Lars Holm Nielsen Jun 7, 2018 Invenio

Welcome to Invenio 3!

We are proud to announce the release of Invenio v3.0.0. Invenio has been completely rewritten from scratch with a radically improved architecture and technical implementation. Invenio 3 is now a framework, like a Swiss Army knife, complete with battle-tested, safe and secure modules providing all the features you need to build and run a trusted digital repository.

Whilst Invenio 3 is officially released to the world today, in reality it has already been relied upon in large-scale production systems for more than 1.5 years on sites such as:

Also other sites are already in process of being built on Invenio 3:

  • INSPIRE HEP - an aggregator for High-Energy Physics.
  • WEKO3 - repository platform for 500+ Japanese universities.

What's new

Invenio functionality is being released in bundles of modules. Invenio v3 contains the following bundles totaling more than 27 individual Invenio modules:

  • Base: the core application framework with e.g. distributed task queue support.
  • Auth: accounts management, role-based access control, OAuth 2.0 client and provider, user profiles management.
  • Metadata: record and persistent identifier management including indexing, querying and OAI-PMH server.

The following bundles are being prepared for release in v3.1:

See our roadmap for further details.

Getting started

In order to get started developing with Invenio v3 follow our getting started guide.

Next, head over https://invenio.readthedocs.io to understand how to develop with Invenio.

In addition, each Invenio module also has extensive documentation:

Base bundle

Auth bundle

Metadata bundle

Compatibilities

Python compatibility

Invenio v3.0 supports Python 2.7, 3.5, 3.6. We highly recommend only using the latest official release in each series.

Python 2.7 end-of-life is scheduled for April 2020. Invenio will only support Python 2.7 until that date. We highly recommend that all new projects are started on the latest available Python 3 version.

Elasticsearch compatibility

Invenio v3.0 supports Elasticsearch 2, 5 and 6.

Elasticsearch v2 has reached end-of-life (February 2018) and Invenio v3.0 is the last release to support Elasticsearch v2.

PostgreSQL compatibility

Invenio v3.0 supports PostgreSQL 9.4, 9.5 and 9.6. We have not yet tested Invenio v3.0 with PostgreSQL 10.

MySQL compatibility

Invenio v3.0 supports MySQL 5.6+.

Deprecations

AMD/RequireJS

Invenio v3.0's current static assets management system is based on e.g. RequireJS will be replaced with Webpack. We expect this work to be ready for Invenio v3.1, and thus we are already deprecating the current support. Specifically this means that Invenio-Assets and Invenio-Theme will change significantly in Invenio v3.1. We would have liked to already have this ready for this v3.0 release, but unfortunately it was time-wise not possible.

AngularJS

Invenio v3.0 comes with one AngularJS 1.4 application (Invenio-Search-JS). AngularJS is by now already outdated, and we are planning a rewrite of the application in another JavaScript framework that is currently in process of being selected. Essentially this means that you should not extend Invenio-Search-JS at this point, since it will change significantly.

Maintenance policy

Invenio v3.0 will be supported with bug and security fixes until the release of Invenio v3.2 and minimum one year.

We aim at one Invenio release with new features every 6 months. We expect upgrades between minor versions (e.g. v3.1 to v3.2) to be fairly straight-forward as in most cases only new features are added.

Sprint: v3.0.0RC2 + updated website

Lars Holm Nielsen May 28, 2018 Invenio Framework

This sprint (May 14-25) was focused on preparing Invenio for the final v3.0.0 release so it is ready for launch at the Open Repositories 2018 Conference, June 4th-7th in Bozeman, Montana.

Overview

Result of the sprint (45 developer days, 107 commits, 8.5k lines touched):

  • inveniosoftware.org now has shiny new blog and roadmap.
  • inveniosoftware.org was moved to Lektor static CMS and is now also hosted on GitHub Pages.
  • Cookiecutter templates for Invenio instances and Invenio data models where finalised and made part of the "Getting started" guide.
  • Invenio v3.0.0 Release Candidate 2 was released.
  • Flask v1.0 support

Releases

  • invenio-access v1.0.1
  • invenio-accounts v1.0.1
  • invenio-base v1.0.1
  • invenio-db v1.0.1
  • invenio-formatter v1.0.1
  • invenio-oauth2server v1.0.1
  • invenio-records-rest v1.1.0
  • invenio-userprofiles v1.0.1
  • pytest-invenio v1.0.2

List of all changes

  • cookiecutter-invenio-datamodel:
    • Fixing, testing, documenting and cleaning.
  • cookiecutter-invenio-instance:
    • Fixing, testing, documenting and cleaning.
    • The development server is now by default running SSL thanks to the recently release Flask v1.0.
  • invenio-access:
    • Fixed annoying warning message.
  • invenio-accounts:
    • Fixed build failure due to Flask-CeleryExt version.
  • invenio-base:
    • Added support for blueprint factory functions in the invenio_base.blueprints/invenio_base.api_blueprints entry point groups. In addition to specifying an import path to an already created blueprint, you can now specify an import path of a blueprint factory function with the signature create_blueprint(app), that will create and return a blueprint. This allows moving dynamic blueprint creation from the extension initialization phase to the blueprint registration phase.
  • invenio-formatter:
    • Fixed issue where badges would be cached wrongly.
  • invenio-jsonschemas:
    • Fixed build error related to Flask v1.0
  • invenio-oaiserver:
    • Fixed build error related to Flask v1.0
  • invenio-oauth2server:
    • Fixed build error related to Flask v1.0
  • invenio-oauth2server:
    • Fixed build error related to Flask v1.0
  • invenio-oauthclient:
    • Fixed build error related to Flask v1.0
  • invenio-records-rest:
    • Added marshmallow fields used for record deserialization.
    • Added support for other Invenio extensions to auto-register REST API record endpoints.
    • Fixed build error related to Flask v1.0
  • invenio-rest:
    • Fixed build error related to Flask v1.0
  • invenio-userprofiles:
    • Fixed build error related to Flask v1.0
  • invenio:
    • Documentation updates.
  • inveniosoftware.org:
    • Rewritten using Lektor Static CMS system.
    • Retargeted website to pass the message that Invenio is now a framework.
    • Added a blog.
    • Added a roadmap
    • Moved all information about legacy releases to separate page.
    • Many fixes to design and style.
    • Reworked most text sections.
    • Moved hosting to GitHub Pages.
  • pytest-invenio:
    • Fixed problem with the celery_config fixture.
    • Fixed issue with elasticsearch import.

Sprint: v3.0.0RC1, Elasticsearch v6 and MIT license

Lars Holm Nielsen Mar 23, 2018 Invenio Framework

This sprint was focused on releasing the Invenio v3.0.0 Release Candidate 1.

As a result of 95 developer days, 485 commits and 88k lines touched (33k additions and 55k deletions):

  • Invenio v3.0.0 Release Candidate 1 was released.
  • License has been changed from GPL to MIT for ~30 repositories (part of v3RC1).
  • Documentation for API authentication in Invenio was added (in OAuth2Server).
  • Elasticsearch v6 support was added.

Releases

  • invenio: v3.0.0rc1
  • invenio-access: v1.0.0
  • invenio-accounts: v1.0.0
  • invenio-admin: v1.0.0
  • invenio-app: v1.0.0
  • invenio-assets: v1.0.0
  • invenio-base: v1.0.0
  • invenio-cache: v1.0.0
  • invenio-celery: v1.0.0
  • invenio-config: v1.0.0
  • invenio-db: v1.0.0
  • invenio-formatter: v1.0.0
  • invenio-i18n: v1.0.0
  • invenio-indexer: v1.0.0
  • invenio-jsonschemas: v1.0.0
  • invenio-logging: v1.0.0
  • invenio-mail: v1.0.0
  • invenio-oaiserver: v1.0.0
  • invenio-oauth2server: v1.0.0
  • invenio-oauthclient: v1.0.0
  • invenio-pidstore: v1.0.0
  • invenio-records: v1.0.0
  • invenio-records-rest: v1.0.0
  • invenio-records-ui: v1.0.0
  • invenio-rest: v1.0.0
  • invenio-search: v1.0v.0
  • invenio-search-ui: v1.0.0
  • invenio-theme: v1.0.0
  • invenio-userprofiles: v1.0.0
  • dcxml: v0.1.1
  • citeproc-py-styles: v0.1.1
  • pytest-invenio: v1.0.0
  • invenio-marc21: v1.0.0a8

Backward incompatible changes

  • invenio-oauthclient:

    • Added creation/modification timestamps to database models. Requires upgrade of database using the included Alembic recipe. The recipe will set the creation/modification timestamp to the current date + time (same value for all rows).
  • invenio-oaiserver:

    • XSL stylesheet that allow nice rendering of the OAI-PMH feed have been removed as it was GPL licensed. Documentation describes how to add it back.

List of all changes:

  • All packages:
    • License change and general repository clean up.
  • cookiecutter-invenio-datamodel:
    • New template for bootstrapping data models in Invenio.
  • cookiecutter-invenio-instance:
    • New template for bootstrapping new instances.
  • invenio-accounts:
    • Removed Invenio-Admin dependency.
  • invenio-base:
    • Documentation for CLI commands.
  • invenio-config:
    • Documentation for how to set complex values in environment variables.
  • invenio-db:
    • Changed dependency psycopg2 to psycopg2-binary due to warning message.
  • invenio-indexer:
    • Elasticsearch v6 support
  • invenio-jsonschemas:
    • Fixed functools dependency for Python 2
  • invenio-marc21:
    • Elasticsearch v6 support
  • invenio-oaiserver:
    • Elasticsearch v6 support
    • XSL stylesheet removal and documentation.
    • Removal of invenio-query-parser dependency
  • invenio-oauth2server:
    • Usage documentation
    • Fixed missing field length validation in form.
  • invenio-oauthclient:
    • Added timestamps to all models
    • Hide part of access token when displayed.
  • invenio-pidstore:
    • Allow modifying "control_number" field for recid minter via config.
  • invenio-records-rest:
    • Usage documentation
    • Elasticsearch v6 support
    • Removed invenio-query-parser dependency.
  • invenio-rest:
    • Always install Flask-CORS dependency. Please update your install from invenio-rest[cors] to just invenio-rest.
    • Add sentry event ID in error response.
    • Change from dicttoxml to xmltodict (to avoid GPL dependency).
  • invenio-search-js:
    • Upgrade jquery to v3.3.1
    • Fixed facet selection with no results
  • invenio-search-ui:
    • Fixed facet selection with no results
  • invenio-search:
    • Elasticsearch v6 support
    • Removed invenio-query-parser dependency.
  • invenio:
    • Removed provisioning scripts and docker configuration (in favour of cookiecutter-invenio-instance).
    • Bump all dependencies to v1.0.0
  • pytest-invenio:
    • Fixed click logging output with latest Flask development version.

Sprint: Elasticsearch v5 support

Lars Holm Nielsen Dec 1, 2017 Invenio Framework

This sprint was focused on:

  • Elasticsearch v5 support
  • Preparing to release the metadata bundle.
  • Removing remaining Invenio-DB warnings.

During the sprint 60 developer days were spent, 52 commits were created and 4.3k lines were touched (3.1k additions and 1.2k deletions).

List of changes:

  • Cookiecutter-Invenio-Module:
    • Minor various fixes for build issues.
  • IDUtils (v1.0.0):
    • global: fix DOI Unicode issues
  • Invenio-Base (v1.0.0b1)
    • Cookiecutter template removal (to be replaced by upcoming cookiecutter-invenio-instance).
    • Release checklist, docs build errors, 100% test coverage.
  • Invenio-DB (v1.0.0b9)
    • Remove annoying warning.
  • Invenio-I18N (v1.0.0b4)
    • Example app rendering on ReadTheDocs
  • Invenio-Indexer (v1.0.0b1):
    • New BulkRecordIndexer class with RecordIndexer-compatible API (to be used by Invenio-Records-REST).
    • Elasticsearch v5 support.
    • Release checklist.
  • Invenio-JSONSchemas (v1.0.0a7)
    • Documentation improvements.
    • Release checklist.
  • Invenio-Mail (v1.0.0b1)
    • Broken docs build fix (related to Celery problem).
  • Invenio-OAIServer (v1.0.0b1)
    • Description support in Identify verb (eprints, friends etc).
    • Elasticsearch v5 support.
    • Release checklist + documentation improvements.
    • License change PR
  • Invenio-OAuth2Server (v1.0.0b3)
    • Unpinned oauthlib.
  • Invenio-PIDStore (v1.0.0b2)
    • Release checklist.
  • Invenio-Records-REST (v1.0.0b5)
    • Index after create, update and delete record creation (this will later impact Invenio-Deposit).
    • Serializers refactored to be more easily composable.
    • Improved tombstone handling for the REST API (e.g. include removal reason).
    • Elasticsearch v5 support.
    • Improved documentation.
    • Dynamic aggregations.
    • Release checklist.
  • Invenio-Records-UI (v1.0.0b2)
    • Release checklist + example app docs fix for RTD.
  • Invenio-Search (v1.0.0b4)
    • Elasticsearch v5 support (via version-specific mappings).
    • Support for creating only specific indexes.
    • CLI for listing all indexes and aliases.
  • Invenio-Search-UI (v1.0.0a9)
    • Release checklist.
  • Invenio-MARC21 (v1.0.0a6)
    • Elasticsearch v5 support.

Elasticsearch 5 support

Search, Indexer, Records-REST, OAIServer and MARC21 have all been upgraded to support Elasticsearch v2 and v5. Elasticsearch v6 is not yet support due to the elasticsearch-dsl package not yet supporting v6 (support already merged in master branch but not yet released).

Other Invenio packages have not yet been upgraded to Elasticsearch v5. E.g. Records-Files, OpenAIRE, Stats, Collections, OpenDefinition, Query-Parser have not yet been tested with Elasticsearch v5.

Choosing version

You will need to know at install-time which version of Elasticsearch you'd like to use. For instance to use Elasticsearch 5 you need to install Invenio-Search like this:

$ pip install invenio-search[elasticsearch5]

Mappings

Due to the differences between Elasticsearch versions, we have opted for version-specific mappings. This means that Invenio modules must provide a mapping per Elasticsearch version they wish to support. E.g. today, the mapping is placed in a directory like e.g.:

mappings/records/record-v1.0.0.json

To support Elasticsearch v2 and v5 you now need two mappings:

mappings/v2/records/record-v1.0.0.json
mappings/v5/records/record-v1.0.0.json

Note that mappings for Elasticsearch v2 may use either the mappings/v2 directory or the mappings/ directory like previously (for backward compatibility).

Adding Elasticsearch v5 support to a module

In case you have site-specific modules and would like to add Elasticsearch v5 support here's a rough guide:

  • Update travis.yml to test both v2 and v5 (example)
  • Update setup.py by moving Invenio-Search dependency to extra_require (example).
  • Move existing mappings and add new mappings for v5 (example)
    • Most common change from v2 to v5 is the change from string type to either text or keyword type.
  • Update docs/requirements.txt by adding elasticsearch5 as an extra requirement to ensure ReadTheDocs builds will be fine.
  • Fix any API specific calls (see below).

Canonical way of checking for ES version.

from elasticsearch import VERSION as ES_VERSION

if ES_VERSION[0] == 2:
    # ...

Completion Suggesters

The Completion Suggesters have changed from v2 to v5. In v2, suggesters supported an index-time payloads option, which was used to store and return metadata with suggestions. In v5, completions are now returned with their associated document in the _source field.

If you have completion suggesters for v2 you will need to make them compatible with v5. This involves:

  • API clients should read metadata from _source instead of payload. For v2 the payload is copied to _source by Records-REST, which allows you to already now upgrade API clients to use the new _source field.
  • On indexing, you need to add the payload only for v2.

Elasticsearch v2 end-of-life

Elasticsearch v2 reaches end of life in February 2018. Elasticsearch v2 support in Invenio will be removed in Invenio v3.1, thus Invenio v3.0 will be released with Elasticsearch v2 support and be maintained until v3.0 end of life (currently TBD).

Sprint: JSONB, bugs & bouncing search

Lars Holm Nielsen Oct 20, 2017 Invenio Framework

This sprint was focused on:

  • Metadata bundle: Finalize Invenio-Records, Invenio-Search, Invenio-JSONSchemas.
  • Fixing new bugs in Base/Auth bundles.

As a result of 40 developer days, 37 commits and 2.8k lines touched (2k additions and 0.8k deletions), the following improvements were implemented:

  • Fixed bouncing of search results (sorting of the same query could change depending on which Elasticsearch node your query would be answered by).
  • JSONB now being used for record storage (Thanks to Javier for the PR).
  • Rendering of JSONSchemas, meaning allOf and $refs can now be resolved on-the-fly to generate an self-contained schema for e.g. deposit forms (Thanks to Pamfilos for the PR).
  • Boring fixes that makes sure the Base and Auth bundles are stable.

List of changes:

  • Invenio-Records (v1.0.0b4):
    • Beta release (release checklist).
    • Changed data storage from JSON to JSONB (requires data migration and PostgreSQL v9.4+) Thanks to Javier for PR.
    • Changed signals receiver signatures (backward-incompatible!).
    • Fixed invalid MARC in demo records causing export errors.
    • Fixed CLI for deletion of records.
    • Fixed fractional seconds problem in MySQL causing tests to fail.
  • Invenio-Search (v1.0.0b1):
    • Beta release (release checklist, documentation)
    • Fix bouncing search results.
    • Bumped Travis PostgreSQL to v9.4 to support JSONB.
  • Invenio-JSONScheams (v1.0.0a7):
    • Fixed double registration of schema endpoint on both UI and API app.
    • Added support for resolving allOf and JSONRefs in JSONSchemas (thanks to Pamfilos for PR).
    • Fixed some documentation issues.
  • Invenio-Records-REST (v1.0.0b3):
    • Fix bouncing search results.
    • Bumped Travis PostgreSQL to v9.4 to support JSONB.
  • Invenio-OAuth2Server (v1.0.0b2):
    • Improved authorization template design (text alignment, cover page usage and display number of users).
    • Broken tests fix due to cryptography package changes.
  • Invenio-DB (v1.0.0b8):
    • Alembic documentation refactored and integrated in docs.
  • Invenio-Records-UI:
    • Bumped Travis PostgreSQL to v9.4 to support JSONB.
  • Invenio-OAIServer (v1.0.0a14):
    • Invenio-Records signals signature compatibility.
  • Invenio-App-ILS:
    • Test for ensuring all records export formats are working.