Impex, Bootstrap, Demo and Local Data

Initial data setup within KOLE happens during the db reset process which essentially consists of the following steps;

  1. Drop OLEFS schema and create a new one.

  2. Load Impex data.

  3. Load Liquibase changesets data divided into three categories (bootstrap, demo and institution specific data).

  4. Apply constraints.

Data is either expressed in .xml files or .csv. It is then converted into .sql by impex and liquibase tools. The generated sql is then run against the db. 


1. Impex data - Schema definition for the various module along with the default values for the corresponding tables in the ole ls,ds and fs modules. This is essentially data in xml format used by the impex tooling that is needed by the system to function.

2. Bootstrap data  - This is data to be used by the liquibase tooling and is generic data which is essential for the system to be up and running. The data is organized under the following folder - ole-app/ole-db/ole-liquibase/ole-liquibase-changeset/src/main/resources/ole-bootstrap

This data consists of the following;

  1. Users: 
    1. admin is a default user that comes as part of the Rice data set and has permissions on Rice document type (all the functions in the "admin" tab can be accessed by this user).K
    2. kr and ole are users for system functions.  They should not be removed or modified.

      Basic User

      ole-quickstart is the user that has permissions to perform basic navigation. You will need to create a similar user with roles/permissions as ole-quickstart for your institution. The default demo data set comes with this user; but if you are going to try the bootstrap data with your institutional load, then a user will need to be created.

       

  2. Roles/permissions: Default roles from KFS and Rice have been retained. Roles specific to library circulations etc have been made part of the bootstrap data sets and pending review can be either left here or moved to the demo set which would mean institutions will come up with their own defined roles)

3. Modifications to impex & bootstrap data - The rare occasion when changes may need to be made to the base table data, it can be done by modifying the files under the following folders:

  1. ole-app/ole-db/ole-impex
    1. ole-impex-ls: This folder contains all the data files in .xml format for tables defined in the schema.xml. Essentially this is for OLE specific tables pertaining to circulation functions etc.
    2. ole-impex-ds: This folder only contains table definitions (schema.xml). 
    3. ole-impex-master: This folder contains all the data files in xml format for KFS/OLEFS tables defined in the schema.xml. This is mainly all the acquisition function reference data.
    4. ole-impex-rice: This folder contains all the data for Rice bootstrap tables in xml format.
    5. ole-impex-test: This module contains one table definition (schema.xml) that is needed for running unit tests in OLE.
  2. ole-app/ole-db/ole-liquibase/ole-liquibase-chageset/src/main/resources/ole-bootstrap - This folder has data organized by data types and is needed for basic setup of users, roles/permissions etc..
     

    Loading Changes

    After modifying, follow the db change process to ensure the updated .sql files have been generated for resetting the database at OLE Database Change Management

4. Demo Data - This is data that is needed for testing purposes. This is currently a mix of IU/Chicago/Duke/Lehigh data sets but mostly IU data for financial aspects of the system.

The data is organized under the following folders;

  1. ole-app/ole-db/ole-liquibase/ole-liquibase-changeset/src/main/resources/ole-demo

5. Modifications to Demo Data : Demo data is in the format of .csv files which is used by liquibase tooling. The data resides under ole-app/ole-db/ole-liquibase/ole-liquibase-changeset/src/main/resources/ole-demo folder.  Updates to existing demo data or creating data for new tables can be applied here. 

6. Files that control order and .csv file loading of bootstrap, demo and institutional data: The liquibase tool needs to know which .csv files are to be loaded and in which order. The various xml files that have this information as follows;

    1. ole-app/ole-db/ole-liquibase/ole-liquibase-changeset/src/main/resources/bootstrap_krcr_data.xml - This file contains liquibase changeset information to populate Rice data.
    2. ole-app/ole-db/ole-liquibase/ole-liquibase-changeset/src/main/resources/bootstrap_krim_entity_data.xml - This file contains liquibase changeset information to populate users and related information.
    3. ole-app/ole-db/ole-liquibase/ole-liquibase-changeset/src/main/resources/bootstrap_krim_role_perm_data.xml - This file contains liquibase changeset information for roles/permissions etc..
    4. ole-app/ole-db/ole-liquibase/ole-liquibase-changeset/src/main/resources/demo_data.xml - This file contains liquibase change set for any demo related table data.
    5. ole-app/ole-db/ole-liquibase/ole-liquibase-changeset/src/main/resources/bootstrap_other_data.xml - This file contains liquibase change set that didn't quite fit any specific category. This will be retired and the information moved to the demo-data.xml file.
    6. ole-app/ole-db/ole-liquibase/ole-liquibase-changeset/src/main/resources/inst_data.xml - This file contains liquibase change set for institution specific requirements.

7. Local or Institution specific data: For implementing institutions who do not want the default demo data that comes as part of the default OLE install can provide their own data. The list of required tables are documented here.

8. Providing Local Data (Institution specific):  Institution specific data can be provided in .csv format under ole-app/ole-db/ole-liquibase/ole-liquibase-changeset/src/main/resources/ole-inst folder and the control file that has the order and required tables can be documented in the particular file ole-app/ole-db/ole-liquibase/ole-liquibase-changeset/src/main/resources/inst_data.xml

9. Maven profiles for generating sql for different kinds of data: Once data is available in the .csv, generating the appropriate sql can be achieved using the following maven profiles;

Before you execute the commands below; you will need the following;

database user LIQUIBASEBLANK/LIQUIBASEBLANK must already exist, 
and have sufficient privileges to drop/create tables in the LIQUIBASEBLANK schema in the db instance.

MySql;

[Replace localhost with your db host]

CREATE USER 'LIQUIBASEBLANK'@'localhost' IDENTIFIED BY 'LIQUIBASEBLANK';
CREATE DATABASE LIQUIBASEBLANK;
GRANT ALL PRIVILEGES ON LIQUIBASEBLANK.* TO 'LIQUIBASEBLANK'@'localhost';

 

  1. Impex - This typically shouldn't have to be run. In the rare occasion that you have to, you can run the following maven command to generate the necessary sql.

    cd ${OLE_DEVELOPMENT_WORKSPACE_ROOT}/ole-app/ole-db/ole-impex
    mvn clean install -Pinst-sql -Dimpex.scm.phase=none (for Institutions)

    mvn clean install -Psql -Dimpex.scm.phase=none (for developers)

  2. Liquibase
    1. bootstap-sql-only profile (bootstrap sql only): This will generate sql for files under /ole-app/ole-db/ole-liquibase/ole-liquibase-changeset/src/main/resources/ole-bootstrap. 

      cd ${OLE_DEVELOPMENT_WORKSPACE_ROOT}/ole-app/ole-db/ole-liquibase/ole-liquibase-changeset
      mvn clean install -Pbootstrap-sql-only,mysql -Dimpex.scm.phase=none (for MySQL)

      mvn clean install -Pbootstrap-sql-only,oracle -Dimpex.scm.phase=none (for Oracle)

    2. inst-mysql profile (for institution specific data loading) : This will generate sql files for the data in the .csv files under /ole-app/ole-db/ole-liquibase/ole-liquibase-changeset/src/main/resources/ole-inst; and you can find the sql files under: ole-app/ole-db/ole-sql/ole-liquibase-sql/src/main/resources/org/kuali/ole/sql/mysql/inst_data.sql

       

      cd ${OLE_DEVELOPMENT_WORKSPACE_ROOT}/ole-app/ole-db/ole-liquibase/ole-liquibase-changeset
      mvn clean install -Pinst-mysql,mysql -Dimpex.scm.phase=none (for MySql)

    3. inst-oracle profile (for institution specific data loading): This will generate Oracle specific sql files for the data in the .csv under /ole-app/ole-db/ole-liquibase/ole-liquibase-changeset/src/main/resources/ole-inst; and you can find the sql files under: ole-app/ole-db/ole-sql/ole-liquibase-sql/src/main/resources/org/kuali/ole/sql/mysql/inst_data.sql

       

      cd ${OLE_DEVELOPMENT_WORKSPACE_ROOT}/ole-app/ole-db/ole-liquibase/ole-liquibase-changeset
      mvn clean install -Pinst-oracle,oracle -Dimpex.scm.phase=none (for Oracle)

    4. sql profile (for demo data only - This includes bootstrap data as default): This will generate sql files for the data in the .csv under /ole-app/ole-db/ole-liquibase/ole-liquibase-changeset/src/main/resources/ole-demo; and you can find the sql files under: ole-app/ole-db/ole-sql/ole-liquibase-sql/src/main/resources/org/kuali/ole/sql/mysql/demo.sql

       

      cd ${OLE_DEVELOPMENT_WORKSPACE_ROOT}/ole-app/ole-db/ole-liquibase/ole-liquibase-changeset
      mvn clean install -Psql,mysql -Dimpex.scm.phase=none (for mysql)

      mvn clean install -Psql,oracle -Dimpex.scm.phase=none (for oracle)

       

Once appropriate data modifications have been done either in the impex or the .csv files and corresponding sql generated, follow the db reset process as explained in Section B of Database Change Management to get data loaded. 

 

Operated as a Community Resource by the Open Library Foundation