Creating and Configuring an Amazon EC2 AMI OSF Instance

From OSF Wiki
Jump to: navigation, search


This documentation page outline all the steps required to create and configure a vanilla Open Semantic Framework version 3.1 EC2 AMI instance. This includes the configuration of all the users, firewall and security settings, along with the creation of a new, non-vanilla, OSF network.

Creating a New OSF EC2 Instance

The first step is to create a new instance Amazon EC2 instance. If you are not familiar with Amazon EC2, you should read the Getting Started with Amazon EC2 Linux Instances guide. The one thing you have to do in the Launch an Amazon EC2 Instance section is to click the "Community AMIs" left tab, and then to search for the AMI ID listed below. Once you see it appearing, you will be able to click the Select button to start creating the OSF instance.

Available AMIs are:

Region arch root store OSF Version Ubuntu Version AMI
us-east-1 64-bit EBS 3.3.0 14.04 LTS ami-70f4bf18
us-west-1 64-bit EBS 3.3.0 14.04 LTS ami-74786231
us-west-2 64-bit EBS 3.3.0 14.04 LTS ami-99280ca9
eu-west-1 64-bit EBS 3.3.0 14.04 LTS ami-d3d05aa4
eu-central-1 64-bit EBS 3.3.0 14.04 LTS ami-b6794bab
sa-east-1 64-bit EBS 3.3.0 14.04 LTS ami-9ead98cc
ap-southeast-2 64-bit EBS 3.3.0 14.04 LTS ami-f7f483cd
ap-southeast-1 64-bit EBS 3.3.0 14.04 LTS ami-cd0bb4d0
ap-northeast-1 64-bit EBS 3.32.0 14.04 LTS ami-89aa4c89

Configure the OSF Instance

Once you created the new OSF instance, and once you logged into your newly created instance, you have to configure it such that it is secure and that it works using the domain name of your choice.

Make yourself root

The first step is to make yourself root before executing the commands outlined below:

# sudo -i

Configure Ubuntu Users

The first step is to create the users that will be used to access the OSF server and to properly configure the SSH daemon.

Create New Administrator User Account

The goal is to create a new administrator user account that is not any default users on a Ubuntu server.

Key Generation and Distribution

This is how to create similar keypairs for all users who need access to your instances.

In the example below (to be run on your local machine, not your EC2 instance) replace "user" with the actual user's login, name or some other unique identifier.

# cd /tmp
# ssh-keygen -b 1024 -f user -t dsa

This will create 2 files:

  • user (private key)
  • (public key)Copy all the public key files that you generated to a temporary place on your instance:
# scp -i root *.pub
Administrator User Account Creation

Log in to the instance as root. For each user you are creating, add the user to your instance with the

# useradd -m -c "firstname lastname" user

For simplicity's sake, use the same "user" name as you did for key generation. Now we need to place the key into their ssh authorized keys file (again, replacing "user" with the username you chose earlier)

# cd ~user
# mkdir .ssh
# chmod 700 .ssh
# chown user:user .ssh
# cat /tmp/ >> .ssh/authorized_keys
# chmod 600 .ssh/authorized_keys
# chown user:user .ssh/authorized_keys

Finally create a new password for that user:

# passwd user

If you want to use the bash shell by default for that new user, just run the following command:

# chsh -s /bin/bash user
Make the User Sudoer

This step is optional. You only perform these steps if you want the new user to be a sudoer.

# vim /etc/sudoers

After the line:

root    ALL=(ALL:ALL) ALL

add the line:

user    ALL=(ALL:ALL) ALL

Finally save the file.

Disable Password-based Login

Log in to your instance as root and edit the ssh daemon configuration file:

# vim /etc/ssh/sshd_config
Note: whenever making changes to your sshd.config file, be certain that you have an active shell session in case you've made a fatal syntax error. After restarting sshd, log in from another session to test it before terminating your active terminal session

# vim /etc/ssh/sshd_config

find the line:

PermitRootLogin without-password

and add the AllowUsers entry and change PermitRootLogin to:

PermitRootLogin no
AllowUsers user

Again, be sure that you have an active login, save the file and restart sshd:

# service ssh restart
Delete Default Ubuntu User

Finally let's delete the default ubuntu user.

First, log into another shell without closing the one you are connected to with the ubuntu user. Once you validate that you can log into the instance using the admin user you just created above, to close the other shell terminal where you are logged in using the ubuntu user. Then delete the ubuntu user:

# userdel -r ubuntu

Configure Firewall

The next step is to properly configure the firewall such that only the necessary ports are open and available on the Internet.

ufw allow ssh/tcp
ufw allow 80/tcp
ufw allow proto tcp from to any port 8890
ufw allow proto tcp from to any port 8983
ufw logging on
ufw enable
ufw status

Here is the explanation of each of these commands:

  1. Enable everyone to send queries to the port 22
  2. Enable everyone to send queries to the port 80
  3. Enable the localhost to send queries on the port 8890 (Virtuoso)
  4. Enable the localhost to send queries on the port 8983 (Solr)
  5. Enable firewall logging
  6. Enable firewall
  7. Get the status of the firewall and make sure it is properly running

Additionally, you can add more ports and IP addresses using modified versions of these commands.

Delete Ontologies

We have to delete the ontologies before re-creating the OSF network that uses the new domain name. You can delete them by using the following commands:

# omt --delete --osf-web-services="http://localhost/ws/"

Then delete all the ontologies that appears in the list of loaded ontologies.

Change Default Passwords

The vanilla Opens Semantic Framework AMI is packed with default passwords. This section covers all the places where default passwords need to be modified.

Important: If you do not change all these passwords, your OSF instance may be at risk!


To change the password of the dba and the dav users, you have to:

# /usr/bin/isql-v

Then, in the isql command line tool, once you are logged-in, to type the following commands:

set password dba NEW-DBA-PASSWORD;

The default password of the dba and dav users is "dba".


To change the password of the admin user, you have to:

# sed -i "s>define('ADMIN_PASSWORD','admin');>define('ADMIN_PASSWORD','NEW-PASSWORD');>" /usr/share/memcached-ui/index.php

The default password of the admin is "admin".

Reconfigure the Open Semantic Framework

Now that we changed all the default passwords, we have to re-configure them into the OSF instance. To reconfigure it, you to:

# vim /data/osf-web-services/configs/osf.ini

The first thing you have to modify is the URL of the OSF instance on the web. By default, it is defined as localhost. To change it, search for the following lines, and update the wsf_base_url setting accordingly:

wsf_base_url = "http://localhost"

Then you have to update the default WSF graph URI. It should be using the same domain that you defined for the [network] setting:

wsf_graph = "http://localhost/wsf/"

Finally you have to update the password of the Virtuoso server as you defined it for the dba user above:

password = "dba"

Reconfigure API Key

You have to create a new API Key for the administer Application ID. You can generate a unique 32 characters API Key by using the following command:

# php -r 'echo "\n\n".strtoupper(bin2hex(openssl_random_pseudo_bytes(16)))."\n\n";'

Then run the following command to change the API key you just created into the registry of API Keys:

# sed -i "s>E74ACB52F3F0E54764C786BFBB438E4E>NEW-API-KEY>" /data/osf-web-services/configs/keys.ini
Update API Key References

Once the API Key is updated, we have to update its references in few different configuration files. Run the following commands to update the API Key setting with the new key:

# sed -i "s>E74ACB52F3F0E54764C786BFBB438E4E>NEW-API-KEY>" /usr/share/ontologies-management-tool/omt.ini
# sed -i "s>E74ACB52F3F0E54764C786BFBB438E4E>NEW-API-KEY>" /usr/share/datasets-management-tool/dmt.ini
# sed -i "s>E74ACB52F3F0E54764C786BFBB438E4E>NEW-API-KEY>" /usr/share/permissions-management-tool/pmt.ini 
# sed -i "s>E74ACB52F3F0E54764C786BFBB438E4E>NEW-API-KEY>" /usr/share/osf/StructuredDynamics/osf/tests/Config.php

Then edit the domain reference in the same files:

# sed -i "s>localhost>OSF-NETWORK-DOMAIN>" /usr/share/ontologies-management-tool/omt.ini 
# sed -i "s>localhost>OSF-NETWORK-DOMAIN>" /usr/share/datasets-management-tool/dmt.ini
# sed -i "s>localhost>OSF-NETWORK-DOMAIN>" /usr/share/permissions-management-tool/pmt.ini
# sed -i "s>http://localhost>http://OSF-NETWORK-DOMAIN>" /usr/share/osf/StructuredDynamics/osf/tests/Config.php

Re-create the OSF Network

Delete Vanilla Network

Log into Virtuoso to delete the vanilla network:

# /usr/bin/isql-v

Then once you are logged into Virtuoso, run the following command:

sparql clear graph <http://localhost/wsf/>;
Recreate Network

To re-create the network using the new setup, we will use one of the script used by the OSF Installer to create the network. Run the following set of commands, and properly specify the information in bold:

# cp /usr/share/osf-installer/resources/virtuoso/initialize_osf_web_services_network.php /tmp/
# sed -i "s>\"dba\", \"dba\">\"dba\", \"VIRTUOSO-DBA-PASSWORD\">" /tmp/initialize_osf_web_services_network.php
# sed -i "s>$server_address = \"http://localhost\";>$server_address = \"http://OSF-NETWORK-DOMAIN\";>" /tmp/initialize_osf_web_services_network.php
# php /tmp/initialize_osf_web_services_network.php
# rm /tmp/initialize_osf_web_services_network.php
# /usr/bin/isql-v 1111 dba VIRTUOSO-DBA-PASSWORD /tmp/init_osf.sql
# rm /tmp/init_osf.sql

Re-import the Ontologies

Now that the OSF network is re-created, we have to re-import the core ontologies:

# omt --load-advanced-index="true" --load-all --load-list="/usr/share/osf-installer/resources/osf-web-services/ontologies.lst" --osf-web-services="http://OSF-NETWORK-DOMAIN/ws/"

Run Tests Suites

Now that we re-configured a vanilla EC2 OSF instance, we will rerun the OSF Web Services Tests Suites to make sure that everything is still properly working with all the new settings. To run the tests suites, you have to perform the following commands:

# cd /usr/share/osf/StructuredDynamics/osf/tests/
# phpunit --configuration phpunit.xml --verbose --colors --log-junit log.xml

If all the tests pass, it means that your new OSF instance is properly re-configured.

Install OSF for Drupal

An additional step you can do is to install OSF for Drupal on that new server, or on any other servers that will use this new OSF instance:

# sed -i "s>localhost>OSF-NETWORK-DOMAIN>" /usr/share/osf-installer/installer.ini
# cd /usr/share/osf-installer/
# ./osf-installer --install-osf-drupal