A deep dive into StructrAdvanced Topics

This chapter covers the following topics:

Indexing

Passive Indexing

Passive indexing is the term for reading a dynamic value from a property (e.g. Function Property or Boolean Property) at application level, and writing it into the database at the end of each transaction, so the value is visible to Cypher. This is important for BooleanProperty, because its getProperty() method returns false instead of null even if there is no actual value in the database. Hence a Cypher query for this property with the value false would not return any results. Structr resolves this by reading all passively indexed properties of an entity, and writing them into the database at the end of a transaction.

One of the key features of Structr are Function Properties, which allow you to run a script expression whenever a property value on a node is read or written.

Periodic Task Scheduler

Structr provides a service for periodic task scheduling: the CronService. It is a background service that can be configured in the structr.conf configuration file to schedule the execution of periodic tasks.

Tasks in this context are global schema methods and are executed as an admin user.

The main configuration key is CronService.tasks. It accepts a whitespace-separated list of global schema method names. These are the tasks that are registered with the CronService.

For each of those tasks, a cronExpression has to be configured which determines the execution time/date of the task. It consists of seven fields and looks similar to crontab entries in Unix-based operating systems:

<methodName>.cronExpression = <s> <m> <h> <dom> <m> <dow>
Field  Explanation Value Range
<methodName>  name of the global schema method any existing global schema method
<s>  seconds of the minute 0-59
<m>  minute of the hour 0-59
<h>  hour of the day 0-23
<dom>  day of the month 0-31
<m>  month of the year 1-12
<dow>  day of the week 0-6 (0 = Sunday)

There are several supported notations for the fields:

Notation Meaning
* Execute for every possible value of the field.
x Execute at the given field value.
x-y  Execute for value x up to value y of the field.
*/x  Execute for every multiple of x in the field (in relation to the next bigger field).
Example Meaning
Hours = * Execute at every full hour.
Hours = 12 Execute at 12 o’clock.
Hours = 12-16 Execute at 12, 13, 14, 15, 16 o’clock.
Seconds = */15 In every minute, execute at 0, 15, 30, 45 seconds.
Seconds = */23 Special case: In every minute, execute at 0, 23, 46 seconds. If the unit is not evenly divisible by the given number the last interval is shorter than the others.
Full Example
# CronService is included in the configured.services setting (and thus active)
configured.services = NodeService SchemaService HttpService CronService

# three global schema method names are registered as tasks
CronService.tasks = cleanUpMethod1 cleanUpMethod2 exampleMethod

# Run every 30 minutes
cleanUpMethod1.cronExpression = 0 0,30 * * * *

# Run every 15 minutes
cleanUpMethod2.cronExpression = 0 */15 * * * *

# Run each hour on Mondays during the first 10 days in April and October
exampleMethod.cronExpression = 0 0 * 1-10 4,10 1
Considerations
  • The CronService must be included in the structr.conf setting configured.services to be activated
  • When a cronExpression is added, deleted or edited, the CronService has to be restarted for the changes to take effect. This can be done at runtime via the configuration tool or by restarting structr
  • By default, structr prevents the same cron task to run run while the previous execution is still running. This can be changed via the setting CronService.allowparallelexecution in structr.conf.

Maintenance Commands

Maintenance commands are tools for typical administrative tasks to be performed on the server. They can be executed using the built-in maintenance() function, with the Admin Console or by making a HTTP POST request to the Maintenance Resource, followed by the command name.

Examples

Using the maintenance() function in a JavaScript context

$.maintenance('rebuildIndex', {
"type": "Page"
});

Using a REST POST call

POST /structr/rest/maintenance/rebuildIndex
{
"type": "Page"
}

Using the AdminShell mode of the Admin Console.

Mode set to 'AdminShell'. Type 'help' to get a list of commands.
admin@Structr> init node index for Page
Starting (re-)indexing all nodes of type Page
RebuildNodeIndex: 1 objects processed
RebuildNodeIndex: 1 objects processed
Done with (re-)indexing 1 nodes
admin@Structr>

Available Commands

The following commands are available.

changeNodePropertyKey

Migrates property values from one property key to another. This can for example be used to move all name values to a description property.

Parameters
Name Description
oldKey the source key to migrate the value from
newKey the target key to migrate the value to

clearDatabase

Clears the database, i.e. removes all nodes and relationships and restores the initial schema as if it were an empty Structr instance.

This action cannot be reversed, it will delete your application and all data in the database, even non-Structr nodes and relationships.

copyRelationshipProperties

Copies relationship properties from one key to another.

Parameters
Name Description
sourceKey the source key to migrate the value from
destKey the target key to migrate the value to

createLabels

Updates the type labels of an object. This command looks at the value in the type property of an object and tries to identify a corresponding schema type. If the schema type exists, it creates a type label on the object for each type in the inheritance hierarchy and removes labels that don’t have a corresponding type.

This command will only work for objects that have a value in their type property.

Parameters
Name Description
type node type to limit creation of labels to

deploy

Creates a Deployment Export or Import of the application. This command reads or writes a text-based export of the application (without its data!) that can be stored in a version control system. The maintenance command is used internally in the Dashboard section.

Parameters
Name Value
mode deployment mode (import/export)
source source directory when importing
target target directory when exporting
extendExistingApp incremental import (true/false)

deployData

Creates a Data Deployment Export or Import of the application data. This command reads or writes a text-based export of the application data (not the application itself) that can be stored in a version control system.

Parameters
Name Value
mode deployment mode (import/export)
source source directory when importing
target target directory when exporting
types comma-separated list of data types to export

directFileImport

Imports files from a directory in the local filesystem. The files can either be copied or moved (i.e. deleted after copying into Structr), depending on the mode parameter. The existing parameter determines how Structr handles existing files in the Structr Filesystem. The index parameter allows you to enable or disable indexing for the imported files.

Please note that when using Docker, you first have to copy the files to the Docker container, or use a files volume.

Parameters
Name Description
source the source directory to import files from
mode (copy|move)
existing (skip|overwrite|rename)
index (true|false)

fixNodeProperties

Tries to fix properties in the database that have been stored with the wrong type. This command can be used to convert property values whose property type was changed, e.g. from String to Integer.

Parameters
Name Description
name name of the property that needs fixing
type node type to which the property belongs

flushCaches

Clears all internal caches. This command can be used to reduce the amount of memory consumed by Structr, or to fix possible cache invalidation errors.

letsencrypt

Triggers creation or update of an SSL certificate using Let’s Encrypt.

Please note that the configuration setting letsencrypt.domains must contain the full domain name of the server you want to create the certificate for.

Parameters
Name Description
server required (staging|production) Staging mode is meant for testing and will generate invalid dummy certificates only, while production creates real, valid certificates but is throttled.
challenge optional Overwrite the default challenge method as set in structr.conf. This is convenient to test an alternative challenge type without the need to restart the Structr instance.
wait optional Let the client wait for the given period (in seconds) in order to have enough time to prepare the DNS TXT record in case of the dns challenge type, or the HTTP response in case of the http challenge.
reload optional Reload the HTTPS certificate after updating it. Allows using the new certificate without restarting.

maintenanceMode

Enables or disables the maintenance mode. In maintenance mode, most services are stopped and some other services are restarted on different ports to enable maintenance work to be done without interference.

When the maintenance mode is started, the following services are shut down:

  • FtpService
  • HttpService
  • SSHService
  • AgentService
  • CronService
  • DirectoryWatchService
  • LDAPService
  • MailService

After a brief moment, the following services are restarted with another port:

  • FtpService
  • HttpService
  • SSHService

If for example a cron job is running, it will not be halted. Only the services are stopped so no NEW processes are started. Active processes will keep running until they are finished.

Parameters
Name Description
action required (enable|disable) Enables or disables the maintenance mode.

rebuildIndex

Rebuilds the internal indexes, either for nodes, or for relationships, or for both. Rebuilding the index means that all objects are first removed from the index and then added to the index again with all properties that have the indexed flag set.

Parameters
Name Description
type limit the execution to a certain node type
relType limit to a certain relationship type
mode to rebuild the index only for nodes or relationships (nodesOnly | relsOnly)

rebuildIndexForType

Rebuilds the internal indexes, either for nodes, or for relationships, or for both. Rebuilding the index means that all objects are first removed from the index and then added to the index again with all properties that have the indexed flag set.

This maintenance command is just an alias for the rebuildIndex command above and will likely be removed in a future release.

Parameters
Name Description
type limit the execution to a certain node type
relType limit to a certain relationship type
mode to rebuild the index only for nodes or relationships (nodesOnly | relsOnly)

setNodeProperties

Sets property values on all nodes of a certain type.

Parameters
Name Description
type The database type on which the properties should be set.
newType used to update the type property of nodes (because type is already taken)

If this command is used to change the type property of nodes, the Create Labels command has to be called afterwards to update the labels of the changed nodes - otherwise they will not be accessible.

setRelationshipProperties

Sets property values on all relationships of a certain type. Please note that you cannot set the type property of relationships this way.

Parameters
Name Description
type The database type on which the properties should be set.

setUuid

Adds UUIDs to all nodes and relationships that don’t have a value in their id property.

Parameters
Name Value
type limit the execution to a certain node type
relType limit to a certain relationship type
allNodes process all nodes
allRels process all relationships

snapshot

Creates a schema snapshot and writes it to a local file on the server the Structr instance is running on. This maintenance command is used internally in the Schema section to manage schema snapshots.

Parameters
Name Value
mode the snapshot mode (exportrestoreadddeletepurge)
name the name of the input / output file
types an optional list of types to export

sync

Exports or imports partial database content, based on a query, into a ZIP file. You can specify a Cypher query for the desired export set.

Parameters
Name Value
mode operation mode (import/export)
file data file when importing or exporting (e.g. my-data.zip)
query Cypher query that returns the desired export subgraph

Changelog

The changelog can be activated by setting the structr.conf switch application.changelog.enabled to true. A Structr instance started with changelog enabled keeps a record of every modification of an entity. There are different kinds of actions (called verb) for which a record is kept.

The changelog itself can be retrieved using the builtin changelog() function. The changelog is written to disk at changelog.path with the subfolders n for nodes and r for relationships. If user-centric changelog (application.changelog.user_centric.enabled) is enabled the changelog is written to the subfolder u. In older versions of structr the changelog was written to the structrChangeLog property of a node. The following tables show the possible keys/values of modification records. Modification records are stored as a JSON string. (see the example below)

Verb Event type Keys in changeset
create Creation of an object verb,time,userId,userName,target
delete Deletion of an object verb,time,userId,userName,target
link A relationship to/from another object has been created verb,time,userId,userName,target, rel
unlink A relationship to/from another object has been removed verb,time,userId,userName,target, rel
change One of the properties of the object has been changed verb,time,userId,userName,key, prev, val
Key Content
verb type of changelog event (One of create, change, delete, link, unlink)
time Timestamp of the change (ms since epoch)
userId id of the user, 00000000000000000000000000000000 for SuperUser, null for anonymous
userName name of the user who triggered the modification
target id of the target node of the new or deleted relationship
relId id created/deleted relationship
rel relationship type of the created/deleted relationship
relDir relationship direction of the created/deleted relationship (in/out)
key Property key of the modified property
prev Previous value of the modified property
val New value of the modified property
Example
{"time":1455195862431,"userId":"f02e59a47d[...]","userName":"admin","verb":"change","key":"name","prev":null,"val":"My new name"}
{"time":1455195903852,"userId":"f02e59a47d[...]","userName":"admin","verb":"change","key":"name","prev":"My new name","val":"New Name"}
{"time":1455196049579,"userId":"f02e59a47d[...]","userName":"admin","verb":"link","rel":"has","relId":"97d26b5778[...]026eb615e","relDir":"out","target":"4e32a9f6eb[...]bd7a6a"}
{"time":1455195961348,"userId":"f02e59a47d[...]","userName":"admin","verb":"unlink","rel":"has","relId":"97d26b5778[...]026eb615e","relDir":"out","target":"4e32a9f6eb[...]bd7a6a"}
{"time":1455196115875,"userId":"00000000000000000000000000000000","userName":"superadmin","verb":"unlink","rel":"OWNS","relId":"b29e98329fb[...]864aa038","relDir":"out","target":"f02e59a47d[...]"}
Note

The file-based changelog can consist of many files. You should make sure the file system can handle the amount of files. Depending on the file system it can happen that no more files can be written even if enough space is available.

Backup Options

Cold backup

To backup a Structr installation, do the following:

  • stop the Structr instance
  • copy db/ and files/ directories to a safe place
  • copy structr.conf to a safe place
  • start instance

Structr Services Layer

The Structr Services Layer is a part of the Structr architecture. It manages the configured services that run in an instance.

SSHService

The SSHService allows admin users to connect to the structr instance via the SSH protocol. The SSH port can be configured via the application.ssh.port key in structr.conf.

Authentication can be done via public key or via password (unless explicitly forbidden via application.ssh.forcepublickey).

After successful authentication the SSH connection allows exactly the same functionality as the Admin Console but via SSH.

Public Key Authentication

The users public key can be stored in the property publicKey on the user node. The transmitted key is compared to the one stored on the user node. If successful, login is granted.

Password Authentication

Authentication works just like regular password-based authentication. The given password is salted, hashed and compared to the stored password hash.

Example
$ ssh -p 8022 admin@localhost
Password authentication
Password: *****
Welcome to the Structr 3.6-SNAPSHOT 27066 202006301221 JavaScript console. Use <Shift>+<Tab> to switch modes.
admin@Structr/>

FtpService

The FtpService allows users to connect to the structr instance via the FTP protocol. The FTP port can be configured via the application.ftp.port key in structr.conf.

Authentication is only possible via password. After successful authentication the FTP connection lists all the files a user has read rights to starting from the root directory.
Files which the user is allowed to read, but which reside in a directory which the user is not allowed to “see”, the file will not show up in the FTP listing.

Regular users will also not be able to see file owner if they do not have read rights on those nodes.

$ lftp -p 8021 -u user1 localhost
Password:
lftp user1@localhost:~> DIR
drwx------ 1 0 Jun 30 15:22 testFolder
-rw------- 1 user1 347 Jun 30 09:24 test1.txt
-rw------- 1 25 Jun 30 15:41 test2.txt
-rw------- 1 5 Jun 30 09:24 test3.txt
-rw------- 1 user1 5 Jun 30 09:24 test4.txt

Admin users always see every file/folder available as well as their owner.

$ lftp -p 8021 -u admin localhost
Password: *****
lftp admin@localhost:~> DIR
drwx------ 1 admin 0 Jun 30 15:22 testFolder
-rw------- 1 user1 347 Jun 30 09:24 test1.txt
-rw------- 1 admin 25 Jun 30 09:24 test2.txt
-rw------- 1 user2 5 Jun 30 09:24 test3.txt
-rw------- 1 user1 5 Jun 30 09:24 test4.txt
lftp admin@localhost:/>

Events

Structr provides events which can be implemented to influence behavior. The four most important ones are grouped as so-called Lifecycle Methods. There are also 2 global events and one event specific to nodes of type File (and types inheriting from File).

These events are only triggered if the trigger action is done via structr. For example, setting the name of a node via Cypher does not trigger the onSave method of that node type to run because the structr layer is not used.

Lifecycle Methods

Lifecycle events are events that occur during the lifecycle of an object in the database. A lifecycle method is an optional method on any type of the data model that will be executed when a lifecycle event occurs. There are several that can be bound to different events that happen in a Structr application. When a data object is created, modified or deleted in the database, the corresponding lifecycle method is called (if it exists on that particular type).

onCreate

  • Is called when an object is created (but the transaction is not yet committed)
  • Multiple onCreate methods can exist per type - every method prefixed with onCreate is called when an object is created (e.g. onCreate01, onCreate02)
  • Schema constraint checks (uniqueness constraint, notNull constraint) are performed after onCreate has run
  • can be rolled back
    • if an error occurs
    • if an error is raised via error() function
    • if a constraint is violated
  • All regular keywords can be used - no special keywords are introduced for this method

Example:

{
let self = Structr.get('this');

if (self.name === 'foo') {
// don't allow Nodes named "foo"
Structr.error('Type', 'name', 'name can not be "foo"!');
} else {
Structr.log('Node with name ' + self.name + ' has just been created.');
}
}

afterCreate

  • Is called after an object has been created (onCreate has already been executed), all checks have passed successfully and the transaction has been committed
  • Multiple afterCreate methods can exist per type - every method prefixed with afterCreate is called after an object has been created successfully (e.g. afterCreate01, afterCreate02)
  • can not roll back the transaction
  • is useful for actions that should only happen if the node is sure to have been created. For example sending a mail: Calling send_html_mail() in onCreate would send the email even if the transaction would be rolled back due to an error.
  • All regular keywords can be used - no special keywords are introduced for this method

Example:

{
let self = Structr.get('this');

let subject = 'Node was created!';
let text = 'Node named ' + self.name + ' was created';

Structr.send_html_mail('info@structr.com', 'Structr', 'user@domain.com', 'Test User', subject, text, text);
}

onSave

  • Is called when an object is modified. (Either modification of local attributes of relationships. When a relationship is created, the onSave is triggered for both nodes)
  • creation of an object is not a modification, thus the onSave is not run when an object is created (even if the object is ‘modified’ in onCreate)
  • Multiple onSave methods can exist per type - every method prefixed with onSave is called when an object is created (e.g. onSave01, onSave02)
  • Schema constraint checks (uniqueness constraint, notNull constraint) are performed after onSave has run
  • can be rolled back
    • if an error occurs
    • if an error is raised via error() function
    • if a constraint is violated
  • All regular keywords can be used - one special keyword is introduced for this method

The state of the object in the scripting environment represents the state of the object after the modification. To get information about the previous state the keyword modifications can be used.

Keyword Value
modifications Object containing the modifications made to the object in the current operation

Example:

{
let self = $.get('this');
let modifications = $.retrieve('modifications');

if (modifications.after.name === 'foo') {
// don't allow Nodes named "foo"
Structr.error('Type', 'name', 'name can not be "foo"!');
}
}
Modifications

The modifications object is a read-only representation of all changes to an object in the current transaction. It consists of 4 elements:

Key Value
after The changed local attributes of the node after the modification
before The previous values of all changed local attributes
added Remote attributes (relationships) that were added
removed Remote attributes (relationships) that were removed

Example 1: The visibility flags visibleToAuthenticatedUsers and visibleToPublicUsers were both set from false to true.

{
"before":{"visibleToAuthenticatedUsers":false,"visibleToPublicUsers":false},
"after":{"visibleToAuthenticatedUsers":true,"visibleToPublicUsers":true},
"added":{},
"removed":{}
}

Example 2: The owner for a node was removed. The value of the owner property in the modifications object is a single string because the relationship is defined as a “to one” relationship (a node can only have one owner).

{
"before":{"owner":null},
"after":{"owner":null},
"added":{},
"removed":{"owner":"5ba37699ca8f4a8b92ded77c93629f0e"}
}

Example 3: The user with id 5ba37699ca8f4a8b92ded77c93629f0e was removed from the maintains remote attribute. The value of maintains is an array of strings because the relationship is defined as a “to many” relationship.

{
"before":{"maintains":null},
"after":{"maintains":[]},
"added":{},
"removed":{"maintains":["5ba37699ca8f4a8b92ded77c93629f0e"]}
}

onDelete

  • is run after an object has been deleted (but the transaction has not yet been committed)
  • Using the this keyword results in an error as the object has already been deleted
  • can be rolled back
    • if an error occurs
    • if an error is raised via scripting See <a data-id="651d6bf220e3422488fa61a34109745c" class="mention">error()</a> function
  • All regular keywords can be used - no special keywords are introduced for this method

Global Events

Currently there are two global events which can only be implemented via global schema methods.

onStructrLogin

  • called when a User logs in
  • only this exact function name is called and only as a global schema method
  • Is run in a super-user context (not in the context of the user logging in!)
  • All regular keywords can be used - one special keyword is introduced for this method
Special keyword Function
user The user logging in

Example:

{
let user = Structr.retrieve('user');
Structr.log(user.name + ' just logged in');
}

onStructrLogout

  • called when a User logs out
  • only this exact function name is called and only as a global schema method
  • Is run in a super-user context (not in the context of the user logging out!)
  • All regular keywords can be used - one special keyword is introduced for this method
Special keyword Function
user The user logging out
{
let user = Structr.retrieve('user');
Structr.log(user.name + ' just logged out');
}

onAcmeChallenge

This method is called when an ACME challenge of type dns is triggered, typically by using the maintenance method letsencrypt like this:

$.maintenance('letsencrypt', { server: 'production', challenge: 'dns', wait: '60', reload: true });

The primary use case of this method is creating a DNS TXT record via an external API call to a DNS provider.

The following parameters are passed to this method and can be retrieved with $.retrieve(key):

Parameter name Description Examples
type The type of the ACME authorisation challenge dns
domain The domain the ACME challenge is created for subdomain.example.com
record The name of the DNS record including prefix _acme-challenge. and suffix . _acme-challenge.subdomain.example.com.
digest The token string that is probed by the ACME server to validate the challenge X6Rea0DdZ-5XGotp1geAxfsdDR0x1T9d_kAseA4YMCA

Type-specific events

Currently there are only tow type-specific events. They are only applicable to nodes of type File and types inheriting from it. They can only be implemented by creating an onDownload and onUpload method on the type.

onDownload

  • Is called after a file has been requested for download
  • can not prevent the download of a file
  • All regular keywords can be used - no special keywords are introduced for this method
{
$.log($.this.name + ' was just downloaded');
}

onUpload

Available in structr v4.2+

  • Is called after a file has been uploaded
  • can not prevent the upload of a file
  • All regular keywords can be used - no special keywords are introduced for this method
{
$.log($.this.name + ' was just uploaded');
}

onOAuthLogin

Available in structr v4.2+

  • Is called after a user successfully logs into the system via a configured OAuth provider
  • can not prevent the login of a user
  • Is called with the two parameters provider containing the OAuth provider name that was used for login and userinfo containing the information pulled from the userinfo endpoint of the OAuth provider.
{
$.log('User ' + $.this.name + ' just logged in via ' + $.methodParameters.provider);
$.log('User information: ', JSON.stringify($.methodParameters.userinfo, null, 2))
}

Java-Logging

Structr is written in Java, and we use the Log4J logging framework (the API) and the Logback implementation of that API. The Logback configuration lives in the classpath, in a file called logback.xml. Structr contains a logback.xml file that specifies settings for the console appender and the file appender which writes to /var/log/structr.conf.

logback-include.xml

The logback configuration file supplied with Structr contains a reference to an optional `logback-include.xml’ in which additional settings can be made. In this file you can change the log level for individual Java packages to gain a more detailed insight into internal processes.

XMPP

Structr provides a scriptable XMPP Client Implementation based on the XMPPClient entity.

XMPPClient is a base type that provides a Scriptable XMPP Client Implementation.

XMPPClient extends AbstractNode and provides the following additional properties.

Name Type Description
xmppUsername String The username of this XMPPClient
xmppPassword String The password for this XMPPClient
xmppService String The service name to connect to
xmppHost String The host name to connect to
xmppPort Integer The port to connect to
isEnabled Boolean A boolean value that indicates whether this XMPPClient is enabled (should connect to a server and receive messages)
isConnected Boolean A boolean value that indicates whether this XMPPClient is connected to a server

Example:

let xmppClient = $.this;
xmppClient.doJoinChat('chatRoom@conference.server.org', 'nickname', 'password');