A deep dive into StructrAdvanced Topics
This chapter covers the following topics:
Indexing
Passive Indexing
Passive indexing is the term for reading a dynamic value from a property (e.g. Function Property or Boolean Property) at application level, and writing it into the database at the end of each transaction, so the value is visible to Cypher. This is important for BooleanProperty, because its getProperty() method returns false instead of null even if there is no actual value in the database. Hence a Cypher query for this property with the value false would not return any results. Structr resolves this by reading all passively indexed properties of an entity, and writing them into the database at the end of a transaction.
One of the key features of Structr are Function Properties, which allow you to run a script expression whenever a property value on a node is read or written.
Periodic Task Scheduler
Structr provides a service for periodic task scheduling: the CronService
. It is a background service that can be configured in the structr.conf configuration file to schedule the execution of periodic tasks.
Tasks in this context are global schema methods and are executed as an admin user.
The main configuration key is CronService.tasks
. It accepts a whitespace-separated list of global schema method names. These are the tasks that are registered with the CronService.
For each of those tasks, a cronExpression
has to be configured which determines the execution time/date of the task. It consists of seven fields and looks similar to crontab entries in Unix-based operating systems:
<methodName>.cronExpression = <s> <m> <h> <dom> <m> <dow>
Field | Explanation | Value Range |
---|---|---|
<methodName> | name of the global schema method | any existing global schema method |
<s> | seconds of the minute | 0-59 |
<m> | minute of the hour | 0-59 |
<h> | hour of the day | 0-23 |
<dom> | day of the month | 0-31 |
<m> | month of the year | 1-12 |
<dow> | day of the week | 0-6 (0 = Sunday) |
There are several supported notations for the fields:
Notation | Meaning |
---|---|
* | Execute for every possible value of the field. |
x | Execute at the given field value. |
x-y | Execute for value x up to value y of the field. |
*/x | Execute for every multiple of x in the field (in relation to the next bigger field). |
Example | Meaning |
---|---|
Hours = * | Execute at every full hour. |
Hours = 12 | Execute at 12 o’clock. |
Hours = 12-16 | Execute at 12, 13, 14, 15, 16 o’clock. |
Seconds = */15 | In every minute, execute at 0, 15, 30, 45 seconds. |
Seconds = */23 | Special case: In every minute, execute at 0, 23, 46 seconds. If the unit is not evenly divisible by the given number the last interval is shorter than the others. |
Full Example
# CronService is included in the configured.services setting (and thus active)
configured.services = NodeService SchemaService HttpService CronService
# three global schema method names are registered as tasks
CronService.tasks = cleanUpMethod1 cleanUpMethod2 exampleMethod
# Run every 30 minutes
cleanUpMethod1.cronExpression = 0 0,30 * * * *
# Run every 15 minutes
cleanUpMethod2.cronExpression = 0 */15 * * * *
# Run each hour on Mondays during the first 10 days in April and October
exampleMethod.cronExpression = 0 0 * 1-10 4,10 1
Considerations
- The CronService must be included in the structr.conf setting
configured.services
to be activated - When a
cronExpression
is added, deleted or edited, the CronService has to be restarted for the changes to take effect. This can be done at runtime via the configuration tool or by restarting structr - By default, structr prevents the same cron task to run run while the previous execution is still running. This can be changed via the setting
CronService.allowparallelexecution
in structr.conf.
Maintenance Commands
Maintenance commands are tools for typical administrative tasks to be performed on the server. They can be executed using the built-in maintenance() function, with the Admin Console or by making a HTTP POST request to the Maintenance Resource, followed by the command name.
Examples
Using the maintenance() function in a JavaScript context
$.maintenance('rebuildIndex', {
"type": "Page"
});
Using a REST POST call
POST /structr/rest/maintenance/rebuildIndex
{
"type": "Page"
}
Using the AdminShell mode of the Admin Console.
Mode set to 'AdminShell'. Type 'help' to get a list of commands.
admin@Structr> init node index for Page
Starting (re-)indexing all nodes of type Page
RebuildNodeIndex: 1 objects processed
RebuildNodeIndex: 1 objects processed
Done with (re-)indexing 1 nodes
admin@Structr>
Available Commands
The following commands are available.
changeNodePropertyKey
Migrates property values from one property key to another. This can for example be used to move all name
values to a description
property.
Parameters
Name | Description |
---|---|
oldKey | the source key to migrate the value from |
newKey | the target key to migrate the value to |
clearDatabase
Clears the database, i.e. removes all nodes and relationships and restores the initial schema as if it were an empty Structr instance.
This action cannot be reversed, it will delete your application and all data in the database, even non-Structr nodes and relationships.
copyRelationshipProperties
Copies relationship properties from one key to another.
Parameters
Name | Description |
---|---|
sourceKey | the source key to migrate the value from |
destKey | the target key to migrate the value to |
createLabels
Updates the type labels of an object. This command looks at the value in the type property of an object and tries to identify a corresponding schema type. If the schema type exists, it creates a type label on the object for each type in the inheritance hierarchy and removes labels that don’t have a corresponding type.
This command will only work for objects that have a value in their type property.
Parameters
Name | Description |
---|---|
type | node type to limit creation of labels to |
deploy
Creates a Deployment Export or Import of the application. This command reads or writes a text-based export of the application (without its data!) that can be stored in a version control system. The maintenance command is used internally in the Dashboard section.
Parameters
Name | Value |
---|---|
mode | deployment mode (import/export) |
source | source directory when importing |
target | target directory when exporting |
extendExistingApp | incremental import (true/false) |
deployData
Creates a Data Deployment Export or Import of the application data. This command reads or writes a text-based export of the application data (not the application itself) that can be stored in a version control system.
Parameters
Name | Value |
---|---|
mode | deployment mode (import/export) |
source | source directory when importing |
target | target directory when exporting |
types | comma-separated list of data types to export |
directFileImport
Imports files from a directory in the local filesystem. The files can either be copied or moved (i.e. deleted after copying into Structr), depending on the mode
parameter. The existing
parameter determines how Structr handles existing files in the Structr Filesystem. The index
parameter allows you to enable or disable indexing for the imported files.
Please note that when using Docker, you first have to copy the files to the Docker container, or use a files volume.
Parameters
Name | Description |
---|---|
source | the source directory to import files from |
mode | (copy|move) |
existing | (skip|overwrite|rename) |
index | (true|false) |
fixNodeProperties
Tries to fix properties in the database that have been stored with the wrong type. This command can be used to convert property values whose property type was changed, e.g. from String
to Integer
.
Parameters
Name | Description |
---|---|
name | name of the property that needs fixing |
type | node type to which the property belongs |
flushCaches
Clears all internal caches. This command can be used to reduce the amount of memory consumed by Structr, or to fix possible cache invalidation errors.
letsencrypt
Triggers creation or update of an SSL certificate using Let’s Encrypt.
Please note that the configuration setting letsencrypt.domains
must contain the full domain name of the server you want to create the certificate for.
Parameters
Name | Description |
---|---|
server | required (staging|production) Staging mode is meant for testing and will generate invalid dummy certificates only, while production creates real, valid certificates but is throttled. |
challenge | optional Overwrite the default challenge method as set in structr.conf. This is convenient to test an alternative challenge type without the need to restart the Structr instance. |
wait | optional Let the client wait for the given period (in seconds) in order to have enough time to prepare the DNS TXT record in case of the dns challenge type, or the HTTP response in case of the http challenge. |
reload | optional Reload the HTTPS certificate after updating it. Allows using the new certificate without restarting. |
maintenanceMode
Enables or disables the maintenance mode. In maintenance mode, most services are stopped and some other services are restarted on different ports to enable maintenance work to be done without interference.
When the maintenance mode is started, the following services are shut down:
- FtpService
- HttpService
- SSHService
- AgentService
- CronService
- DirectoryWatchService
- LDAPService
- MailService
After a brief moment, the following services are restarted with another port:
- FtpService
- HttpService
- SSHService
If for example a cron job is running, it will not be halted. Only the services are stopped so no NEW processes are started. Active processes will keep running until they are finished.
Parameters
Name | Description |
---|---|
action | required (enable|disable) Enables or disables the maintenance mode. |
rebuildIndex
Rebuilds the internal indexes, either for nodes, or for relationships, or for both. Rebuilding the index means that all objects are first removed from the index and then added to the index again with all properties that have the indexed
flag set.
Parameters
Name | Description |
---|---|
type | limit the execution to a certain node type |
relType | limit to a certain relationship type |
mode | to rebuild the index only for nodes or relationships (nodesOnly | relsOnly) |
rebuildIndexForType
Rebuilds the internal indexes, either for nodes, or for relationships, or for both. Rebuilding the index means that all objects are first removed from the index and then added to the index again with all properties that have the indexed
flag set.
This maintenance command is just an alias for the rebuildIndex
command above and will likely be removed in a future release.
Parameters
Name | Description |
---|---|
type | limit the execution to a certain node type |
relType | limit to a certain relationship type |
mode | to rebuild the index only for nodes or relationships (nodesOnly | relsOnly) |
setNodeProperties
Sets property values on all nodes of a certain type.
Parameters
Name | Description |
---|---|
type | The database type on which the properties should be set. |
newType | used to update the type property of nodes (because type is already taken) |
If this command is used to change the type
property of nodes, the Create Labels command has to be called afterwards to update the labels of the changed nodes - otherwise they will not be accessible.
setRelationshipProperties
Sets property values on all relationships of a certain type. Please note that you cannot set the type
property of relationships this way.
Parameters
Name | Description |
---|---|
type | The database type on which the properties should be set. |
setUuid
Parameters
Name | Value |
---|---|
type | limit the execution to a certain node type |
relType | limit to a certain relationship type |
allNodes | process all nodes |
allRels | process all relationships |
snapshot
Creates a schema snapshot and writes it to a local file on the server the Structr instance is running on. This maintenance command is used internally in the Schema section to manage schema snapshots.
Parameters
Name | Value | ||||
---|---|---|---|---|---|
mode | the snapshot mode (export | restore | add | delete | purge) |
name | the name of the input / output file | ||||
types | an optional list of types to export |
sync
Exports or imports partial database content, based on a query, into a ZIP file. You can specify a Cypher query for the desired export set.
Parameters
Name | Value |
---|---|
mode | operation mode (import/export) |
file | data file when importing or exporting (e.g. my-data.zip ) |
query | Cypher query that returns the desired export subgraph |
Changelog
The changelog can be activated by setting the structr.conf switch application.changelog.enabled
to true
. A Structr instance started with changelog enabled keeps a record of every modification of an entity. There are different kinds of actions (called verb
) for which a record is kept.
The changelog itself can be retrieved using the builtin changelog()
function. The changelog is written to disk at changelog.path
with the subfolders n
for nodes and r
for relationships. If user-centric changelog (application.changelog.user_centric.enabled
) is enabled the changelog is written to the subfolder u
. In older versions of structr the changelog was written to the structrChangeLog
property of a node. The following tables show the possible keys/values of modification records. Modification records are stored as a JSON string. (see the example below)
Verb | Event type | Keys in changeset |
---|---|---|
create | Creation of an object | verb ,time ,userId ,userName ,target |
delete | Deletion of an object | verb ,time ,userId ,userName ,target |
link | A relationship to/from another object has been created | verb ,time ,userId ,userName ,target , rel |
unlink | A relationship to/from another object has been removed | verb ,time ,userId ,userName ,target , rel |
change | One of the properties of the object has been changed | verb ,time ,userId ,userName ,key , prev , val |
Key | Content |
---|---|
verb | type of changelog event (One of create , change , delete , link , unlink ) |
time | Timestamp of the change (ms since epoch) |
userId | id of the user, 00000000000000000000000000000000 for SuperUser, null for anonymous |
userName | name of the user who triggered the modification |
target | id of the target node of the new or deleted relationship |
relId | id created/deleted relationship |
rel | relationship type of the created/deleted relationship |
relDir | relationship direction of the created/deleted relationship (in/out) |
key | Property key of the modified property |
prev | Previous value of the modified property |
val | New value of the modified property |
Example
{"time":1455195862431,"userId":"f02e59a47d[...]","userName":"admin","verb":"change","key":"name","prev":null,"val":"My new name"}
{"time":1455195903852,"userId":"f02e59a47d[...]","userName":"admin","verb":"change","key":"name","prev":"My new name","val":"New Name"}
{"time":1455196049579,"userId":"f02e59a47d[...]","userName":"admin","verb":"link","rel":"has","relId":"97d26b5778[...]026eb615e","relDir":"out","target":"4e32a9f6eb[...]bd7a6a"}
{"time":1455195961348,"userId":"f02e59a47d[...]","userName":"admin","verb":"unlink","rel":"has","relId":"97d26b5778[...]026eb615e","relDir":"out","target":"4e32a9f6eb[...]bd7a6a"}
{"time":1455196115875,"userId":"00000000000000000000000000000000","userName":"superadmin","verb":"unlink","rel":"OWNS","relId":"b29e98329fb[...]864aa038","relDir":"out","target":"f02e59a47d[...]"}
Note
The file-based changelog can consist of many files. You should make sure the file system can handle the amount of files. Depending on the file system it can happen that no more files can be written even if enough space is available.
Backup Options
Cold backup
To backup a Structr installation, do the following:
- stop the Structr instance
- copy
db/
andfiles/
directories to a safe place - copy structr.conf to a safe place
- start instance
Structr Services Layer
The Structr Services Layer is a part of the Structr architecture. It manages the configured services that run in an instance.
SSHService
The SSHService allows admin users to connect to the structr instance via the SSH protocol. The SSH port can be configured via the application.ssh.port
key in structr.conf.
Authentication can be done via public key or via password (unless explicitly forbidden via application.ssh.forcepublickey
).
After successful authentication the SSH connection allows exactly the same functionality as the Admin Console but via SSH.
Public Key Authentication
The users public key can be stored in the property publicKey
on the user node. The transmitted key is compared to the one stored on the user node. If successful, login is granted.
Password Authentication
Authentication works just like regular password-based authentication. The given password is salted, hashed and compared to the stored password hash.
Example
$ ssh -p 8022 admin@localhost
Password authentication
Password: *****
Welcome to the Structr 3.6-SNAPSHOT 27066 202006301221 JavaScript console. Use <Shift>+<Tab> to switch modes.
admin@Structr/>
FtpService
The FtpService allows users to connect to the structr instance via the FTP protocol. The FTP port can be configured via the application.ftp.port
key in structr.conf.
Authentication is only possible via password. After successful authentication the FTP connection lists all the files a user has read rights to starting from the root directory.
Files which the user is allowed to read, but which reside in a directory which the user is not allowed to “see”, the file will not show up in the FTP listing.
Regular users will also not be able to see file owner if they do not have read rights on those nodes.
$ lftp -p 8021 -u user1 localhost
Password:
lftp user1@localhost:~> DIR
drwx------ 1 0 Jun 30 15:22 testFolder
-rw------- 1 user1 347 Jun 30 09:24 test1.txt
-rw------- 1 25 Jun 30 15:41 test2.txt
-rw------- 1 5 Jun 30 09:24 test3.txt
-rw------- 1 user1 5 Jun 30 09:24 test4.txt
Admin users always see every file/folder available as well as their owner.
$ lftp -p 8021 -u admin localhost
Password: *****
lftp admin@localhost:~> DIR
drwx------ 1 admin 0 Jun 30 15:22 testFolder
-rw------- 1 user1 347 Jun 30 09:24 test1.txt
-rw------- 1 admin 25 Jun 30 09:24 test2.txt
-rw------- 1 user2 5 Jun 30 09:24 test3.txt
-rw------- 1 user1 5 Jun 30 09:24 test4.txt
lftp admin@localhost:/>
Events
Structr provides events which can be implemented to influence behavior. The four most important ones are grouped as so-called Lifecycle Methods. There are also 2 global events and one event specific to nodes of type File (and types inheriting from File).
These events are only triggered if the trigger action is done via structr. For example, setting the name of a node via Cypher does not trigger the onSave
method of that node type to run because the structr layer is not used.
Lifecycle Methods
Lifecycle events are events that occur during the lifecycle of an object in the database. A lifecycle method is an optional method on any type of the data model that will be executed when a lifecycle event occurs. There are several that can be bound to different events that happen in a Structr application. When a data object is created, modified or deleted in the database, the corresponding lifecycle method is called (if it exists on that particular type).
onCreate
- Is called when an object is created (but the transaction is not yet committed)
- Multiple onCreate methods can exist per type - every method prefixed with onCreate is called when an object is created (e.g.
onCreate01
,onCreate02
) - Schema constraint checks (uniqueness constraint, notNull constraint) are performed after onCreate has run
- can be rolled back
- if an error occurs
- if an error is raised via
error()
function - if a constraint is violated
- All regular keywords can be used - no special keywords are introduced for this method
Example:
{
let self = Structr.get('this');
if (self.name === 'foo') {
// don't allow Nodes named "foo"
Structr.error('Type', 'name', 'name can not be "foo"!');
} else {
Structr.log('Node with name ' + self.name + ' has just been created.');
}
}
afterCreate
- Is called after an object has been created (onCreate has already been executed), all checks have passed successfully and the transaction has been committed
- Multiple afterCreate methods can exist per type - every method prefixed with afterCreate is called after an object has been created successfully (e.g.
afterCreate01
,afterCreate02
) - can not roll back the transaction
- is useful for actions that should only happen if the node is sure to have been created. For example sending a mail: Calling
send_html_mail()
in onCreate would send the email even if the transaction would be rolled back due to an error. - All regular keywords can be used - no special keywords are introduced for this method
Example:
{
let self = Structr.get('this');
let subject = 'Node was created!';
let text = 'Node named ' + self.name + ' was created';
Structr.send_html_mail('info@structr.com', 'Structr', 'user@domain.com', 'Test User', subject, text, text);
}
onSave
- Is called when an object is modified. (Either modification of local attributes of relationships. When a relationship is created, the onSave is triggered for both nodes)
- creation of an object is not a modification, thus the onSave is not run when an object is created (even if the object is ‘modified’ in onCreate)
- Multiple onSave methods can exist per type - every method prefixed with onSave is called when an object is created (e.g.
onSave01
,onSave02
) - Schema constraint checks (uniqueness constraint, notNull constraint) are performed after onSave has run
- can be rolled back
- if an error occurs
- if an error is raised via
error()
function - if a constraint is violated
- All regular keywords can be used - one special keyword is introduced for this method
The state of the object in the scripting environment represents the state of the object after the modification. To get information about the previous state the keyword modifications
can be used.
Keyword | Value |
---|---|
modifications | Object containing the modifications made to the object in the current operation |
Example:
{
let self = $.get('this');
let modifications = $.retrieve('modifications');
if (modifications.after.name === 'foo') {
// don't allow Nodes named "foo"
Structr.error('Type', 'name', 'name can not be "foo"!');
}
}
Modifications
The modifications
object is a read-only representation of all changes to an object in the current transaction. It consists of 4 elements:
Key | Value |
---|---|
after | The changed local attributes of the node after the modification |
before | The previous values of all changed local attributes |
added | Remote attributes (relationships) that were added |
removed | Remote attributes (relationships) that were removed |
Example 1: The visibility flags visibleToAuthenticatedUsers
and visibleToPublicUsers
were both set from false
to true
.
{
"before":{"visibleToAuthenticatedUsers":false,"visibleToPublicUsers":false},
"after":{"visibleToAuthenticatedUsers":true,"visibleToPublicUsers":true},
"added":{},
"removed":{}
}
Example 2: The owner
for a node was removed. The value of the owner
property in the modifications object is a single string because the relationship is defined as a “to one” relationship (a node can only have one owner).
{
"before":{"owner":null},
"after":{"owner":null},
"added":{},
"removed":{"owner":"5ba37699ca8f4a8b92ded77c93629f0e"}
}
Example 3: The user with id 5ba37699ca8f4a8b92ded77c93629f0e
was removed from the maintains
remote attribute. The value of maintains
is an array of strings because the relationship is defined as a “to many” relationship.
{
"before":{"maintains":null},
"after":{"maintains":[]},
"added":{},
"removed":{"maintains":["5ba37699ca8f4a8b92ded77c93629f0e"]}
}
onDelete
- is run after an object has been deleted (but the transaction has not yet been committed)
- Using the
this
keyword results in an error as the object has already been deleted - can be rolled back
- if an error occurs
- if an error is raised via scripting
See <a data-id="651d6bf220e3422488fa61a34109745c" class="mention">error()</a> function
- All regular keywords can be used - no special keywords are introduced for this method
Global Events
Currently there are two global events which can only be implemented via global schema methods.
onStructrLogin
- called when a User logs in
- only this exact function name is called and only as a global schema method
- Is run in a super-user context (not in the context of the user logging in!)
- All regular keywords can be used - one special keyword is introduced for this method
Special keyword | Function |
---|---|
user | The user logging in |
Example:
{
let user = Structr.retrieve('user');
Structr.log(user.name + ' just logged in');
}
onStructrLogout
- called when a User logs out
- only this exact function name is called and only as a global schema method
- Is run in a super-user context (not in the context of the user logging out!)
- All regular keywords can be used - one special keyword is introduced for this method
Special keyword | Function |
---|---|
user | The user logging out |
{
let user = Structr.retrieve('user');
Structr.log(user.name + ' just logged out');
}
onAcmeChallenge
This method is called when an ACME challenge of type dns
is triggered, typically by using the maintenance method letsencrypt
like this:
$.maintenance('letsencrypt', { server: 'production', challenge: 'dns', wait: '60', reload: true });
The primary use case of this method is creating a DNS TXT record via an external API call to a DNS provider.
The following parameters are passed to this method and can be retrieved with $.retrieve(key)
:
Parameter name | Description | Examples |
---|---|---|
type | The type of the ACME authorisation challenge | dns |
domain | The domain the ACME challenge is created for | subdomain.example.com |
record | The name of the DNS record including prefix _acme-challenge. and suffix . | _acme-challenge.subdomain.example.com. |
digest | The token string that is probed by the ACME server to validate the challenge | X6Rea0DdZ-5XGotp1geAxfsdDR0x1T9d_kAseA4YMCA |
Type-specific events
Currently there are only tow type-specific events. They are only applicable to nodes of type File
and types inheriting from it. They can only be implemented by creating an onDownload
and onUpload
method on the type.
onDownload
- Is called after a file has been requested for download
- can not prevent the download of a file
- All regular keywords can be used - no special keywords are introduced for this method
{
$.log($.this.name + ' was just downloaded');
}
onUpload
Available in structr v4.2+
- Is called after a file has been uploaded
- can not prevent the upload of a file
- All regular keywords can be used - no special keywords are introduced for this method
{
$.log($.this.name + ' was just uploaded');
}
onOAuthLogin
Available in structr v4.2+
- Is called after a user successfully logs into the system via a configured OAuth provider
- can not prevent the login of a user
- Is called with the two parameters
provider
containing the OAuth provider name that was used for login anduserinfo
containing the information pulled from the userinfo endpoint of the OAuth provider.
{
$.log('User ' + $.this.name + ' just logged in via ' + $.methodParameters.provider);
$.log('User information: ', JSON.stringify($.methodParameters.userinfo, null, 2))
}
Java-Logging
Structr is written in Java, and we use the Log4J logging framework (the API) and the Logback implementation of that API. The Logback configuration lives in the classpath, in a file called logback.xml
. Structr contains a logback.xml file that specifies settings for the console appender and the file appender which writes to /var/log/structr.conf
.
logback-include.xml
The logback configuration file supplied with Structr contains a reference to an optional `logback-include.xml’ in which additional settings can be made. In this file you can change the log level for individual Java packages to gain a more detailed insight into internal processes.
XMPP
Structr provides a scriptable XMPP Client Implementation based on the XMPPClient entity.
XMPPClient is a base type that provides a Scriptable XMPP Client Implementation.
XMPPClient extends AbstractNode and provides the following additional properties.
Name | Type | Description |
---|---|---|
xmppUsername | String | The username of this XMPPClient |
xmppPassword | String | The password for this XMPPClient |
xmppService | String | The service name to connect to |
xmppHost | String | The host name to connect to |
xmppPort | Integer | The port to connect to |
isEnabled | Boolean | A boolean value that indicates whether this XMPPClient is enabled (should connect to a server and receive messages) |
isConnected | Boolean | A boolean value that indicates whether this XMPPClient is connected to a server |
Example:
let xmppClient = $.this;
xmppClient.doJoinChat('chatRoom@conference.server.org', 'nickname', 'password');