admin's blog

sifbuilder is an application framework and an open source program supporting multiple wcm cores to automatically generate and deploy modular web sites defined by declarative xml files. Using sifbuilder is risky and can result is substantial data loss. Never use sifbuilder without a backup in place. Once said that, the sifbuilder site is regularly updated and maintained using sifbuilder.

posted by admin at 2025-01-01 00:00:00 0 comments read more

A site is defined in XML and running a URL generates a drupal web site

> php sifcmd.php http://www.sifbuilder.com/downloads/sibs/sib_drupal5_sif_sif.xml

The url includes the identification (the place where it will be created) and definition (its omponents and form) of the target site. The site identification may be modified interactively. Download the file and set the site identifying properties to sif run its uri (php sifcmd.php file://path_to_file) in force mode (-f).

Create also postnuke, xoops and joomla sites

> php sifcmd.php http://www.sifbuilder.com/downloads/sibs/sib_PostNuke08_sif_sif.xml

> php sifcmd.php http://www.sifbuilder.com/downloads/sibs/sib_XOOPS20_sif_sif.xml

> php sifcmd.php http://www.sifbuilder.com/downloads/sibs/sib_joomla1_sif_sif.xml


posted by admin at 2007-09-30 00:00:00 0 comments read more

This theorem establishes that, independently of the underlying core, a web site may be computed by a function of its constituent components.. It is devided into three parts:

  • 1. The build process of an open source component-based web site may be structured as follows:
    • Transpose: obtaining the constituent components from the source into the building environment
    • Transfer: placing the components from the building environment into the target site code base
    • Configure: setting the identity of the target site
    • Install: initializing and activating core, module and theme components
    • Conform: configuring panels, blocks, and menus; setting site variables; defining permissinos; creating users and content
  • 2. A site may be defined as a directed series of actions on the constituent logic and presentation components that may be declared in a xml file. Actions may be:
    • those associated to the core and available to all packages: install package, update package, set variable, create block in panel, add row to menu. Internal functions common to all cores include: say if the core is installed, say if a module is installed and installed version
    • those exposed by installation packages associated to specific components -modules-
  • 3. Given a site, it is possible to automate the update the components from a reference components library.
  • 4. It is possible to clone a site as follows:
    • Backup the site: data base, file system and site identification
    • Restore: database and file system processing source and target site identification
    • Configure: setting the identity of the target site

This theorem is proved as follows:

Create a drupal site as identified in sbd501_at_localhost from the sib_drupal5_sif_sif.xml definition file

> php sifcmd.php b --sifpacksdir=../sifbuilder_local/packages --sifnetsite=sbd501_at_localhost --siburi=./sibs/sib_drupal5_sif_sif.xml -x

Create a drupal site with a version-defined component

> php sifcmd.php b --sifpacksdir=../sifbuilder_local/packages --sifnetsite=sbd501_at_localhost drupal5_mod_views_1_5 -x

Update the component in the site

> php sifcmd.php b --sifpacksdir=../sifbuilder_local/packages --sifnetsite=sbd501_at_localhost drupal5_mod_views --nohistory --doupdate

Create a postnuke site as identified in sbpn501_at_localhost from the sib_PostNuke08_sif_sif.xml definition file

> php sifcmd.php b --sifpacksdir=../sifbuilder_local/packages --sifnetsite=sbpn501_at_localhost --siburi=./sibs/sib_PostNuke08_sif_sif.xml --dev -x

Create a postnuke site with a version-defined component

> php sifcmd.php b --sifpacksdir=../sifbuilder_local/packages --sifnetsite=sbpn501_at_localhost PostNuke08_mod_EZComments_1_3 --dev -x

Update the component in the site

> php sifcmd.php b --sifpacksdir=../sifbuilder_local/packages --sifnetsite=sbpn501_at_localhost PostNuke08_mod_EZComments --dev -f --doupdate --doupdate

Create a xoops site as identified in sbx501_at_localhost from the sib_XOOPS20_sif_sif.xml definition file

> php sifcmd.php b --sifpacksdir=../sifbuilder_local/packages --sifnetsite=sbx501_at_localhost --siburi=./sibs/sib_XOOPS20_sif_sif.xml --dev -x

Create a xoops site with a version-defined component

> php sifcmd.php b --sifpacksdir=../sifbuilder_local/packages --sifnetsite=sbx501_at_localhost XOOPS20_mod_multimenu_1_74 --dev -x

Update the component in the site

> php sifcmd.php b --sifpacksdir=../sifbuilder_local/packages --sifnetsite=sbx501_at_localhost XOOPS20_mod_multimenu --dev --doupdate --doupdate

Create a joomla site as identified in sbj501_at_localhost from the sib_joomla1_sif_sif.xml definition file

> php sifcmd.php b --sifpacksdir=../sifbuilder_local/packages --sifnetsite=sbj501_at_localhost --siburi=./sibs/sib_joomla1_sif_sif.xml --dev -x

The Corolarium I to this theorem establishes that, in two years, all open source component-based content management systems will include this function.


posted by admin at 2007-09-29 00:00:00 0 comments read more

We attended the drupalcon conference at Barcelona and presented the concepts underlying sifbuilder. It was a great event -more than 400 participants-, incredible city and superb organization. We met very competent professionals that are doing great things with whom to keep in contact and collaborate in the future.

The presentation by Nick Blundell from illuminateict about Drool was very interesting and he is publishing the tool. We feel curious to know how it applies the skin command to themes and how it could find out and install the ecommerce module.

During our presentation we got interesting questions. A participant asked if the modules installed were available locally. We prefer to get them from the original source but it is possible to placethem locally and once locally available they do not need to be downloaded. The basic insert node function is provided by the site and it is available to all installation package refers. A module may implement the logic to export/import specific types of nodes and this be used by the installation packages.

A colleague asked if one to many relationships were taken into account in the automatic generation of a module. The module automatically generated based on its data model currently implements the basic hooks, control, view and model functions. It may evolve in the future to include related tables and linked forms. A participant asked if we had any recommendation for the development of a module. Code generation is very important in the measure that it reflects fundamental decisions of the architecture of the code to develop. In the case of drupal, it is important to understand the architecture of the core and then decide the best implementation of the modules. The modules generated at this time do not pretend to be a recommendation of style.

One question enquired if the site generation provided by sifbuilder supports multisites. Multisite sites may share tables in the database or folders the file system. In the first case, one site typically has a default table prefix and lists shared tables with other prefixes. Thus, there is not a relationship between a site and a prefix but between self-contained site components and specific table prefixes. These components are normally predefined. This implies a lost of abstraction in the site specification model and, in this way, it is not supported by sifbuilder.

When multisites share folders, a multisite site is identified by the source script virtual directory and the configuration file associated to that site with a predefined criteria, implemented in site_conf. To implement the sifbuilder logic, it is necessary to identified when a site is installed. However, the sifbulder script does not reveal the identity of the target site. To find out if a site is installed as a multisite site, it is necessary to recognize the database path and prefix in the chain of possible configuration files, from specific to default. This is now the default behaviour in sifbuilder for drupal.

Here follows a summary of our presentation.

  • Automatic generation of drupal sites, backup and cloning of drupal sites, and automatic creation of drupal modules
  • Automatic generation of drupal sites
  • What is the rational
    • Enable the declarative definition and automatic generation of modular web applications.
      • Relying on web application cores
      • Integrating open modules and themes
      • Abstracting collection, installation, update and configuration of layout and functional components
      • Supporting the implementation of functional components automating the generation of new modules based on the declarative definition of their data model
  • Definition of a site
    • A site is defined as a set of building actions (building chain) carried out by installation packages associated to the site components. This definition may be reflected in a XML file (build file or sib) which is a representation of the calls to the underlying sif installation framework.
      • if an action refers to a component, it is associated to the most recent release of that component.
      • if the component has dependencies, these may be added to the build chain.
      • if a building component is already present in the target site, an installation action may be changed to an upgrade.
  • Basic components
    • A set of installation packages (sips) that expose the behavior and complement the functionalities of individual web site components (core, modules, themes). Installation packages offer the interface to
      • Transpose. Download and explode the site component from the source if not locally available.
      • Transfer. Copy the package files from the packages repository to the site codebase.
      • Config. Configure the site. Done only by the core installation package.
      • Install or Upgrade the component. Installation is dependent on the type of component: core, moduel, theme.
      • Conform, including configuration of panels and blocks, setting of system variables and permission and creation of content.
    • A set of XML files (sibs) listing actions that represent web site building blocks or define operational web site configurations and profiles
    • Sample site creation:
      • > php sifcmd.php b --sifpacksdir=../sifbuilder_local/packages --sifnethost=localhost --sifnetsite=sbd501_at_localhost drupal5_core_drupal --dev -x -v
        > php sifcmd.php b --sifpacksdir=../sifbuilder_local/packages --sifnethost=localhost --sifnetsite=sbd502_at_localhost --siburi=./sibs/sib_drupal5_sif_sif.xml --dev -x -v
  • Backup, restore and clone sites
    • If the system is consistent, it cannot be complete, and the consistency of the axioms cannot be proved within the system. Therefore, a site cannot be restored from within itself.
  • Backup, restore, clone
    • The sifbuilder backup contains three files:
      • file.xml, with the configuration of the backup'ed site (including the reference to the pod)
      • file.sql, with the sql dump of the pod database
      • file.zip, with the tar file of the pod file system
    • The sifbuilder restore recreates the database and file system.
    • The sifbulder clone takes into account the configuration of the source and target sites to create the site with the new configuration, including database and site credentials, table prefix, and file system location. When specified, the installation package of the core configures the cloned site.
    • Sample backup and clone action:
      • >php sifcmd.php backup_pod --sifnetsite=sbd502_at_localhost
      • >php sifcmd.php clone_pod --sifnetbcksite=sbd502_at_localhost --sifnetsite=sbd501_at_localhost drupal5_core_drupal -x
    • Creation of a new module
  • Input
    • The XML representation of the relational model of the entities used by the module logic.
    • Smarty templates reflecting the module architecture.
    • Creation of a site with the module generator module installed:
      • >php sifcmd.php b --sifnetsite=sbd501_at_localhost drupal5_mod_sifmoduler -f -x
  • Output
    • for the module, it generates the .module and .install files. The .module file includes the module hooks and the action router based on the menu hook.
    • for each entity, it generates the .controlapi file, which contains the control and view functions including add, edit, delete, list and delete multiple records; and the .dao file, that includes the select, insert, update and delete interface to the entity model.
  • Process
    • Creation of modules is done by the sifmoduler and sifclasses drupal modules, where the sifclasses module exposes the sif and smarty classes and the sifmoduler module creates the module files from the module templates and the module definition.
    • The generative spirit of sifmoduler is transmitted to the generated modules and it is possible for the admin to create other drupal modules from any module created in that way.
  • Vision
    • The vision of sifbuider is that of an XML file containing the actions on core, modules and themes that define a web site. That XML file is referenced by a URI and running that URI builds the site.


posted by admin at 2007-09-28 00:00:00 0 comments read more

This release adds support for multisites in drupal and modifies the way in which site identification parameters is processed.

There are four ways to specify the site identification properties of the target site: as individual parameters in the command line, as classes defined in the sifnet_config file and named in the command line, as siteParams in the sib uri, as elements in the sifSiteParams.xml file.

Those parameteters may be set interactively or taken as passed when the script is run in force mode. This release sets the preference order to consider the uri site parameters before those in the sifSiteparams file . Thus, it is possible to create a site in any host just passing the uri to sifbuilder (php sifcmd.php http://url or php sifcmd.php file://path_to_file).

This release implements the default behaviour of drupal modules for drupal sites to supports the installation of drupal modules directly from the official drupal module library without the need to have the package or the installation package locally available. Thus, it is possible to include reference to new standard-behaved drupal modules in the installation beans (sibs) without having to update the sifbuilder or creating corresponding installation packages (sips). The sif_sif bean for drupal shows the installation of the project module in this way.


posted by admin at 2007-09-27 00:00:00 0 comments read more

sifbuilder 1.10 introduces the automatic generation of drupal modules based on the xml representation of the target module relational data model. The sifmoduler module includes the Smarty templates files that generate the target files using the information passed in a file containing the xml representation of the target module relational data model.

sifbuilder, through invokation of the drupal sifmoduler module functions, generates the .module and .install files. For each entity, it generates the .dao and .controlapi files. The module .module file includes the module hooks and the action router based on the menu hook. The entity .controlapi file contains the control and view functions including add, edit, delete, list and delete multiple records. The entity .dao file contains the select, insert, update and delete interface to the entity model.

The sifmoduler module carries out three basic actions: processes the xml module definition file, generates the sql ddl statements and creates the module files based on the Smarty templates. The libraries supporting these functions are exposed by the dependency sifclasses drupal module that is invoked by the sifmoduler, in line with an architecture based on a basic bootstrap providing discovery and loading mechanisms and functions offered by self contained modules aware of the installation and communication logic.

The sample resulting _sifgroups module may be created by

> php sifcmd.php b --sifpacksdir=../sifbuilder_local/packages --sifnetconfigfile=../sifbuilder_local/sifnet_config.php --sifnetsite=sbd_at_localhost drupal5_smod_sifmoduler

It is also created within the drupal sif_sif profile.

>php sifcmd.php b --sifpacksdir=../sifbuilder_local/packages
-sifnetconfigfile=../sifbuilder_local/sifnet_config.php --sifnetsite=sbd_at_localhost --sifnethost=localhost --siburi=./sibs/sib_drupal5_sif_sif.xml -x -f -v

It can be seen at the sif site.


Beyond the input data model, the generative spirit of sifmoduler is transmitted to the generated modules and it will be possible for the admin to create other drupal modules from any module created in that way.

posted by admin at 2007-09-10 00:00:00 0 comments read more

The sifbuilder site is generated with the sif profile for drupal, with sifnethost and sifnetsite correctly defined to point to the sifbuilder host and site,

> php sifcmd.php b --sifpacksdir=../sifbuilder_local/packages
-sifnetconfigfile=../sifbuilder_local/sifnet_config.php --sifnetsite=sbd_at_localhost --sifnethost=localhost --siburi=./sibs/sib_drupal5_sif_sif.xml -x -f -v

on the nethost. The definition of this site is set in the sib_drupal5_sif_sif.xml sib. That file specifies ... install the drupal core, install and apply the sif theme, activate core modules -search, blog-, get and install contributed modules -views, site-map-, configure blocks -Navigation, Primary links-, create blocks and block rows -sif, my links-, set permissions -access content, search content-, set config variables -site_name, site_footer-. That xml definition includes also the information to dynamically generate, install, and configure a drupal module -_sifgroups- based on its relational model.

Content is created applying a content sib, always with sifnethost and sifnetsite correctly defined to point to the sifbuilder host and site.

> php sifcmd.php b --sifpacksdir=../sifbuilder_local/packages
-sifnetconfigfile=../sifbuilder_local/sifnet_config.php --sifnetsite=sbd_at_localhost --sifnethost=localhost --siburi=./sibs/sib_drupal5_sif_sif.xml -f -v

In support to the creation of the sifbuilder site, sifbuilder 1.10 includes the drupal5_mod_sifbuilder_0_0_1 sip for drupal. That package generates the nodes for the sif installation packages (sips) that is accessible from the sif menu block.


posted by admin at 2007-09-09 00:00:00 0 comments read more

"now will I tell the purpose of the sifbuilder functions and the functions in their order": the automatic generation of sites from a XML site definition specifying constituting modules, blocks, themes and content, and based on various core CMS -the specification files are located in the sibs folder in the sifbuilder distribution-; association of modules and themes to specific site profiles; management of dependencies and conflicts between modules and site components; structure the actions constituting the creation of a site; access to the functions of the cores and modules from the command line; comparison of the version of installed packages against that of the most recent releases; update of modules in a site with those most recently released aiming to the automatic update of web sites; backup and restore of web sites; cloning of sites using site backup, restore and site configuration; generation of a module based on the XML representation of the relational schema as a means to automate the development of a Web application building of the functions offered by the CMS cores; installation of sites on remote hosts through SSH connections; creation or removal of multiple sites using use cases.

These examples assume the configuration of target hosts and sites in ../sifbuilder_local/sifnet_config.php. Examples of those definitions can be found here. Omitting -v and adding --expandsiteparams option, the command is shown in simulated mode. Without -f, specification of the target site can be done interactively through the command line.

Using sifbuilder is risky and can result is substantial data loss. Never use sifbuilder without a backup in place. Once said that, the sifbuilder site is regularly updated and maintained using sifbuilder.

Domain Action Instruction Result instance
program configure php sifcmd.php sip_configure_site
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnethost=devhost --sifnetsite=sbpn_at_localhost
PostNuke08_core_postnuke -f -v
Configure the distribution
create php sifcmd.php build_program
--sifnethost=devhost --sifnetconfigfile=../sifbuilder_local/sifnet_config.php -v
Create the sifbuilder program
in siftmp/tmpUpload/ from the development environment
tar php sifcmd.php
sifnet_create_program_zip --sifnethost=devhost -v
Create the sifbuilder
program tar
upload php sifcmd.php sifnet_upload_program
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php --sifnethost=localhost -v
Upload the program
tar to the nethost sifdir
explode php sifcmd.php sifnet_explode_package
--sifnethost=localhost --sifnetconfigfile=../sifbuilder_local/sifnet_config.php -v
Explode the program
tar in the nethost sifdir
local dir php sifcmd.php a ac001i
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php --sifnethost=localhost -v
Create the sifbuilder_local file tree
local config php sifcmd.php a ac023
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php --sifnethost=localhost -v
Upload the local sifnet config file
packages php sifcmd.php sifnet_upload_packages
--sifnethost=sfhost --sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifpacksdir=../sifbuilder_local/packages
Upload site packages if necessary
site php sifcmd.php b
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbd501_at_localhost --sifnethost=localhost
--siburi=./sibs/sib_drupal5_sif_sif.xml -x -f -v
Build the target site
release create php sifcmd.php build_release
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnethost=devhost -f -v
Create a new sif packages release
upload php sifcmd.php sifnet_upload_release
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnethost=devhost -f -v
Upload a sip release to the sif host
create old site php sifcmd.php b
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnethost=localhost
--sifnetsite=sbpn501_at_localhost --siburi=./sibs/sib_PostNuke07_sif_old.xml
--dev -x -f -v
Create a site with old site packages
check new versions php sifcmd.php rlist
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnethost=localhost
--sifnetsite=sbpn501_at_localhost --dev -f -v
Update site with most recent packages
update php sifcmd.php b
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnethost=localhost --sifnetsite=sbpn501_at_localhost
--siburi=./sibs/sib_PostNuke07_sif_sif.xml
--dev --doupdate -f -v
Update site with most recent packages
drupal create php sifcmd.php b
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbd501_at_localhost
--siburi=./sibs/sib_drupal47_sif_sif.xml -f -x
Creates in force mode the
http://localhost/sifbuilder/d501 site
Pre empties the d501 database and pre removes the
d:/webroot/sifbuilder/d501/ folder
backup php sifcmd.php backup_pod
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbd501_at_localhost -f
backup the d501 site into the
autodefined bck_pod_200707220010_sifbuilder_d501.zip tar at
./siftmp/tmpUpload/

restore php sifcmd.php restore_pod
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbd502_at_localhost drupal47_core_drupal -x -f
Restores in force mode the
database and file system from the latest backup tar with the backuped
siteParms :into the  sbd502_at_localhost pre-cleaned site
Configures the site with the drupal47_core_drupal core with the new
siteParmas
clone php sifcmd.php clone_pod
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetbcksite=sbd501_at_localhost --sifnetsite=sbd502_at_localhost
drupal47_core_drupal -f -x
Backup the sbd501_at_localhost
site and restore it into sbd502_at_localhost with post configuration by
the core sip
clean php sifcmd.php clean_pod
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbd501_at_localhost -f
Clean sbd501_at_localhost site
and test sbd502_at_localhost at  http://localhost/sifbuilder/d502
restore with uri php sifcmd.php b
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbd501_at_localhost
--siburi=./sibs/sib_drupal47_sif_sif.xml -f -x
Build the site
php sifcmd.php backup_pod
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbd501_at_localhost
--siburi=bck_pod__sbd501_at_localhost.zip -f
Backup the site into the
bck_pod__sbd501_at_localhost.zip tar
php sifcmd.php restore_pod
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbd502_at_localhost
--siburi=bck_pod__sbd501_at_localhost.zip drupal47_core_drupal -x -f
Restore with pre-cleaning (ie.
create from new) the sbd502_at_localhost from the
bck_pod__sbd501_at_localhost.zip backup with post configuration by the
core
php sifcmd.php clean_pod
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbd501_at_localhost -f
Clean sbd501_at_localhost site
and test sbd502_at_localhost at  http://localhost/sifbuilder/d502
postnuke create php sifcmd.php b
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbpn501_at_localhost
--siburi=./sibs/sib_PostNuke08_sif_sif.xml --dev -x -f
Creates in force mode the
http://localhost/sifbuilder/pn501 site
Pre empties the pn501 database and pre removes the
d:/webroot/sifbuilder/pn501/ folder
backup php sifcmd.php backup_pod
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbpn501_at_localhost -f
backup the pn501 site into the
autodefined bck_pod_200707220010_sifbuilder_pn501.zip tar at
./siftmp/tmpUpload/
restore php sifcmd.php restore_pod
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbpn502_at_localhost PostNuke08_core_postnuke --dev -f -x
Restores in force mode the
database and file system from the latest backup tar with the backuped
siteParms :into the  sbpn502_at_localhost pre-cleaned site
Configures the site with the PostNuke08_core_postnukecore with the new
siteParmas
clone php sifcmd.php clone_pod
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetbcksite=sbpn501_at_localhost --sifnetsite=sbpn502_at_localhost
PostNuke08_core_postnuke --dev -x -v -f
Backup the sbpn501_at_localhost
site and restore it into sbpn502_at_localhost with post configuration
by the core sip
clean php sifcmd.php clean_pod
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbpn501_at_localhost -f
Clean sbpn501_at_localhost site
and test sbpn502_at_localhost at  http://localhost/sifbuilder/pn502
install this news php sifcmd.php b
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnethost=devhost
--sifnetsite=sbpn501_at_localhost
--siburi=./sibs/sib_PostNuke08_sif_delta.xml --dev -f -v
publish this news in a clean
site with all the required modules
create module php sifcmd.php b
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnethost=devhost
--sifnetsite=sbpn501_at_localhost
--siburi=./sibs/sib_PostNuke07_modgen_xlite.xml --dev -f -v
Create a module based on the
relational data model specified in the sib file
xoops create php sifcmd.php b
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbx501_at_localhost
--siburi=./sibs/sib_XOOPS20_sif_sif.xml --dev -x -f
Creates in force mode the
http://localhost/sifbuilder/x501 site
Pre empties the x501 database and pre removes the
d:/webroot/sifbuilder/x501/ folder
backup php sifcmd.php backup_pod
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbx501_at_localhost -f
backup the x501 site into the
autodefined bck_pod_200707220010_sifbuilder_x501.zip tar at
./siftmp/tmpUpload/
restore php sifcmd.php restore_pod
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbx502_at_localhost XOOPS20_core_xoops --dev -f -x
Restores in force mode the
database and file system from the latest backup tar with the backuped
siteParms :into the  sbx502_at_localhost pre-cleaned site
Configures the site with the XOOPS20_core_xoops core with the new
siteParmas
clone php sifcmd.php clone_pod
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetbcksite=sbx501_at_localhost --sifnetsite=sbx502_at_localhost
XOOPS20_core_xoops --dev -x -v -f
Backup the sbx501_at_localhost
site and restore it into sbx502_at_localhost with post configuration by
the core sip
clean php sifcmd.php clean_pod
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbx501_at_localhost -f
Clean sbx501_at_localhost site
and test sbx502_at_localhost at  http://localhost/sifbuilder/x502
joomla create php sifcmd.php b
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbj501_at_localhost
--siburi=./sibs/sib_joomla1_sif_sif.xml --dev -x -f
Creates in force mode the
http://localhost/sifbuilder/j501 site
Pre empties the j501 database and pre removes the
d:/webroot/sifbuilder/j501/ folder
backup php sifcmd.php backup_pod
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbj501_at_localhost -f
backup the j501 site into the
autodefined bck_pod_200707220010_sifbuilder_j501.zip tar at
./siftmp/tmpUpload/
restore php sifcmd.php restore_pod
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbj502_at_localhost joomla1_core_joomla --dev -f -x
Restores in force mode the
database and file system from the latest backup tar with the backuped
siteParms :into the  sbj502_at_localhost pre-cleaned site
Configures the site with the joomla1_core_joomla core with the new
siteParmas
clone php sifcmd.php clone_pod
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetbcksite=sbj501_at_localhost --sifnetsite=sbj502_at_localhost
joomla1_core_joomla --dev -x -v -f
Backup the sbj501_at_localhost
site and restore it into sbj502_at_localhost with post configuration by
the core sip
clean php sifcmd.php clean_pod
--sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbj501_at_localhost -f
Clean sbj501_at_localhost site
and test sbj502_at_localhost at  http://localhost/sifbuilder/j502


posted by admin at 2007-08-18 00:00:00 0 comments read more

When in 2004 sifbuilder moved from an oscommerce contribution to a multi core site generator, it lost Carine's kind comments and had to look for a host server. At that time, sourceforge supported the distribution of os programs and was introducing the web, shell and database services for hosted projects. Access to shell services was available through SSH connections authenticated with SSH2 keys. sifbuilder supported the automatic generation of web sites on the localhost and those requirements seemed reasonable to try to automate the installation of sites on remote hosts, such as those of sourceforge. This was done introducing the sifnet concept based on the sifnethost, sifnetrelation and sifnetsite entities.

A sifnetrelation establishes a relationhip between a lochost -usually instantiated by the local devhost- and a nethost -usually a remote host. This note will take as motiv the installation of a site on a nethost instantiated to sourceforge defined by the sfhost class in the sifnetconfigfile file-. The sifnetsite describes the site to be installed on the nethost. The relation is reflexive and transitive but, unfortunately, not commutative. The non equivalence may be read as 'having the site hosted on Linux matters'.

Currently, a sifnetrelation is based on a SSH connection -in the case of sourceforge, the connection is controlled by a SS2 pair of keys-. The origin of the relation -identified by the lochost- is a winbox hosting the sifbuilder development package; the destination point -identified by the nethost- is a Linux host offering the following services: a shell to the sifdir where the sifbuilder package will be transferred to, a database hosting the remote site database, and a webdir hosting the remote site file system.

On the origin winbox, we have available a SSH client -ex. putty among various ssh clients - and a tar/untar tool -in our case 7-zip - and we have created the private and public ppk keys following the sf instructions.

The information on the lochost, the nethost, the sifnetsite and the type of sifnet relation (SS2 keys or login/password) is passed to sifbuilder in the sifnet_config.php file. A model of this file is included in the sifbuilder root folder. Normally, the customized file is stored in a parallel file tree as defined with the --sifnetconfigfile option. The following classes show the information assigned there.

The sfhost describes the target host at sourceforge. It extends a sifnet_linux_host and this a sifnet_host.

class sfhost extends sifnet_linux_host
{
var $hostParams = array(
'hosttype'=>'linux'
,'access'=>'ssh2key'
,'userdir'=>'/home/users/m/my/myuser'
,'sifdir'=>'/home/users/m/my/myuser/myproject/'
,'webdir'=>'/home/groups/m/my/myproject/htdocs/'
,'host_domain'=>'myproject.sf.net'
,'user_login'=>'my_user_login'
,'user_pwd'=>'my_user_pwd'
,'user_key'=>'D:\\sf_private_key.ppk'
,'php_bin'=>'php '
,'rexec_bin'=>'ssh '
,'rcp_bin'=>'scp '
);
}

The devhost describes the local host, including the syntax of the local php, tar and remote exec commands.

class devhost extends sifnet_windows_host
{
var $hostParams = array(
'hosttype'=>'windows'
,'access'=>'local'
,'userdir'=>'.\\'
,'sifdir'=>'.\\'
,'webdir'=>'D:\\webroot\\htdocs\\'

,'downloadsdir'=>'D:\\webroot\\htdocs\\downloads\\'

,'packagesdir'=>'D:\\webroot\\htdocs\\downloads\\packages\\'
,'host_domain'=>'localhost'
,'user_login'=>''
,'user_pwd'=>''
,'user_key'=>''
,'php_bin'=>'php '
,'zip_bin'=>'"C:\\Program Files\\7-Zip\\7z.exe "
a -tzip '
,'unzip_bin'=>'"C:\\Program Files\\7-Zip\\7z.exe"
x '
,'rexec_bin'=>'D:\\BIN\\plink.exe '
,'rcp_bin'=>'D:\\BIN\\pscp.exe '
);
}

The sfhost describes the remote host, including the syntax of the remote php, tar and remote exec commands, and the location of the SSH2 keys to be passed to the SSH client of the local host.

class sfhost extends sifnet_linux_host
{
var $hostParams = array(
nbsp; 'hosttype'=>'linux'
,'access'=>'ssh2key'
,'userdir'=>'/home/users/m/my/myuser'
,'sifdir'=>'/home/users/m/my/myuser/myproject/'
,'webdir'=>'/home/groups/m/my/myproject/htdocs/'
,'host_domain'=>'myproject.sf.net'
,'user_login'=>'my_user_login'
,'user_pwd'=>'my_user_pwd'
,'user_key'=>'D:\\sf_private_key.ppk'
,'php_bin'=>'php '
,'rexec_bin'=>'ssh '
,'rcp_bin'=>'scp '
);
}

The remote site describes the site to be created on the remote host. This instance describes the structure of a sourceforge site but mirrors the structure and semantics of any site created or updated with sifbuilder.

class sbd501_at_sfhost extends sifnet_site
{
var $siteParams = array(
'site_host'=>'myproject.sourceforge.net'
,'site_name'=>'mysite'
,'site_pod'=>'d501'
,'db_server'=>'mysql4-m'
,'db_database'=>'db_myproject'
,'db_prefix'=>'d501_'
,'db_prefix_sep'=>''
,'db_dbtype'=>'mysql'
,'db_tabletype'=>'myisam'
,'db_username'=>'db_admin'
,'db_password'=>'db_admin_pwd'
,'site_admin_username'=>'my_user_login'
,'site_admin_password'=>'my_user_pwd'

,'dir_fs_document_root'=>'/home/groups/m/my/myproject/htdocs/'
,'sys_temp'=>''
,'encodedpwd'=>1
,'enable_ssl'=>0
);
}

To create the site on the remote sourceforge host server for a sourceforge project, we follow the following steps. Note that, when running in sifnet mode, the -v option launches the sifnet command in real mode. Without that option, the translated command will appear in simulation mode. Before running any of those commands, pick up any of the disclaimers in here and apply them to the rest of the note. Age quod agis.

Test the connection to the remote host with the ac001h admin case, as described in sifnet_usecases.php:

>php sifcmd.php a ac001h
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnethost=sbhost -v

This command will list the entries in the home dir in the remote host. Since the local sifbuilder distribution may have changed because of the development process or because new installation packages (sip) or installation profiles (sib) have been added, we create a new distribution. We create it with

>php sifcmd.php build_program
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php

and its tar file.

>php sifcmd.php sifnet_create_program_zip --sifnethost=devhost -v

Up to here the steps followed to create each new sifbuilder release package to be ftp uploaded to the sourceforge incomming area. To create the remote site, we upload the tar file, located in the ./siftmp/tmpUplaod folder from the sifbuilder distribution root, to the remote host.

>php sifcmd.php sifnet_upload_program
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnethost=sfhost -v

and we explode the tar there.

>php sifcmd.php sifnet_explode_package --sifnethost=sfhost
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php -v

As it appears in that command and most of the examples of the site, the local sifnet config file is located in a parallel file tree. The same happens with the site packages that constitute the generated site. This is the time to create it. It can be done connecting to the site or with

>php sifcmd.php a ac001i
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnethost=sfhost -v

Since we want to use a single and locally edited customized config file, we upload it to the remote sifbuilder_local/ folder:

> php sifcmd.php a ac023
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnethost=sfhost -v

sifbuilder takes care to download the site packages required to build the site. However, sourceforge does not allow establishing an outgoing httpd connection. It is therefore necessary to upload them to the sifbuilder_local folder. First we copy them to siftmp/tmpUpload/packages folder -since we plan to create a drupal site using the sib_drupal5_sif_sif.xml sib, we copy the site packages that correspond to the drupal5_core_drupal, drupal5_mod_views and drupal5_mod_site_map installation packages (sips) - and run

> php sifcmd.php sifnet_upload_packages --sifnethost=sfhost
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifpacksdir=../sifbuilder_local/packages

Then, the site may be created.

> php sifcmd.php b --sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbd501_at_sfhost --sifnethost=sfhost
--siburi=./sibs/sib_drupal5_sif_sif.xml -x -f -v

Running a sif command with -x means erasing the target site file system and dropping all the tables in the target database. -f means doing that without the predefine settings and without the possibility to fix them. -v means confirming those options in a remote site. Collect all the disclaimers in the site above before doing this.

The difference between that command and the usual sif commands used to generate a local site resides on setting the sifnethost option. By setting the sifnethost option the command is translated into a remote instruction, in this case the command that creates the remote target site.


If the -v option is ommitted, the remote command is shown but not run. It shows

> D:\BIN\plink.exe -ssh -2 -l my_user_login -i
D:\LOCAL\sf_private_key.ppk myproject.sourceforge.net
" php /home/users/m/my/myuser/myproject/sifcmd.php sif_sip_build_site
--dev
--expandsiteparams --noupd --emptypod --force --complete --dbgtype=both
--siburi=./sibs/sib_drupal5_sif_sif.xml
--site_host=myproject.sourceforge.net --site_name=mysite
--site_pod=d501 --db_server=mysql4-m --db_database=db_myproject
--db_prefix=d501_ --db_dbtype=mysql --db_tabletype=myisam
--db_username=db_admin --db_password=db_admin_pwd
--site_admin_username=my_user_login --site_admin_password=my_user_pwd
--dir_fs_document_root=/home/groups/m/my/myproject/htdocs/
--encodedpwd=1 --sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifpacksdir=../sifbuilder_local/packages
--siflochost=devhost
drupal5_core_drupal drupal5_theme_sif
drupal5_mod_views
drupal5_mod_site_map drupal5_script_locals "

The result of these steps was a sourceforge site available here, until it was removed in the natural course of things.

The sifbuilder site was created and is regularly maintained following this process. if not automatic, the process is systematic. And every systematic process requires a table of definitions.

domain terms definition
hostparams usedir root folder of the use work environment. Usually, the sifbuilder tar will be downloaded to this location.
sifdir folder containing the working sifbuilder distribution. Usually, the result of exploding the sifbuilder tar to this location.
webdir folder containing the root to the web environment.
access type of authentication, currently ssh2key or ssh2pwd.
siteParams --site_name folder under the webdir that will contain the target sites.
--site_pod folder under that the site_name folder that will contain the specific target site. Plays, in a certain way, for the file system, the role of the prefix in the target database.
runParams --sifpacksdir indicates where the site packages (core, modules and themes) are downloaded. By storing them outside the sifbuilder folder, updating new versions of sifbuilder will not remove them.
--sifnetconfigfile points to the file hosting the
classes
describing the target sites referenced by sifnetsite. The sample sifnet_config.php is included in the sifbuilder root folder. It should always be adapted to the local configuration before starting using sifbuilder.
--sifnetsite refers to the class in the sifnetconfigfile with properties the site configuration options typically set during the installation of a new content management site. They coincide with the elements in the sifSiteParams.xml file in the sifbuilder root. Having these options on file permits to refer to each site configuration by name. Wrong setting of those
options may erase a whole system. No need to look for a demonstration.
--sifnethost refers to the (usually remote) host where the sifcommands will be run. By setting this option, the normal sif commands are translated into remote execution commands using the information in the associated class defined in the sifnetconfig file.
-x may be added to the command to remove the site root and empty the site database before starting the installation. Do that only if you have the possibility of and know how to restore your working environment.
-f may be used to force the execution of the command in non-interactive mode. It can be used and will be used if you are a programmer when you know to be in control of what you are doing.
-v sifnet commands are run on the nethost by default in simulation mode. When the -v option is set, they run in real mode.
posted by admin at 2007-08-15 00:00:00 0 comments read more

Let's address this issue once more with the same principles but with different manners. First we will create a local site pod. Then we will backup it to restore it back with precleaning. Finally, we will clone a new site from an existing one. The sample site will be based on a drupal core. It could be as well a joomla, postnuke or xoops core.

To create the site locally, we suppose that it is defined in sifnet_config.php pointed by sifnetconfigfile. The fundamental importance of defining the target site properly, particularly when the sif commands are run with the -f, -v and -x options, is indicated here. If the implications of wrongly defining the target site are by now not yet clear, do loop here.

The running environment, the web server and the database server are in place. The latest version of sifbuilder has been downloaded to a local folder.


Site creation

The following creates the site as defined in sib_drupal47_sif_sif.xml (try to understand why -f -x options are useful and even then do not use them).

> php sifcmd.php sip_build_site --sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbd501_at_localhost
--siburi=./sibs/sib_drupal47_sif_sif.xml  --dev  -x -f

The sip_build_site is the sif command for the installation of modules or the creation of a complete site. sip_build_site is shorcut by 'b'.

--sifpacksdir indicates where the site packages (core, modules and themes) are downloaded. By storing them outside the sifbuilder folder, updating new versions of sifbuilder will not remove them.

--sifnetconfigfile points to the file hosting the classes describing the target sites referenced by sifnetsite. The sample sifnet_config.php is included in the sifbuilder root folder. It should always be adapted to the local configuration before starting using sifbuilder.

--sifnetsite refers to the class in the sifnetconfigfile with properties the site configuration options typically set during the installation of a new content management site. They coincide with the elements in the sifSiteParams.xml file in the sifbuilder root. Having these options on file permits to refer to each site configuration by name. Wrong setting
of those options may erase a whole system. No need to look for a demonstration.

The --dev option makes installation packages related to development cores or modules be considered during the site generation process

-x may be added to the command to remove the site root and empty the site database before starting the installation. Do that only if you have the possibility of and know how to restore your working environment.

-f may be used to force the execution of the command in non-interactive mode. It can be used and will be used if you are a programmer when you know to be in control of what you are doing.

Site backup

The site may then be backup by:

> php sifcmd.php backup_pod
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbd501_at_localhost -f

It is possible to specify the backup file with the --siburi option. if it is not indicated, the tar file is created and timestamped in the folder defined by backup_dir_name in the sifConfigParams, by default siftmp/tmpUpload in the sifbuilder directory. In this case, the backup file is bck_pod_200707120020_sifbuilder_drupal.zip and it contains three files:

  • podcf.xml, with the configuration of the backup'ed site (including the reference to the pod)

  • poddb.sql, with the sql dump of the pod database

  • podfs.zip, with the tar file of the pod file system

The creation of the sql dump is based on standard calls to the database, such as SHOW FIELDS FROM table in mysql, provided by the abstract metabase interface. Since the information received is not exhaustive, there will most probably be some difference between the correct statement and the dumped one. For example,

CREATE TABLE `pn501__admin_module` (
  `pn_amid` int(11) NOT NULL auto_increment,
  `pn_mid` int(11) NOT NULL default '0',
  `pn_cid` int(11) NOT NULL default '0',
  PRIMARY KEY  (`pn_amid`),
  KEY `mid_cid` (`pn_mid`,`pn_cid`),
  KEY `mid` (`pn_mid`)
)

is generated as by sifbuilder:

CREATE TABLE IF NOT EXISTS `pn501__admin_module` (
  `pn_amid` int(11) NOT NULL  auto_increment ,
  `pn_mid` int(11) NOT NULL  default '0',
  `pn_cid` int(11) NOT NULL  default '0',
 PRIMARY KEY (`pn_amid`),
 KEY `pn_mid` (`pn_mid`)
);

Other differences may appear and therefore this technology may sometimes need be complemented with manual intervention or other more targeted tool. In fact, this -that- is the real reason why backup in sifbuilder is an art and not a science.

Site restore

When a site has been backup, the site may then be restored by :

> php sifcmd.php restore_pod
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbd501_at_localhost -f -x

The backup restores the sbd501_at_localhost site with the -x option having cleaned the pod before the restore.

Since the site identification is the same (sbd501_at_localhost) the restored site resides normally on the same server than the site of departure.

Summarizing,

  1. We have created the local sbd501_at_localhost site as defined in ../sifbuilder_local/sifnet_config.php and declared in sib_drupal47_sif_sif.xml .
  2. Then, we have created a timestamped backup zip including the site definition file, the file system tar and the site database sql dump.
  3. Then, we have removed the newly created site -by grace of the dangerous -x option in the restore statement-.
  4. Finally, the site has been restored because the same site definition has been specified in sifnetsite and because, since no backup uri has been specified, the most recently created backup file has been taken as source.

Clonage of a site

A substantially different operation is the creation of a new site, with the same structure (core, modules, themes) and contents than an existing one but with different configuration (database or database prefix and credentials). This scenario is applicable with there is a need to create a test site mirroring a production site on the same host.

> php sifcmd.php sip_build_site --sifpacksdir=../sifbuilder_local/packages
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbd501_at_localhost
--siburi=./sibs/sib_drupal47_sif_sif.xml  --dev  -x -f

If the sbd501_at_localhost was available or has been created as descrbed above, it can be cloned with the following (do it without the -f and -x options first).

> php sifcmd.php clone_pod
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetbcksite=sbd501_at_localhost --sifnetsite=sbd502_at_localhost
drupal47_core_drupal -f -x

In this case, the sifnetbcksite refers to an existing site. sifbuilder will take care to create a backup from it. Then, it will restore the backup file system and database sql dump taking into account the specifications in sifnetsite , normally different from the source site.

By now, sifbuilder -though the site definition specified in the source sifnetbcksite and the target sifnetsite site -has created a new tree in the file system and a new database, eventually with a different db prefix.

However, most wcm have a configuration file on the file system that, at this point, still refers to the original site. Importantly, all the operations until now have been blindly done, in the ignorance of the core being served. However, the generation of the configuration file on the site file tree is part of the installation process of the core, as described here, and sifbuilder must be made aware of the core installation package that should address that task. The core sip is thus included in the command so that the sif function clone_pod calls the sip configuration function of the core sip.

A different scenario appears when a site has to be migrated to a new host and eventually a new operating system. In this case, it is necessary to generate the backup, move the backup to the target host and do the restore there. This is done specifying the siburi in the backup and restore commands.

> php sifcmd.php backup_pod
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbd501_at_localhost
--siburi=bck_pod__sbd501_at_localhost.zip -f

> php sifcmd.php restore_pod
--sifnetconfigfile=../sifbuilder_local/sifnet_config.php
--sifnetsite=sbd502_at_localhost
--siburi=bck_pod__sbd501_at_localhost.zip -x -f

Other cores

These actions are usable mutatis mutandis for joonla, postnuke or xoops cores and sites. The correspondence is shown in the table below.

core sib (xml site defintion) site settings (defined in sifnet_config.php)
drupal sib_drupal47_sif_sif.xml sbd501_at_localhost
joomla sib_joomla1_sif_sif.xml sbj501_at_localhost
postnuke sib_PostNuke08_sif_sif.xml sbpn501_at_localhost
xoops sib_XOOPS20_sif_sif.xml sbx501_at_localhost

During this recasting, a major event has taken place. A new siteParam has seen the light. The db_prefix_sep. Cloning sites implies unprefixing and prefixing tables. For this, it is necessary to know the syntax of table names. Most cores use the underscore as prefix separator. joomla does not. In sifbuilder 1_8_9 that parameter defaults to underscore in the unprefix and prefix functions in SifMdb. The builder is sceptical about historical and quantitative reasons and, if does not arise a good reason for that, chances are that it will be changed to null.

posted by admin at 2007-07-22 00:00:00 0 comments read more