When I was learning Pharo Smalltalk, I found the Pharo By Example book a great help.  It was well written and available at a good price ;). However as Pharo advances at great speed and the original authors are busy documenting and implementing new and advanced features – this learning resource has become out of date with some of its examples.  So I thought I’d contribute back to help update it to the latest release.

For those that came in late, Smalltalk is a programming development environment that has been around since the 1970s.  Derived from the original Smalltalk-80,  Apple Computer released Squeak around 1996 as a platform for developing educational software.  In 2006 Squeak was released open source.  This was later forked to focus on business software development and released in 2009 as Pharo 1.0.  At this time the 2007 book Squeak By Example was updated and published as Pharo By Example.  Since then there have been several revisions of Pharo, with the most recent release in April 2012 of Pharo 1.4.

Pharo has a goal of refurbishing a lot of the inherited infrastructure, so it is uncertain how closely the book aligns with the current release.  Now I could just mark changes and submit for someone else to integrate these into the document, but that is really only half the job and creating work for someone else.  Since the LaTeX source of the PBE1 book has open access on github, I’ll decided I would have a crack editing the source direct – except I haven’t used LaTeX or git before.  So here I am documenting my setup for two reasons:  One is that some adept might point out improvements in my process; the other is to see if I can spark an interest in anyone else to come along for the ride.  Now lets see what I’m getting into…

Getting started

The PBE1 book is written in LaTeX , the source of which is managed on github.  This is my first experience with both LaTeX and git, so I’ll be happy to update this post with any hints from readers.  The README describes an overview of the contribution process.  What I document here is my particular circumstance of doing this with Windows 7 – as much for my own reference as trying to draw in an army of willing helpers.

The README specifies use of TeX Live, for which I first used the Windows installer.  Serge Stinckwich suggested starting with help.github.com which I found useful create an account and install Git For Windows as instructed here – and also play with some tutorials.  However while I successfully cloned the PBE1 repository to my local machine, when I went to do my first build to reproduce the existing book prior to any changes, the lack of make on the Windows platform became problematic.

To get make, I ended up choosing Cygwin.  However it seemed awkward to get the make from inside Cygwin to play well with TeX Live outside its root in Windows.  Since Git For Windows was only a text shell anyway, I chose to discard the Windows versions of git and TeX Live and use these both from inside Cygwin. So my Cygwin install (currently version 1.7.14-2) went like this:

• Logged in as admin user (since my daily use is under a non-admin account)
• Download and run setup.exe
• Install from Internet
• All Users ; Root Directory=C:\Apps\Cygwin\root
• Local Package Directory=C:\Apps\Cygwin\installer
• Direct Connection & Download Site – ftp://mirror.aarnet.edu.au
• Selecting packages…
  • searched “make” and selected “Devel > make”
  • searched for “git” and selected “Devel > git,  git-completion” (I did wonder about git-gui & gitk but wasn’t sure about about the operation of a gui from Cygwin and what dependencies would be loaded)
  • searched “texlive” and selected every “Publishing > texlive” package that did not have a foreign language in it.  I did this by changing Publishing from “Default” to  “Install” then searching on “live-collection-do” and “live-collection-lan” and deselecting all but English packages.
  • searched “ssh” and selected “Net > openssh”
  • search “vim” and selected “Editors > vim” (since I know ‘vi’ from long ago, but you might like something else if you browse the whole list of Editors – actually ‘lyx’ looks interesting to try later so I selected that also)
  • searched “time” and selected “Utils > time”
• Ticked “select required packages to resolve dependencies.”

The install took quite a while (upwards of 20 minutes with most of that being TexLive) after which I switched back to my non-admin user.

Github setup and practice

After running Start > Programs > Cygwin > Cygwin Terminal, I followed the instructions for linux-set-up-git jumping in at “Set Up SSH Keys.”  I then created directory ~/dev and changed into it before doing all parts of Create a Repo for Hello-World.  For this learning experience I unticked “Initialize this repository with a README”.  I followed up doing the fork-a-repo tutorial to test creating a fork of the “spoon-knife” project.

So now it was time to get my own local fork of PBE1 to work on.  Serge Stinckwich had already created a 1.4 branch seen on the right and here.  This is where I first thought I needed to fork from, so I first selected the branch to 1.4 at (1) below and then created the fork from that by clicking at (2)….

…but actually it doesn’t seem to operate that way.  Doing the following clone ended up with my  repository having both master and 1.4 branches.

$ cd ~/dev
$ git clone git [at] github [dot] com:bencoman/PharoByExample-english.git
(which downloads about 70MB)
$ cd PharoByExample-english
$ git branch -a
* master
remotes/origin/HEAD -> origin/master
remotes/origin/master
remotes/origin/pharo1.4

So it seems that I only select a particular branch for editing on my local machine.  After reading ‘man git-remote’ and ‘man git-branch’ from the Cygwin shell,  I continue with the tutorial…

$ git remote add upstream git://github.com/SquareBracketAssociates/PharoByExample-english.git
$ git fetch upstream
$ git branch -a
* master
remotes/origin/HEAD -> origin/master
remotes/origin/master
remotes/origin/pharo1.4
remotes/upstream/master
remotes/upstream/pharo1.4

So at this point I’m not sure if I should checkout remotes/upstream/pharo1.4 or just pharo1.4, or create my own pharo1.4 branch.  However I try this, which you can see moves the current-marker from master to pharo1.4…
$ git branch pharo1.4
$ git checkout pharo1.4
Switched to branch ‘pharo1.4′
$ git branch -a
master
* pharo1.4
remotes/origin/HEAD -> origin/master
remotes/origin/master
remotes/origin/pharo1.4
remotes/upstream/master
remotes/upstream/pharo1.4

Now at this stage, the upstream/pharo1.4 changes have have not been merged into my pharo1.4.  For example, I know that Serge Stinckwich has updated worldMenu.png in the upstream/pharo 1.4 branch – but at the moment if I browse Windows Explorer to “C:\Apps\Cygwin\root\home\Ben\dev\PharoByExample-english\QuickTour\figures” I see worldMenu.png looks like “Old World Menu” below.

To get the New World Menu above, I executed the following line the reopen worldMenu.png from Windows Explorer.
$ git merge upstream/pharo1.4
Removing QuickTour/figures/SqueakMap.png
Merge made by the ‘recursive’ strategy.
QuickTour/figures/SqueakMap.png                   |  Bin 23867 -> 0 bytes
QuickTour/figures/worldMenu.png                   |  Bin 36937 -> 27729 bytes
QuickTour/figures/yellowButtonMenuOnWorkspace.png |  Bin 36518 -> 38602 bytes
pbe1-examples.txt                                 |    2 +-
4 files changed, 1 insertions(+), 1 deletions(-)
delete mode 100644 QuickTour/figures/SqueakMap.png

Producing the book

Before starting making any changes of my own and potentially break something in the build I thought should confirm I could produce the existing book.  This is actually its pretty damn simple.  I just tried the obvious and this worked….
$ pwd
~/dev/PharoByExample-english
$ make
[lots of stuff]
Output written on PBE1.pdf (338 pages, 13825502 bytes).
Transcript written on PBE1.log.
real    0m45.536s

So now browsing Windows Explorer to…
C:\Apps\Cygwin\root\home\Ben\dev\PharoByExample-english
and opening PBE1.pdf to page 6 shows the updated WorldMenu.

Making a change, commitment, collaboration

(Note this section will change slightly when hopefully my addition of file gitDummyPlay.txt is accepted into the upstream/pharo1.4.)

File gitDummyPlay.txt file in ~/dev/PharoByExample-english is proposed for new contributors to practice pushing changes back to github.  If it does not currently exist, do this…
$ touch githubDummyPlay.txt
$ git add githubDummyPlay.txt
$ git commit -m ‘Added dummy file to experiement with while learning git’
$ git push

Now edit the file ~/dev/PharoByExample-english/githubDummyPlay.txt. Compare this to the same file currently on github.com/{your username} under PharoByExample-english/tree/pharo1.4.   Now do the following and compare both again…
$ git commit -m ‘Test pushing changes to github’
$ git push

This produces the following network graph, which will disapper when the changes are merged into the mainline.

Now the idea would be to issue a Pull Request to the upstream project, but I am waiting on some feedback.

I wonder if it would be useful to take a git branch for each chapter.  This would be similar to one git usage of having a branch for each new feature – perhaps allow more independent updates by multiple editors.

Conclusion

So you can see it is not too difficult to participate at the level of helping out with documentation.  Particularly if you are new to Pharo, you may pick up gaps in the doucmentation that are missed by others more familiar with it, so your input here can be valuable.  Now that I have this process reasonably sorted out in my mind, I just need to learn LaTeX better.

Yet I was most of the way through getting DBXTalk working, so I record my experience here for posterity.  It is the “hack” version since to resolve library dependencies I simply copied everything next to virtual machine executable.  I’ll look into resolving these more correctly later.  So I…

1. Created a new folder Pharo-1.3-13315-cog2522-dbxtalk and into it:
   a. Extracted the “Contents” folder only from Pharo-1.3-13315-OneClick.zip
   b. Extracted all files from cogwin_r2522.zip
   c. Updated croquet.ini file with: ImageFile=Contents\Resources\pharo.image
   .
2. Ran croquet.exe and then:
   World Menu > Monticello Browser.
   .
3. From the Monticello Browser opened:

   http://www.squeaksource.com/MetacelloRepository

   and loaded:
   ConfigurationOfOpenDBXDriver-GuillermoPolito.13
   .
   then from a Workspace executed:
   (ConfigurationOfOpenDBXDriver project version: #’stable’) load.
   .

4. From a Workspace found the required OpenDBX library to be 1.4.4 by executing:
   OpenDBX currentOpenDBXVersion
   .
5. Extracted all nine files from http://linuxnetworks.de/opendbx/download/opendbx-1.4.4_win32.zip into folder Pharo-1.3-13315-cog2522-dbxtalk
   .
6. From a DBXTalk developer I asked which version of PostgreSQL to try, and the prompt response came back as 8.3. I installed the current version at this time (postgresql-8.3.17-1-windows.exe) choosing all the defaults.
   .
7. Now possibly step 3 is made redundant by the following, but this post is a record of what I actually did.
   .
   In Monticello Browser I opened repository:
   http://www.squeaksource.com/DBXTalk
   .
   then loaded:
   ConfigurationOfGlorpDBX-GuillermoPolito.70
   .
   then in a Workspace I executed:
   (ConfigurationOfGlorpDBX project version: #’stable’) load.
   .
   which left the following message in Transcript 

   IMPORTANT FOR GLORP AND OpenDBXDriver DRIVER In order to run sucessfully Glorp tests you should need to change the database connection settings used by them. To do this, change the following methods: -GlorpDatabaseLoginResource>>defaultPostgreSQLInternetLogin -GlorpDatabaseLoginResource>>defaultPostgreSQLLocalLogin -GlorpDatabaseLoginResource>>defaultPostgreSQLLoginForGlorpStore  After doing this all Glorps tests must be green. Evaluated -> GlorpOpenDBXDriver >> postLoadGlorpDriverDBXTalkPharo …finished 2.4

   Looking at GlorpDatabaseLoginResource>>defaultPostgreSQLLocalLogin shows:

   defaultPostgreSQLLocalLogin “To set the default database login to PostgreSQL, execute the following statement.” “self defaultPostgreSQLLocalLogin.”  ^DefaultLogin := (Login new) database: PostgreSQLPlatform new; username: ‘sodbxtest’; password: ‘sodbxtest’; connectString: ’127.0.0.1:5432_sodbxtest’.

   Now rather than changing these connection settings, for a first attempt I thought it might go easier the other way making the database match the tests, so…

8. From the Windows Start Menu ran pgAdminIII.
   .
9. From pgAdminIII created a new user sodbxtest:
   Servers > PostgreSQL 8.3 (localhost:5432) >Login Roles > New Login Role.
   with the widest privileges possible.
   .
10. From pgAdminIII created database sodbxtest:
    Servers > PostgreSQL 8.3 (localhost:5432) > Databases >> New Database
    Name=sodbxtest
    Owner=sodbxtest
    Privileges=ALL, Role Public
    .
11. Then for later comparison I checked that the number of tables in this new database is zero: Servers > PostgreSQL 8.3 (localhost:5432) > Databases > sodbxtest > schemas > public > Tables
    .
12. Now “the hack” (since I was in a rush and would have needed to log out and back in as Admin to adjust the search paths.)
    .
    In Pharo I ran some Glorp Tests and got an error “could not find libpq.dll.”  After finding this file in folder PostgresSQL/8.3/libs, rather than properly resolving the lookup, the quick fix was copying everything from that folder next to the virtual machine in folder Pharo-1.3-13315-cog2522-dbxtalk.  A subsequent error “could not find ssleay32.dll” was fixed similarly by copying the contents of folder PostgresSQL/8.3/bin next to the virtual machine.
    .
13. Then in Pharo from World Menu > Test Runner all 823 tests for GlorpTests-Models, GlorpTests-Database & GlorpTest are successful.
    .
    In addition, if I recheck the number of tables noted in Step 14 and find there are now 75 tables – full of data.
    .
    Excluded from this are GlorpTests-DatabaseTypes, GlorpTests-Extras, GlorpOpenDBXDriverTests since they seem to cycle through testing databases  that I don’t have installed – in particular OCI.dll for Oracle.

One small annoyance is that once a “missing OCI.dll” error occurs, it continues to occur even when the previously successful tests are rerun.  This remains until reset by: GlorpDatabaseLoginResource defaultPostgreSQLLocalLogin.

So this is a good result and I’ll be glad to further DBXTalk later on.  However for now, for my current requirements (a one-time one-way import from an mdb file into Pharo) I think that ConfigurationOfODBC is more suitable.

This post summarises the result of discussion on the pharo-project mailling list where I sought assistance getting ODBC working on Pharo 1.3.  Credit goes to Mariano Peck and Eliot Mirranda for assistance troubleshooting, providing a slightly older configuration that worked, and then the latest VM build.  I am happy to report that ODBC appears to be working with the Pharo 1.3 image on MS Windows 7 using CogVM version 2522.

The purpose of this is that I have a UML design an application has stored in a Microsoft Access database.  I want to use Pharo to implement that design directly into Smalltalk classes since I could not find a common export/import format.  ConfigurationOfODBC looked promising so I followed instructions at http://www.pharocasts.com/2010/12/access-database-through-odbc.html, except that rather than using SQLLite I started with a blank Microsoft Access database.  This worked well on Pharo 1.2.1 but not out of the box Pharo 1.3.

Here is the method to get ODBC working and tested with Pharo 1.3.

1. With this  TestMDB.mdb in a local folder, set up the ODBC data source using
Start > Control Panel > Administrative Tools > Data Sources (ODBC)

ODBC Data Source Administrator

Clicked <Add…> and choose the Microsoft Access Driver

and <Select…> the TestMDB.mdb file configured as Data Source Name “TestMDB”

which gives…

2. Unzip Pharo-1.3-13315-OneClick.zip and then delete all files from
folder “Pharo-1.3-13315-OneClick.app”, leaving just the Contents folder.

3. Extract the files from VM.r2522/cogwin.zip into folder “Pharo-1.3-13315-OneClick.app”

4. Copy the “ImageFile=Contents\Resources\pharo.image” line from the old pharo.ini into Croquet.ini and then run Croquet.exe.

5. Load the ODBC package from within Pharo.  I had some trouble accessing Squeaksource, but I found success using a mirror (you might skip the first two lines…)

Gofer new url: ‘http://dsal.cl/squeaksource/MonticelloRedirect’; package: ‘MontiRedirect’; load.

MRManager redirectFrom: ‘http://www.squeaksource.com/’ to: ‘http://dsal.cl/squeaksource/’.

Gofer new
squeaksource: ‘ODBC’;
package: ‘ConfigurationOfODBC’; load.
(Smalltalk at: #ConfigurationOfODBC) load.

6. And then finally, the successful test of ODBC…

connection := ODBCConnection dsn: ‘TestMDB’ user: ” password: ”.
table := (connection query: ‘select * from user;’) execute asTable.
connection close.
table inspect.

Inspector on ODBCResultTable

Thats all.  Now time to explore that UML design….

Here are some links to the full discussion and a backup cache of the files

• http://thread.gmane.org/gmane.comp.lang.smalltalk.pharo.devel/56184
• http://files.openinworld.com/Pharo/Pharo-1.3-13315-OneClick.zip
• http://files.openinworld.com/Pharo/cogwin_r2522.zip

Update 24 Dec 2011…

7. Just recording an example for the next part of actually processing the database. Following on from the previous step…

( row := table first ) inspect.
( name := row at: #name ) inspect.
( age := row at: #age ) inspect.
( allNames := table collect: [:each | each at: #name] ) inspect.

To go with the above example I have also uploaded a new file TestMDB – 2011-12-24.zip with a few additional rows of data.

Logitech Wireless Mouse M305 Middle Button Setting

That done, there are still a few configuration options in Squeak that affect operation.  In particular [ <F2> System Menu > Mouse Mapping ] affects how  particular direct manipulation actions are invoked.

By way of a specific example, Table 1 shows images of the possible actions and menus for a RectangleMorph, which are:

1. Select – This picks up the morph from the World and gives it to HandMorph, which then moves it around the screen
2. Meta – This brings up the halos around the object
3. Menu 1 – I’d really like to be able to be more distinct in distinguishing between these  menus besides “Next to its name is a small thumbnail of its morph .” Does it have a better designation?
4. Menu 2 – Similar to Menu 1, is there a proper designation of this menu?  I can only concisely describe it as “Next to its name there is no thumbnail of its morph.”
5. Halos > Menu – Just included for comparison (in Table 3) to try and work out the definition of Menus 1 & 2.  This is inconclusive since its menu items are somewhat split between Menus 1 & 2.Table 1 – Direct Manipulation Mouse Actions

Table 1 – RectangleMorph actions & menus

1. Select	2. Meta	3. Menu 1	4. Menu 2
		5. Halos > Menu	6. Halos > Debug

The effect of the different Mouse Mappings is shown in Table 2, which tabulates the effect of the Ctrl & Alt keypress modifiers on mouse button clicks.  The three Mouse Mappings (Default /  3 Button / 1 Button) are shown together.  This shows a few things:

1. The left mouse button remains the same regardless of the mapping
2. For both mappings of three buttons, the button assigned to Halos is always Halos.  Halos are unaffected by any Ctrl or Alt keypress modifier.
3. For both mappings of three buttons, the button assigned to Menu is always Menu. Menus are affected by the Ctrl keypress, which changes from
   Menu 1 to Menu 2.
4. The 1-button mapping continues to work with both mappings of three buttons, and so is a lowest common denominator that might be considered for tutorials.
5. With a two-button mouse and Default mapping it is likely that Menu 2 is missed, requiring both Ctrl & Alt keys to be held together.
6. With a two-button mouse and 3-button mapping, the Alt key can be ignored. The Ctrl key changes Select and Menu 1 respectively to Halo and Menu 2.

Based on this, I’ll configure the Mouse Mapping as “3-Button.”  I’m not sure how the balance between the usage of halos and menus will pan out, but as a default I’ll prefer to have my menus on the same button as the rest of the operating system.  This may come up for review as my usage of the system evolve.

Finally, just for clarity, I compare how items from the various menus overlap.  As a Squeak newbie, there are some small inconsistencies that impede that comfortable feeling we associate with familiarity. This comparison helps build that feeling of familiarity.

1. For the same functionality, one menu says “Select”
   and another says “Add Halo”
2. For the same functionality, one menu says “Send to Back”
   and another says “Go Behind”
   
3. [ Halos > debug menu > control menu ] brings up exactly Menu 2.  Perhaps “control menu” might be a reasonable name for Menu 2, though I’m not sure.

Now to be controversial.  Coming newly into this, the whole red button, blue button, yellow button thing is a PITA.  My wild speculation is… that back at the birth of Smalltalk the guys at Xerox were essentially exploring the creation the modern GUI.  They either did not have sufficient terminology, or did not want lock down the terminology too early. Neuro Linguistic Programming tells us that words constrain thinking, so that was probably a good decision at the time.  However thirty years on, the GUI paradigm and the concepts of Select, Context Menu and Meta are well established.  So why not now use these? …and be a little bit more palatable to the rest of society.

The designations of Select, Menu and Meta are in fact referenced at the Squeak Wiki FAQ: Mouse Buttons and would maintain the “arbitrary” nature of “coloured” buttons discussed at Andrew P. Black on mouse button colors and the command key.

However everything historical is based around the coloured buttons, so I will probably just have to “get used to it.”  Here it is for quick reference until it soaks in…

        "red"                   -> select
        "yellow"                -> menu
        "blue"                  -> meta operation

• http://blog.openinworld.com/2011/03/morphic-performance/
• http://blog.openinworld.com/2011/03/morphic-flavour-performance/

You are able to attach graphs to your comments in those articles.

Although I had seen these flaps while previously playing around with Squeak 4.2, [Projects > New Project > New MorphicProject] now did not show any.  The flaps needed to be enabled through [World menu > flaps > show shared flaps].

I then created a workspace [World > Workspace] and an object in the world [Objects flap > Categories > Graphics > Curve > drag onto world ].

First the default behaviour was tested by dragging Curve over the Workspace.  As expected it did nothing.   Curve was dragged away from Workspace.

It was not clear from Andy’s article where the functionality was configured.  The first tried was “1″
[Halos > Menu > accept drops]. However when Curve was dropped onto Workspace, no text appeared.

Instead Curve became stuck to Workspace such that dragging Curve also dragged Workspace with it.   Curve had to be removed from Workspace using [Curve > Halos > Pick Up].  Then [Halos > Menu > accept drops] was disabled.

Further exploration found another related menu, marked “2″ above. This did have the required menu option.  After enabling [create textual references to dropped morphs], Curve was dropped onto Workspace.  This now created the text “curve2970″ in Workspace.  Then [curve2970 > inspect it] displayed an identical inspector to that displayed for [original graphic object > Halos > Debug > Inspect It].

That is way cool.  Are there any other programming environments that can do that out of the box?

I then found some additional info, which I’ll leave for further reading…
Squeak Wiki FAQ: How do I access a Morph from a Workspace?

On [squeak-dev] Herbert Konig wrote: “Juan Vuletich has done a great job at simplifying cleaning and speeding up his version of Morphic in Cuis.”

So lets check that out.  We should compare some other flavours of Smalltalk.  The code used to generated the graph data is the same  “V1″ code as used for morphic performance. The software versions tested here are:

• Pharo-1.1.1-OneClick
• Pharo-1.1.1-OneClickCogVM
• Cuis 3.0 All-in-One.app
• Squeak 4.2 All-in-One.app
  

Figure 1 - Interflavour Morphic Performance Testing - "v1" code (note x-scale is 10% of Figure 2)

Some interesting points are:

• Cuis is the fastest for creating, hiding and showing morphs.  For deleting morphs Cuis is no faster than non-cog Pharo/Squeak, following the same exponential curve.
• Pharo-CogVM is by far the fastest at deleting morphs, near the fastest at creating morphs, but much slower at hiding and showing morphs.
• Non-CogVM Squeak and Pharo are very similar throughout the four graphs, with Pharo being slightly faster.   Both increase exponentially for creating & deleting morphs.  I’m not sure what to make of the smoother Pharo curves.

An update will follow to include the effect of Juan Vuletich’s “addAllMorphs” performance tweak of the original code.

Update 31 March 2011
I updated the original morphic performance article with Juan Vuletich’s  addAllMorphs performance tweak.  It will be interesting to see how that compares across flavours of Squeak.

In addition, for some flavours of Squeak the previous code didn’t display any morphs at all.  It seemed that the tightness of the creation/deletion loops might have optimised away the actual drawing of the morphs.  I don’t expect that is how applications would operate so the results would be skewed.  More fair I think is to force a redraw between each of the four tests, where I’ve added doOneCycle below.  What do you think?

 rand := Random new.
creating :=
[    m := EllipseMorph  new.
      ellipses add: m.
      m color: Color random.
      m position: ActiveWorld extent * ((rand next)@(rand next))
].
deleting := [ World removeAllMorphsIn: ellipses. ].
hiding := [ ellipses do: [ :item | item hide ]].
showing := [ ellipses do: [ :item | item show ]]. 

Transcript show: 'Morphic Performance Test Code v3.0 - ##FLAVOUR## '; cr.
Transcript show: 'amount, createTime, hideTime, showTime, deleteTime'; cr.
#( 1 10  100 200 300 400 500 600 700 800 900 1000 2000
3000 4000 5000 6000 7000 8000 9000 10000  30000 100000) do:
[ :amount |
         ellipses := OrderedCollection new. 
         createTime :=
         [         amount timesRepeat: creating. 
                   World addAllMorphs: ellipses. 
                   ActiveWorld doOneCycle.
         ] timeToRun . 
    hideTime := [ hiding value. ActiveWorld doOneCycle. ] timeToRun .
    showTime := [ showing value. ActiveWorld doOneCycle. ] timeToRun.
    deleteTime := [ deleting value. ActiveWorld doOneCycle. ] timeToRun.
    Transcript show: amount asString,          ', ', 
                    createTime asString,    ', ', 
                    hideTime asString,        ', ', 
                    showTime asString,       ', ', 
         deleteTime asString;     cr.
]

Figure 2 - Interflavour Morphic Performance Testing - "v3" code with addAllMorphs

It is intriguing how very straight the lines are now in Figure 2 compared to Figure 1.  Somehow I feel like something must be wrong.  Anyhow, thanks to Juan’s suggestions, as well as being ten times faster, the graph shape is now linear rather than exponential - a much better prospect for system scaling.

Also interesting is how similar the flavours of Squeak are now.   For up to 30,000 morphs the createTime is identical for all four flavours.  After that performance does diverge, with Cog being fastest, Squeak & Cuis together in the middle and Pharo the slowest at about half the speed of the Cog.

Strangely, Cog has a hideTime four times slower than the other three.   For showTime, both versions of Pharo (Std & Cog) are together twice as fast as Cuis and Squeak.

Test platform was Windows 7 Professional 32-bit Intel Core2Duo P8700 2.53Ghz 3GB RAM with graphics from Mobile Intel 4 Series Express Chipset.

I am interested in your observations.  Can you replicate these results?
(graphs can be attached to comments)

2011-03-10 03:55:15 procwatch2 INFO: Process(pid=1755, name='php5.cgi',
uid=xxxxx(10537396), tty=None, cpu=2.0, rss=42516.0, vsize=148964.0):
kill for total RAM

And one of many messages from file: logs/http/error.log:

[Thu Mar 10 04:01:59 2011] [error] [client 124.169.187.59]
Premature end of script headers: symposium_bar_functions.php,
referer: http://blog.openinworld.com/2011/03/morphic-performance-testing

Keeping the system balanced for all tusers is fair – but how does that affect me directly?  Am I able to work with this?  WordPress appears to put a relatively heavy strain on server memory.  I did not anticipate it the memory limit could be exceeded so easily, but I guess that is the reality of shared hosting.

DreamHost refrains from specifying what the RAM limit is.   Perhaps its dynamic, or a marketing issue, or it changes with time.  I’ve seen speculation elsewhere of it being 100MB, 120MB & 200MB in posts from 2006, 2010 & 2007.  From a shell with the `top` command I monitored my live processes.  How to Profile Memory in Linux describes the columns below.  Specifically we are interested in the resident memory.  Three WordPress php5.cgi listeners are shown, each idling comfortably around 45MB.  That might peg the RAM limit somewhere between 150MB to 200MB.

Interestingly, comparing this to the `top` snapshots at DreamHost WordPress Optimization I note the RES memory there is around 7.5MB versus my 35MB.  While part of this may be be due to general software bloat between versions, I suspect it may mostly be from RAM usage doubling between 32-bit and 64-bit operating systems.  Hopefully the RAM Limit was doubled accordingly to compensate.

So what to do? The simple suggestion from DreamHost was to “disable all your WordPress plugins.“  That seemed a bit extreme, but will at least help to understand the issue.  Before anything else, I went looking for a WordPress plugin to show plug-in memory usage. I installed TPC! Memory Usage (TPCMU), which immediately started firing the following email at me several times a second.  Note that the 32MB is a user defined soft limit.

WordPress memory usage exceeded 32 MB
WordPress peak memory usage: 34.85 MB
Number of database queries: 36

I then removed every plugin except TPCMU , and `top` then reported RAM usage 10MB better per listener – at around 35MB each.  TPCMU reported WordPress peak memory usage as 22.4MB. 

top2 with 0 plugins

To observe the effect of reinstating each plugin, I tracked the Memory Usage displayed in the footer of the Admin > Plugins > Active Plugins page.  The results are tabulated below.  Marked in red are the plugins I ultimately removed, judging that functionality gain not worth the pain of required RAM.  I believe that remaining plugins are a reasonable balance of functionality versus memory usage  – particularly since two thirds of that is aimed at performance and security, with an overall benefit to other users of the server.

TPCMU now reported 28.5MB and `top` showed php.cgi at 41MB.

I saw several tips suggesting that rather than just deactivating plugins, deleting them saves more space. I had 42 Inactive plug-ins, including those just deactivated and others I had tried and not liked.  Deleting all of them only reduced the TPCMU report down 0.1MB to 28.4MB – not much really.

DreamHost has some community information available.  DreamHost WordPress_Performance discusses page load speed. It recommends a few caching plugins and the use of FastCGI.  Faster processing reduces overall memory usage somewhat due to there being less concurrent php.cgi processes.

DreamHost WordPress Optimization discusses four site configurations in relation to memory usage.  It also recommends caching.  However it recommends NoFastCGI saying:

[Cache + NoFastCGI] has great performance while having the lowest overall resource hit on the server. It should always be using the least amount of memory and CPU time that it possibly can [...] whereas in a FastCGI configuration the number configured to spawn would remain open whether or not any requests were coming in.

When idle, FastCGI processes sit around continuing to consume memory.  Hence the cumulative memory usage over time would be greater than NoFastCGI.  That could be considered a greater load on the server.   However FastCGI provides a more steady and predictable load, which is more desirable for tuning that one that fluctuates.  FastCGI should handle peak loads more gracefully.  NoFastCGI has a fairly important proviso…

The only time [Cache + NoFastCGI] might be less efficient is if a large number of un-cached pages are accessed simultaneously as a number of php5.cgi processes would spawn to handle those.

…in which case ProcWatch would assumedly start killing off random connections for exceeding the user RAM limit. For a high traffic site (wishful thinking!) FastCGI should handle each cache-miss faster, thus requiring less overall RAM with less chance of connections being killed.  Ideally, the number of spawned FastCGI processes could be limited to avoid a “Kill For RAM” scenario.  However I haven’t been able to determine how to limit the number of FastCGI processes spawned.

I might have to keep an eye on FastCGI though, since this  WordPress: PHP CGI vs PHP FCGI post  points to some issues with FastCGI – that I have not experienced that so far.

In conclusion, Dreamhost offer shared hosting that is unlimited in many of its (advertised) parameters.  However overall service capability is limited by the tightest constraint – which in this case is RAM – the one they don’t advertise (and neither does any other shaed host I’ve seen.)  To work within this limit and resolve the “kill for total RAM” issue, I have:

• investigated the memory usage of WordPress plugins
• reduced the number WordPress plug-ins
• activated an application-aware cache “Supercache”
• continued to use FastCGI (which is the DreamHost default)

In the end, these are only simple things – but I’ve learnt a lot along the way.  I was considering further action with some of the methods from the WordPress Optimization Bible, but my error.log file has been clear for a few days, so I’ll leave those for later.

Herbert Konig on [squeak-dev] suggested:

So based on part of a sample electrical connection diagram…

Sample Terminal Strip

…my back of envelope calculation for the number of morphs is:
1000 ==  2 x terminal strips x 100 terminals x 5 columns (core,ferrule,term,link,core)
2400 == 2 x 100 terminals x 4 wires x 3 line segments (two wires per terminal per side)
400 == wire cores identifiers
20 == 20 x 1 cable label per each 20 core cable
500 == miscellaneous border and drawing title info
==> ~4500.  Double it for safety ==> ~10,000 morphs.  This is for an individual drawing.  For an industrial plant the number of drawings numbers in the thousands.

I developed the following code to show the performance of EllipseMorphs creation. I’ve used variables to contain blocks of code to make the main loop more concise.

"Morphic Performance Testing v1"
rand := Random new.
creating := [
    m := EllipseMorph  new.
    m color: Color random.
    m position: ActiveWorld extent * ((rand next)@(rand next)).
    m openInWorld.
    m goBehind.
    "ActiveWorld doOneCycle."   ].
deleting := [ EllipseMorph allInstances do: [ :item | item delete ]].
hiding := [ EllipseMorph allInstances do: [ :item | item hide ]].
showing := [ EllipseMorph allInstances do: [ :item | item show ]].

#( 1 10  100 200 300 400 500 600 700 800 900 1000 2000
 3000 4000 5000 6000 7000 8000 9000 10000  )
do: [ :amount |
    createTime := [ amount timesRepeat: creating ] timeToRun.
    hideTime := [ hiding value ] timeToRun.
    showTime := [ showing value ] timeToRun.
    deleteTime := [ deleting value ] timeToRun.
    Transcript show:
       'amount=',       amount asString,
       ' createTime=',  createTime asString,
       ' hideTime=',    hideTime asString,
       ' showTime=',    showTime asString,
       ' deleteTime=',  deleteTime asString;  cr]

Morphic Performance Evaluation "v1" code

These times were using vanilla Squeak 4.2 All-in-One.app.  Two morph creation times were considered.  A fresh image was unzipped for each run.

The first creation time createTime(gui lockup) was the result of the code executed exactly as above.  The creation of so many morphs did indeed lock up the GUI while the code was executing.  However after that the interactivity returned to normal. Any ellipse could be picked up and moved around with no perceptible delay.  (This of course required the delete block to be commented out so morphs remained to interact with.)

Up to a certain delay, that initial GUI lock up might be considered standard operation for a file load.  Interestingly, the deleteTime(gui lockup) for the morphs takes longer than creating them.  I am speculating that has something to do with required garbage collection.

It can be seen that hideTime and showTime are much faster than create and delete. An alternative to “loading” a file might be for the morphs to exist permanently in the image a hidden onscreen and just toggle their visibility.   I wonder about keeping a standard set of morphs onscreen and swapping out the underlying object model per drawing.  I’ll have to learn a bit more to decide how that might be best achieved.

The temporary gui lockup seemed due to the event processing loop not running.  This is common for long running calculations in a single threaded application.  The standard solution is to periodically invoke event processing from within the calculation.  Rummaging through the system code I found my way to “ActiveWorld doOneCycle.”  You can see this line commented out in the code above.  By uncommenting that line, interactivity during morph creation was absolutely perfect during morph creation! Not a hint of gui lockup!  However the trade-off was greatly increased overall createTime(doOneCycle).  The original deleteTime(gui lockup) was no different from deleteTime(doOneCycle), so only the first is shown.

Perhaps using doOneCycle that way is a bit naïve – I’m open to suggestions.  Some off the cuff alternatives are:

1. Balance morph creation into batches.
2. See if Morphic can track the overall interactivity during any instance creation.
3. See if Morphic event checking can be timer interrupt driven.
4. Run the morph creation in another thread using ‘fork.’ I had tried that however and the experience was a bit weird.

Update 18 March 2011
On [squeak-dev] Juan Vuletich wrote:   

That code is good for benchmarking, but not for a real app. These minor tweaks allow opening a lot (really a lot) of morphs under one second.

"Morphic Performance Testing v2"
rand := Random new.
creating :=
[    m := EllipseMorph  new.
  ellipses add: m.
     m color: Color random.
     m position: ActiveWorld extent * ((rand next)@(rand next))
].
deleting := [ World removeAllMorphsIn: ellipses].
hiding := [ ellipses do: [ :item | item hide ] ].
showing := [ ellipses do: [ :item | item show ] ]. 

Transcript show: 'Morphic Performance Test Code v3.0 - ##FLAVOUR## '; cr.
Transcript show: 'amount, createTime, hideTime, showTime, deleteTime'; cr.
#( 1 10  100 200 300 400 500 600 700 800 900 1000 2000
3000 4000 5000 6000 7000 8000 9000 10000  30000 100000)
do: [ :amount |
         ellipses := OrderedCollection new. 
         createTime := [ amount timesRepeat: creating. 
                   World addAllMorphs: ellipses. 
         ] timeToRun. 
    hideTime := [ hiding value ] timeToRun.
    showTime := [ showing value ] timeToRun.
    deleteTime := [ deleting value ] timeToRun.
    Transcript show: amount asString,          ', ', 
                     createTime asString,    ', ', 
                     hideTime asString,        ', ', 
                     showTime asString,       ', ', 
          deleteTime asString;     cr.
]

Morphic Performance Testing "v2" Code

"Morphic Performance Testing v1"ss

In fact, those tweaks enable dealing with an order of magnitude more morphs (thx Juan.)  The creation time for 10,000 morphs went from 10 seconds to 0.9 seconds.  Alternatively in 10 seconds, 90,000 more morphs can be created.  This is reflected in the updated graph below as createTime(addAllMorphs).  Deletion of morphs is a massive two orders of magnitude faster, as seen by deleteTime(addAllMorphs).  Can you see more?