Scanning is slow - should I split?

Almost 4 years ago, I wrote a post entitled Scanning is slow.  Well, to be fair, it is.

The example code that I gave was for counting how many times a substring appeared within a string...

  temp = list

  total = 0

  scan temp,"ABC"

  while ( $result > 0 )

    total = total+1

    temp = temp[$result+3]

    scan temp,"ABC"

  endwhile

I gave a perfectly simply alternative which performed much better in this scenario, using $replace and $itemcount. 

However, I found myself in a similar situation yesterday, but I couldn't use this trick, because I needed to loop through, searching for certain substrings, but without changing the string that I was looping through.

A thought occurred to me, which was that I was always taking the string from after the value I'd found and scanning again, so in this case, what about using $split instead?

To compare against to above, I came up with something like this...

  temp = list

  total = 0

  $result = $split(temp,1,"ABC",lhs,rhs)

  while ( $result > 0 )

    total = total+1

    temp = rhs

    $result = $split(temp,1,"ABC",lhs,rhs)

  endwhile

I tested this by doing 500 iterations, on a string with 500 instances (so total was 250000 at the end)...

$scan: 20.94, 20.67, 20.85 - almost 21 seconds.
$split: 27.09, 27.48, 26.83 - around 27 seconds.

So $split definitely looked worse off, it looked like $scan was better suited to what I needed (as I wasn't interested in the left hand side).

However, these tests have been seeing whether I can use $split to improve my $scan loop, and this seemed a little unfair now I'd proven it couldn't, so I thought I'd flip it on it's head and see if $scan is better at doing what $split is designed for.


So for $split this is quite easy, a simple line which splits the string into two parts...

  $result = $split(temp,1,"~",lhs,rhs)

Using $scan this is a little more complicated, and requires a little string manipulation...

  $result = $scan(temp,"~")

  lhs = temp[1:$result-1]

  rhs = temp[$result+1]

When I compared the two over 2,000,000 iterations, I got these results...

$split: 10.19, 9.78, 10.01 - around 10 seconds.
$scan: 16.60, 16.51, 16.82 - over 16 seconds.

So I've worked out that $split is good for splitting and $scan is good for scanning... Why am I writing this up in a blog post, you might wonder!

Well I then considered the fact that I was splitting a string on a delimiter that I knew existed.  And in the real world situation I was working on, this wasn't a fair assumption, as often the delimiter would not be there.  So I tried this again, but with a different delimiter, that wouldn't be found in the string...

$split: 10.66, 10.73, 10.87 - a little over 10 seconds.
$scan: 9.20, 9.00, 8.44 - around 9 seconds.

So it looks like it's not quite as straightforward as I thought.  However, the saving when it is found is quite a lot larger than the loss when it's not found, so maybe $split is still best overall.

Summary: If you're looking to scan through a string, use $scan, and if you're looking to split a string (especially if you expect the delimiter to be there), use $split.

Sendmessage versus postmessage

If you wanted to send a message from one form to another, you could simply use sendmessage.  This was deprecated in favour of using activate, a command which has become pretty much a catch-all for all sorts of earlier commands, including run, perform, and spawn.  These all still work in Uniface 9.6, but as they were deprecated in Uniface 9.1, it's well worth trying to write these out of your code.

One example of this might be if you have retrieved multiple records in one form, and also have a second form open with related records in it; when you scroll through the occurrences in the first form you might want to post the primary key details through to the second form, so that only the records related to the current occurrence in the first form are displayed.

In order to do this, the first form would need to have code in the "Occurrence Gets Focus" <OGF> trigger, something like this...

  sendmessage "SECOND_FORM","PKEY","FIELD1.ENT=%%FIELD1.ENT%%%"

As you can see, because sendmessage is now deprecated, it is not highlighted as a reserved word.  You also get a compile warning like this...

  warning: 1000 - Deprecated statement 'sendmessage' used

You then need to have some code to receive this message in your second form, which needs to be in the "Asynchronous Interrupt" <ASYS> trigger, something like this...

  the_result = $result ;will be "message"

  message_id = $msgid ;will be "PKEY"

  message_data = $msgdata ;will be "FIELD1.ENT=xxx"

When using sendmessage or postmessage, $result will always be set to "message".  The second value that you define will become the $msgid and then third value will become the $msgdata.  

Please note:  If the <ASYS> trigger in the called component is blank then the application <ASYS> trigger will be called, as defined in the start-up shell.

So what's the difference between sendmessage and postmessage?  Can you simply go through and replace every sendmessage with a postmessage, or should you be using activate instead?

Well the difference is that sendmessage is synchronous and postmessage is asynchronous, which means that potentially there's a delay before the message will be received and processed with a postmessage.  Using an activate would be synchronous but you'd need to rewrite your code, probably moving the code in the <ASYS> trigger into an operation and then calling that from the <OGF> trigger.

Is this potential delay likely to cause a problem though?  Well for starters, can we see it in practice?  The answer to this is "yes", by putting sending code like this somewhere in a form (I went with the <EXEC> trigger)...

  putmess "Before sendmessage"

  sendmessage "TEST_PERF","MESSAGE","SENDMESSAGE"

  putmess "After sendmessage"

  putmess "Before postmessage"

  postmessage "TEST_PERF","MESSAGE","POSTMESSAGE"

  putmess "After postmessage"

And then putting the receiving code in the <ASYS> trigger, like this...

  putmess "In ASYS: ID=%%$msgid%%%, DATA=%%$msgdata%%%"

We can then run the form and view the message buffer...

  Before sendmessage
  In ASYS: ID=MESSAGE, DATA=SENDMESSAGE
  After sendmessage
  Before postmessage
  After postmessage
  In ASYS: ID=MESSAGE, DATA=POSTMESSAGE

As you can see, the sendmessage happens between the "before" and "after" (synchronously), whereas the postmessage happens at the very end (asynchronously).  This means that you have to be very careful about changing any sendmessage statements into postmessage statements - the code order will change, and you need to be aware of this and check this will not cause a problem.

As far as my testing has gone, the messages are always received and processed in the order that they were posted, as you may expect from a message queue.  Having said that, I've not found anything in the manuals that states explicitly that this order is enforced, so writing code that relies on the messages being processed in a particular order might not be wise.

Summary: The deprecated sendmessage command was sychronous whereas the postmessage command is asychronous, so you must be careful if you replace one with the other.  However, this is likely to be more straight forward than replacing with the activate command, as the manual suggests.

Scanning is slow

I'm always complaining when I see people using a scan or $scan when they don't need to.  Yes, it can be very useful, sometimes it's unavoidable, but here's an example of when it should not be used.

If I asked you to count how many times a substring appeared within a string, you might think about doing it this way...

  temp = list

  total = 0

  scan temp,"ABC"

  while ( $result > 0 )

    total = total+1

    temp = temp[$result+3]

    scan temp,"ABC"

  endwhile

It's perfectly logical code, it looks through the string, scanning for the substring, counting each iteration.  I've written this code myself, a few years back, and didn't think anything of it.  I recently encountered this code that I'd written, it happens to be part of an import process I use quite often.  However, on this particular day, I was importing 10,000 records - far more than usual.  Whilst I was waiting over an hour for this to import, I decided to check the code.

I noticed that I was using a scan and thought for a moment about what alternatives there were.  The first one I thought of seemed a little strange, but I was sure it would work, so I gave it a go.  This is what it was...

  temp = $replace($replace(list,1,"·;","",-1),1,"ABC","·;",-1)

  total = $itemcount(temp)-1

As you can see, I'm first removing an list delimiters (gold-semi-colon characters) from the string, and then replacing the substring with the list delimiter instead.  This now means that I have a Uniface list, and I want to know how many of these delimiters there are in the string.  This easiest way to do this is use $itemcount to count the number of items, and then deduct one, as there's always one more item than there are delimiters.  This worked a lot quicker!

I've reproduced this for testing, using a string with 500 occurrences of the substring, and performing the count 500 times...

• scan = 00:45.01, 00:43.70, 00:44.31 (just under 45 seconds)
• list = 00:00.86, 00:00.84, 00:00.84 (under 1 second)

As you can see, quite a staggering difference.  I hope you'll think twice before using scan again!  Obviously the loop and the rebuilding of the string is contributing, but I hope this is still a convincing argument.

Summary: Scanning a string can be essential, but it's a very costly function, so it's well worth thinking about an alternative approach.

Undocumented feature - interrupt - part nine

I know that I said I was done after part eight, but I've had a report of some additional functionality that I thought I should add.



5 - Uniface "usys" directory


As previously discussed, using numeric code "5" returns the working directory.  However, you can also use it to display the Uniface "usys" directory...   

  interrupt(5,"USYS")
  uniface = $result

EDIT: Thank you to James Rodger for pointing out the alternative in his comment...

  uniface = $valuepart($fileproperties("usys:","FULLPATH"))

This in itself seems to be an undocumented feature; I can't find any reference to "usys:" being accessible like this in $fileproperties or $lfileproperties.


I tried to access some other Uniface directories in a similar way, such as "adm" and "bin", but it seems that you can only access "usys".


Thank you to Mark R for point this one out.

Undocumented feature - interrupt - part eight

Continuing from part seven where we looked at "989", I will now do a full summary of the known functionality that we've looked at in this series of posts...

Code	Description	Example	Alternative
0	File information	interrupt(0,"test.txt") if ( $status < 0 )   ;file doesn't exist else   list = $result   name = $item("Name",list) ;"test"   type = $item("Type",list) ;"txt"   vers = $item("Version",list) ;""   attr = $item("Attrib",list) ;"UUU" endif	This doesn't seem to return any useful information now. Could use: $fileproperties or $lfileproperties These return lots more file information, but not the same.
2	Extract filename	interrupt(2,"C:\temp\test.txt") filename = $result ;"test.txt"	Use string manipulation (although interrupt takes 60% of the time)
3	Extract file path	interrupt(3,"C:\temp\test.txt") filename = $result ;"C:\temp\"	Use string manipulation (although interrupt takes 60% of the time)
5	Working directory	interrupt(5,"") working = $result ;"C:\temp\"	Use: $ldir
5	Uniface "usys" directory	interrupt(5,"USYS") uniface = $result ;"C:\uniface\"	Use: $fileproperties of "usys:"
6	Concatenate file path	$result = "C:\temp" interrupt(6,"test") filepath = $result ;"C:\temp\test\"	Use string manipulation (although interrupt takes 70% of the time)
7	Parent directory	interrupt(7,"C:\temp\test\") parent = $result ;"C:\temp\"	Use string manipulation (although interrupt takes 33% of the time)
10	Directory listing (files)	interrupt(10,"C:\temp") list = $result ;"test.txt·;test2.txt" count = $status ;2	Use: $dirlist or $ldirlist (with "FILE" topic)
11	Directory listing (folders)	interrupt(11,"C:\temp") list = $result ;"test·;test2" count = $status ;2	Use: $dirlist or $ldirlist (with "DIR" topic)
989	Application focus	interrupt(989,"APPLFOCUS")	No alternative (doesn't seem to work anyway)
989	Format string	$result = "RPL" interrupt(989,"FORMAT") string = $result	Use string manipulation (doesn't seem to work anyway)
989	Undo gold characters	$result = "R·;P·!L" interrupt(989,"UNDOGOLD") string = $result ;"R;P!L"	Use: $replace

It seems that the interrupt command isn't really very useful anymore.  The functionality has either been replaced by new Uniface functions, or can be achieved by string manipulation.  Given the performance improvements, it may be worth considering using interrupt if you are heavily handling file paths, but given the number of iterations needed to see the difference, I'd be surprised if this was ever worthwhile.


Given the numeric nature of the code, I'm sure that the unknowns (1, 4, 8 and 9) must do something too - I'd love to know what!  Probably some of 12-988 do something as well, so I'm sure there are some hidden gems within the interrupt command.  If you know of any, please let me know in the comments.

Undocumented feature - interrupt - part seven

Continuing from part six where we looked at "10", "11" and "12" onwards, we're now going to look at "989".  This has a number of different uses, which are used by the debugger, so should be relatively safe to use moving forwards.


989 - Application focus


When the debugger kicks in, maybe caused by a debug statement in your code, the focus is immediately set to the debugger.  Apparently this is done using the interrupt command...


interrupt(989,"APPLFOCUS")


As far as I can tell this doesn't work, maybe because I'm using Windows 7 for my testing.  When you hit a debug statement it does bring the debugger into focus, but if my Uniface application calls the interrupt command above whilst another application has focus, it does not come into focus as I would expect.





989 - Format string


According to PUUU, this is supposed to format a string by adding carriage returns in.  It's not clear on the details, but something like this...


$result = "RPL"
interrupt(989,"FORMAT")
string = $result


However, I couldn't get this to work for me, $result remained unaffected.  If you wanted to do something like this then you could use string manipulation, but I'm not sure why you'd want to.  Again, this is supposed to be used by the debugger, but I can't recall seeing an behaviour like this myself.





989 - Undo gold


When the debugger displays lists, it always uses semi-colon (;) and exclamation mark (!) as the delimiter, instead of their gold equivalents.  Apparently it uses the following command to do it...

$result = "R·;P·!L"

interrupt(989,"UNDOGOLD")

string = $result

This replaces the gold delimiters with their normal equivalents.  It does not replace the wildcard characters, only the delimiters.  This one does actually work!  It's easy to find an alternative using $replace however...

string = $replace($replace("R·;P·!L",1,"·;",";",-1),1,"·!","!",-1)

This code is probably slightly easier to read, as you can easily see which characters are included in the "undo".  How about the performance though, over 2,000,000 iterations...

• interrupt = 00:07.89, 00:07.64, 00:07.73 (almost 8 seconds)
• $replace = 00:07.95, 00:07.83, 00:07.88 (almost 8 seconds)

So as you can see, it takes pretty much exactly the same amount of time either way, so I'll definitely be sticking with the alternative, instead of using interrupt.

Summary: Numeric code "989" has multiple uses, but it seems that they either don't work any more, or are easily replaced with alternatives.  I'll do a full summary of the interrupt command in the next post.

Undocumented feature - interrupt - part six

Continuing from part five where we looked at "7", "8" and "9"...


10 - Directory listing (files)


This sets $result to be a Uniface list of all the files in the specified directory.  It also sets $status to be the count of files returned, like this...


  interrupt(10,"C:\temp")
  list = $result
  count = $status

In my test case it sets $result to be "test.txt·;test2.txt" and $status to be 2.  At last, there's a nice Uniface function to replace this one; $ldirlist or $dirlist can be used.  These actually return folders and files by default, but you can specify the "FILE" topic to limit it.  Both of these methods allow you to use wildcards.  The functions do not return the number of files found, but as they do return the list, we can easily use $itemcount if you want this as well, like this...

  list = $ldirlist("C:\temp","FILE")

  count = $itemcount(list)

This is much cleaner code, very concise and readable.  So how does it perform, I hear you cry! I tested over 200,000 iterations...

• interrupt = 00:17.10, 00:16.89, 00:16.96 (around 17 seconds)
• $ldirlist = 00:14.68, 00:14.72, 00:14.67 (almost 15 seconds)

So there's not a lot in it, but it does performs better.  I think the alternative is definitely the way to go in this case.

11 - Directory listing (folders)


This sets $result to be a Uniface list of all the folders in the specified directory, similar to "10".  It also sets $status to be the count of folders returned, like this...


  interrupt(11,"C:\temp")

  list = $result

  count = $status

In my test case it sets $result to be "test·;test2" and $status to be 2.  Similarly $ldirlist or $dirlist can be used, but specifying the "DIR" topic this time, like this...

  list = $ldirlist("C:\temp","DIR")

  count = $itemcount(list)

Let's test the performance over 200,000 iterations again...

• interrupt = 00:15.31, 00:15.30, 00:15.24 (over 15 seconds)
• $ldirlist = 00:13.33, 00:13.37, 00:13.45 (over 13 seconds)

Again there's not a lot in it, but it does performs better.  I think the alternative is definitely the way to go in this case as well.

More unknowns


Having no indication of what they did, I ran some basic tests on the codes "12" to "988" (see the next post for why I stopped there).  Mostly it was pretty uninteresting, generally speaking they set $status to -1 and leave $result unaffected.  There were some that left $status unaffected as well, and most interestingly "18" sets $result to a blank string.  Unfortunately I couldn't work out anything more for any of them, but I'd like to investigate this further.


Summary: Numeric codes "10" and "11" are useful but have been replaced by $ldirlist and $dirlist which do the same thing, but better.  All the codes beyond that, seem to be useless.  Except "989" which we will tackle in the next post.

Undocumented feature - interrupt - part five

Continuing from part four where we looked at "4", "5" and "6"...


7 - Parent directory


This sets $result to be the parent directory of the file path that you specify, like this...


  interrupt(7,"C:\temp\test\")
  parent = $result

In this case $result would be set to "C:\temp\".  You could do this with string manipulation, but you need to be careful that you check to see if the last character is a delimiter or not...

  filepath = "C:\temp\test\"

  if ( $scan(filepath,"/") < 1 ) ;calculate delimiter

    del = "\"

  else

    del = "/"

  endif

  pos = $length(filepath)

  if ( filepath[pos] = del ) ;remove last character if delimiter

    filepath = filepath[1:pos-1] 

  endif

  pos = $rscan(filepath,del) ;find last delimiter

  filepath = filepath[1,pos] ;truncate string

As you can see, this isn't very concise either, and uses $rscan as well as $scan, so it's not likely to perform very well.  As with "6", I'll test with and without the code which calculates the delimiter, over 2,000,000 iterations...

• interrupt = 00:05.16, 00:05.20, 00:05.09 (just over 5 seconds)
• alternative = 00:17.74, 00:17.52, 00:17.67 (over 17 seconds)
• without $scan = 00:15.62, 00:15.39, 00:15.41 (over 15 seconds)

Looking at these times, I'm struggling to argue for using the alternative!  I'm thinking I must be doing something wrong, so I'm going to try and refactor my alternative, without $rscan.  

I've just tried using $replace to convert the file path into a list, remove the last item (or two) and then convert it back again - this took slightly longer!  I'm going to have to give up with this for now, but I hope to tackle this again.

8 - Unknown


I haven't found any indication of what functionality this might be, but it does behave differently to "1" and "4", which are also unknown.  In this case it always returns a $status of 0 and $result is set to the string that you pass in.  I've tried all sorts of weird and wonderful things without getting anywhere, I'm giving up on this one for now too.



9 - Unknown


I haven't found any indication of what functionality this might be either.  This also behaves differently, as it always returns a $status of 0 but $result remains unaffected. 


Summary: Numeric code "7" could be very useful for finding the parent directory quickly, but "8" and "9" remain unknown.

Undocumented feature - interrupt - part four

Continuing from part three where we looked at "2" and "3"...


4 - Unknown


I haven't found any indication of what functionality this might be, but it does behave differently to "1", which was also unknown.  In this case it always returns a $status of 0 and $result is a blank string.  I tried folders, subfolders, filenames and a blank string, with no difference in behaviour.  I'm giving up, for now.



5 - Working directory


As discussed in part one, using numeric code "5" returns the working directory...

  interrupt(5,"")
  working = $result

The alternative is to use $ldir...

  working = $ldir()

The alternative is more maintainable, safer and also performs better.

6 - Concatenate file path

This takes the file path which is currently in $result and then concatenates the path you pass in, automatically adding the folder delimiters where necessary.  Here's some example code...

  filepath = "C:\temp"

  $result = filepath

  interrupt(6,"test")

  filepath = $result

This will then populate $result with "C:\temp\test\".  As you can see, I had excluded the delimiter from the end of both the original $result value and the value I passed it, but the interrupt command fixed that.  Again, this is another string manipulation example, I came up with something like this...

  filepath = "C:\temp" 

  if ( $scan(filepath,"/") < 1 ) ;calculate delimiter

    del = "\"

  else

    del = "/"

  endif

  pos = $length(filepath)

  if ( filepath[pos] != del ) ;check filepath ends with delimiter

    filepath = "%%filepath%%del%%%"

  endif

  filepath = "%%filepath%%%test%%del%%%" ;concatenate filepath

As you can see, this isn't very concise, primarily because I'm accounting for Windows and Unix environments again, meaning that I have to first calculate the "del" local variable.  You could of course use $concat instead of using substitution, like I have.


I tested these two methods over 2,000,000 iterations with the following results...

• interrupt = 00:06.67, 00:06.75, 00:06.72 (almost 7 seconds)
• alternative = 00:12.02, 00:12.00, 00:12.04 (about 12 seconds)
• without $scan = 00:09.04, 00:09.78, 00:09.74 (almost 10 seconds)

I added a third option which was the alternative but with a hardcoded delimiter instead of calculating it, so the performance can be improved by having a global register that holds this.  Still, in this case the interrupt command is better for performance and is less lines of code.  

Summary: Numeric code "4" remains unknown for now, "5" has been replaced by a better alternative and "6" seems to still be useful, although I think I'd still opt for string manipulation which I know that I can control over an undocumented feature.