Saturday, 2 May 2015

Performance of getitem

I've previously posted about how scanning is slow, and I stand by that.  But I've recently discovered that in some situations, it can be less slow than using getitem.  

I was doing a code review of some old code and found that it was using $scan in a situation where I thought you'd usually use getitem, and wondered why someone would have done it this way.  But before I replaced it, I wanted to check the performance to see what difference I would be making by changing it, and I was surprised by the results!

The code was designed to check if a variable matched one of a reasonably large number of items.  For this example, if stuck with 10 items...

  if ( temp = "ONE" | temp = "TWO" | temp = "THREE" | temp = "FOUR" | temp = "FIVE" | temp = "SIX" | temp = "SEVEN" | temp = "EIGHT" | temp = "NINE" | temp = "TEN" )
    ;testing condition
  endif

I could have used a line continuation marker, but you get the idea.

What the developer had done is replaced this set of conditions with a single $scan, like this...

  list = "|ONE|TWO|THREE|FOUR|FIVE|SIX|SEVEN|EIGHT|NINE|TEN|"
 
if ( $scan(list,"|%%temp%%%|") > 0 )
    ;testing condition
  endif

Note that this is not a Uniface list, a "bar" or "pipe" character has been used as the delimiter - this is placed at the beginning and end of the value to ensure it is not found as a sub-part of another longer value.  It This takes up a lot less space, and is perfectly readable, but I was concerned about performance.  

To test this, I wanted to make sure it was fair, so I decided to always check both the first and the last item in the list in each iteration.  I also wanted to test in a few different ways, and I came up with 4...

1) Set of conditions

  temp = "ONE"
  if ( temp = "ONE" | temp = "TWO" | temp = "THREE" | temp = "FOUR" | temp = "FIVE" | temp = "SIX" | temp = "SEVEN" | temp = "EIGHT" | temp = "NINE" | temp = "TEN" )
    ;testing condition
  endif
  temp = "TEN"
  if ( temp = "ONE" | temp = "TWO" | temp = "THREE" | temp = "FOUR" | temp = "FIVE" | temp = "SIX" | temp = "SEVEN" | temp = "EIGHT" | temp = "NINE" | temp = "TEN" )
    ;testing condition
  endif

2) $scan a bar delimited string

  list = "|ONE|TWO|THREE|FOUR|FIVE|SIX|SEVEN|EIGHT|NINE|TEN|"
 
if ( $scan(list,"|ONE|") > 0 )
    ;testing condition
  endif
  list = "|ONE|TWO|THREE|FOUR|FIVE|SIX|SEVEN|EIGHT|NINE|TEN|"
  if ( $scan(list,"|TEN|") > 0 )
    ;testing condition
  endif

3) getitem/id a Uniface list

  list = "ONE·;TWO·;THREE·;FOUR·;FIVE·;SIX·;SEVEN·;EIGHT·;NINE·;TEN"
  getitem/id temp,list,"ONE"
  if ( $status > 0 )
    ;testing condition
  endif
  list = "ONE·;TWO·;THREE·;FOUR·;FIVE·;SIX·;SEVEN·;EIGHT·;NINE·;TEN"
  getitem/id temp,list,"TEN"
  if ( $status > 0 )
    ;testing condition
  endif

4) $item a Uniface list

  list = "ONE·;TWO·;THREE·;FOUR·;FIVE·;SIX·;SEVEN·;EIGHT·;NINE·;TEN"
  if ( $item("ONE",list) != "" )
    ;testing condition
  endif
  list = "ONE·;TWO·;THREE·;FOUR·;FIVE·;SIX·;SEVEN·;EIGHT·;NINE·;TEN"
  if ( $item("TEN",list) != "" )
    ;testing condition
  endif


I wasn't really sure what I was expecting, but I thought the $scan would be the worst performance.  I tested over 2,000,000 iterations, and here's what I got...

1) Set of conditions: 37.30, 36.95, 37.83 = 37.36 secs
2) $scan a bar delimited string: 21.76, 21.97, 21.24 = 21.66 secs
3) getitem/id a Uniface list: 24.46, 24.22, 24.89 = 24.52 secs
4) $item a Uniface list: 22.65, 23.25, 23.47 = 23.12 secs

So the slowest was the set of conditions.  I didn't test it, but knowing that if statements shortcut I figure that if the item was always passing the first condition it would be quick, but because half of my test items were only passing the last condition, it would have to check each of the conditions in the set before it passed.

Although there wasn't a big difference, what surprised me is that the getitem/id and $item were actually slower than the $scan in this case.  I then remembered back to a conversation on the Uniface-L mailing list, which talked about how Uniface handles lists in the background.  The description there indicates that an array is built in the background, which means there is an upfront cost for calling getitem/id (or $item) once, but then if you're looping through it is much quicker to access the rest, because the array can be used.  However, because I'm rebuilding the list each time, that means the array needs to be rebuilt each time.  

This means that actually $scan can be used to improve the performance when checking that an item is in a list, as long as you're only checking this list once and it's not going to be re-used.  I expect there to be a point at which the number of times the list is re-used means that using getitem/id (or $item) would become better for performance.

Also, I've not tested different lengths of lists.  However, given the results and my reasoning for why the results ended up this way, I would have thought extending the list would simply emphasize the results.

Summary:  Checking if an item is in a list can be done a number of ways, and if it's being done a lot of times, performance can be eeked out by using a $scan, surprisingly!

Saturday, 11 April 2015

Modernising the client application - icons in the menu

A lot of the Uniface development that I do these days is actually web development, which means I spend a lot of time with Javascript and JQuery, and less time with Uniface procscript.  However, we still have client applications as well, and they can look tired and dated, so anything we can do to make them look more modern, can be a big win!

This is a simple little trick that appeared in Uniface 9.6.  It's actually been around for a while in dropdown menus, first as an undocumented feature, and then documented in Uniface 9.5 - I wrote a blog post on this almost three years ago.  However, the same trick now works in menus.

You can reference images in the menu items like this...


You might not be able to read the image clearly, so the three menu items are...

  1. ^U_REM_SELECT·!Cu%t
  2. ^U_SAVE·!Copy
  3. ^U_INS_SELECT·!Paste

In this case, the image is at the start, with a gold-not (or gold-exclamation-mark) as a delimiter, and the menu item text after that.

This appears in the client application this this...



As you can see - lovely icons!  Please note that you would need these glyphs in your system for it to work, these are not provided as part of Uniface itself.  

You can include icons in one of two ways...

  • ^glyph - glyphs need to be compiled into UOBJ (always available and performs well).
  • @filename - images loaded from a fileserver (not always available and performs badly).

I would recommend the glyph option though, for performance and reliability.

You may also note that the percent character (%) can be used to denote which character should work when traversing the menu using the "Alt" key.  If you do not explicitly denote this, the first character will be used.

This is documented in the "Defining and Using Menus" section of the manuals, but can be hard to find unless you know what you're looking for!  It's worth reading this for more information though.

Summary:  It's possible to make the client application look a little more modern now by including icons in the menu bars.

Thursday, 9 April 2015

Undocumented feature - current assignment file

One great feature of Uniface is it's great flexibility and configurability, through the use of different assignment (.asn) files.  These are used to define paths to the different runtime files and server locations, etc.  The problem can then come though, especially with multiple environments, that you don't know which assignment file you're using!

I wrote a post a while ago about the undocumented $assignments function, which allows you to get access to the assignment file settings that you are currently using, but this doesn't help you get to the filename itself.

One suggestion I've seen before is to add a logical to each assignment file which specifies the name/location of the file.  However, it's way too easy for someone to move, copy or rename the file and neglect to update the logical, and then worse than not knowing, you think you do know but the information you're being given is inaccurate.

Luckily Uniface 9.6.06 has solved this problem with a new undocumented feature of $processinfo...


  putmess $processinfo("cmdline")

This will put the command line used to access the application into the message buffer.  For example...


  "\\server\location\uniface\bin\uniface.exe" /asn=\\server\location\adm\uclidev.asn /ini=\\server\location\ini\me.ini APPSHELL


As you can see, the assignment file and ini file are both specified in the command line, along with the application shell as well.  This should give you all the information you need, although you'll need to do some string manipulation to pull out the bit(s) you want.

This was recently discussed on the UnifaceInfo.com forums, so you can check there for more context, and sign up if you haven't already!

Summary: You can access the current assignment file location, but accessing the command line, as long as you're happy to use an undocumented/unsupported feature.

Monday, 30 March 2015

Indirection with dollar registers

Indirection in Uniface is a wonderful thing.  You can populate a local/component/global variable with a field name, and then use indirection to get the field value of the field that the variable references.  You can also use it to set the referenced field to a value, like this...

  fld1 = "fld2"
  @fld1 = "Test"
  if ( fld2 = "Test" )
    ;this will be true
  endif

In this example, using the "@" character to indicate indirection, it's the field which is referenced by "fld1" and not "fld1" itself which is updated.

You can even do recursive indirection, which is very cool!  Not something I've had a need for myself, but very clever all the same.

However, today I was trying to populate a register the same way, using code something like this...

  reg50 = "$50"
  @reg50 = "Test"
  if ( $50 = "Test" )
    ;this will be false!
  endif

In this example, the register was still blank (or whatever the register was before hitting this code).

To be fair, this is clearly stated in the Uniface manuals...

Only fields can be referenced indirectly; the name reached by the indirect reference cannot be a variable. The following example is incorrect because the indirection refers to a general variable rather than to a field:

$1 = "$99"

@$1 = "This string does not go to $99."


But I still wanted to be able to set registers in an indirect way, so I came up with this alternative...

  putitem/id list,"$50","Test"
  getlistitems/id list
  if ( $50 = "Test" )
    ;this will be true
  endif

This works by populating a list with the register and the value, and then uses getlistitems to populate them.  The "/id" mode copies items from the associative list into one or more fields, or registers in this case.

For example, you could use a simple for loop to clear out all of the dollar registers...

  for count = 1 to 99
    putitem/id list,"$%%count%%%",""
  endfor
  getlistitems/id list

This would set each of the 99 dollar registers to blank.

Summary: Indirection works great for fields, but not for registers - however, you can use getlistitems to achieve the same outcome.

Tuesday, 25 November 2014

Syntax strings in different modes

Earlier I wrote a post about syntax strings versus literal strings, which involved me diving into the Uniface manuals to check my facts.  I talked about using $syntax to convert a literal string into a syntax string - very useful!  

However, what I didn't know before was that you can have different modes.  This is something that appears to have been added in Uniface 9.4.01, without me noticing.

There's a lovely table in the Uniface manuals to describe them, but I've tested their examples and I think there are some inaccuracies.  So I'm going to try and lay them out for you here...

Classic

This is the default behaviour, but is also represented by '%[X]'.  In this case, the usual pattern matching rules apply, with syntax codes being used as wildcards to represent patterns of characters.

  • Proc code: $syntax("D&G")
  • Syntax string: '%[X]%D&G'
  • Matches: "DOG", "DIG", "DUG", etc.

You can see the syntax string starts with the mode defined in this case, which is optional for classic mode, because this is the default.

Case Sensitive

In this mode, all characters will only match with characters of the same case.  Also, syntax codes will be treated as their literal characters, and not as wildcards.

  • Proc code: $syntax("D&G","CS")
  • Resulting syntax string: '%[CS]D%&G%[X]'
  • Matches: "D&G"

You can see the syntax string starts with the mode defined in this case, but it also ends with the exit mode, switching back to the default/classic mode.

The manuals suggest that in the proc code the mode is passed in without the double quotes, but I found this had to be a string for it to work.  They also suggest that the mode "S" can be used instead of "CS" for the same thing - I found that only "CS" works in proc code, but you can use '%[CS]' or '%[S]' in syntax strings you write yourself.


Case Insensitive

In this mode, all characters will match with characters of either upper or lower case.  Again, syntax codes will be treated as their literal characters, and not as wildcards.

  • Proc code: $syntax("D&G","CI")
  • Resulting syntax string: '%[CI]D%&G%[X]'
  • Matches: "D&G", "d&g", "D&g" and "d&G"

Again, I had to use a string to define the mode in the proc code, and only "CI" worked, but '%[CI]' and '%[I]' both worked in syntax strings I wrote myself.

NLS Locale

In this mode, all characters will match with characters of either upper or lower case, depending on the National Language Support (NLS) locale that the system is in (can be checked or set using $nlslocale).  Again, syntax codes will be treated as their literal characters, and not as wildcards.

  • Proc code: $syntax("i#B","NLS")
  • Resulting syntax string: '%[NLS]i%#B%[X]'
  • Matches: If your local is Turkish (tr_TR) then "i#B" and "İ#b", but not "I#B"

Once more, I had to use a string to define the mode in the proc code, and only "NLS" worked, but '%[NLS]' and '%[N]' both worked in syntax strings I wrote myself.

Mixing modes

You can also define a combination of modes in a single pattern, something like this...
  • Proc code: $syntax("%[CI]D%&%[CS]G%[X]")
  • Resulting syntax string: '%[CI]D%&%[CS]G%[X]'
  • Matches: "D&G" and "d&G"

There's a slight typo in the manuals with the proc code here, as they have missed the "%" from the beginning!  

Summary: You can use different modes in syntax strings, either specifying the case-sensitivity or relying on the NLS locale, but be careful if you're using $syntax because you'll lose the wildcards.

Monday, 24 November 2014

Syntax strings versus literal strings

There are a number of very important differences between syntax strings and literal strings, which I will attempt to highlight for you here.

Firstly, the Uniface definition of a syntax string...
Uniface enables you to determine if the data in a string value matches a desired pattern using syntax strings.  A syntax string is a group of characters and syntax codes enclosed in single quotation marks (').
So that's the first big difference right there, literal strings are in double quotes (") and syntax strings are in single quotes (')...

  literalString = "Literal string"
  syntaxString = 'Syntax string'

A syntax string is used for pattern matching, but they are much more simplistic than the regular expressions that you might be used to in other languages.  The syntax codes are quite straight forward...

  • # - one digit (0-9)
  • & - one letter (A-Z, a-z)
  • @ - one letter, digit or underscore (A-Z, a-z, 0-9, _)
  • ~& - one extended letter
  • ~@ - one extended letter, digit or underscore
  • ? - one ASCII character
  • A-Z - that letter, in uppercase
  • a-z - that letter, in uppercase or lowercase

On top of these are a few other syntax codes of note...

  • If you want to search for the literal version of a syntax code, eg. you want to search for the hash character (#) not a digit (0-9), then you can escape the syntax code using % before it.  Therefore, to search for a hash character it would be '%#' and to search for a percentage character it would be '%%'.
  • If you want to search for a unknown number of syntax code, eg. you want to search for 2 or more digits, then you can use * after it.  Therefore, to search for 2 or more digits it would be '###*' - in this case the first two hash characters represent one digit each, and the third hash character is part of the the syntax code '#*', which means zero or more (0-n) digits.
  • If you want part of the pattern to be optional (to match the pattern or blank) then you can put rounded brackets around it, using ( and ).  For example, '##(#)' would match with 2 or 3 digits (but not more than 3).
  • Any other character is treated literally as that character.  

Uniface also gives a handy function that can be used to convert a literal string into a syntax string, sensibly named $syntax...

  syntaxString = $syntax("Literal string")

This can be especially useful if you're storing the pattern in the database, or some other configuration.

Here are a few examples from the Uniface manuals...

Proc with Syntax String
Result
if ('#' = "123")
if ('#*' = "123")
FALSE
TRUE
if ('#*' = vValue)
if ('#*' = "%%vValue")
TRUE
TRUE
if ('&###' = "1234")
if ('@###' = "1234")
FALSE
TRUE
if ('?' = "A")
if ('??' = "A")
if ('??*' = "A")
if ('?' = "ABC")
if ('??*' = "ABC")
TRUE
FALSE
TRUE
FALSE
TRUE
if ('(#(-))&&&' = "ABC")
if ('(#(-))&&&' = "1ABC")
if ('(#(-))&&&' = "1-ABC")
if ('(#(-))&&&' = "12ABC")
TRUE
TRUE
TRUE
FALSE


I've had (occasionally heated!) discussions with developers before, when they have said that...

  if ( myVar = "Y" )

...and...

  if ( myVar = 'Y' )

...are interchangable.  

Yes, I can see that the pattern 'Y' only matches with the string "Y" and nothing else, but they are entirely different ideas, and should not be used interchangeably.  If you want to only match with "Y", then use the literal string "Y", because that's what you mean.  

In my experience, developers who say that these are interchangable are developers who don't understand what the differences are.

Summary: Syntax strings are very useful for pattern matching, but very different than literal strings.  Know the difference, and know when to use which.

Thursday, 23 October 2014

Fetching web content

I've been working on fetching web content recently, mostly for building authentication workflows, using the likes of Facebook and Google to act as authentication services.  I'm going to go into detail about how I've done those, not in this post at least, but more simply how I made the requests from my Uniface service form, out to those external services. 

There are a number of different HTTP methods.  Generally speaking, when you are retrieving data only then the method is "GET", whereas if you are sending data that may perform an action as well then the method is "POST".  There are others, but I'm going to stick with these two for now, as they are more commonly used.

In fact, I'm going to start with "GET" first.  

The first thing you need to do is consider the character set.  Your system may be set in different character sets, but you need to make sure the character set of your request matches the service you are calling, in my case it was UTF-8...

  vCharSet = $sys_charset ;backup current setting for later
  $sys_charset "UTF8"   ;set to the character set we need

We then want to create a new instance of the "UHTTP" component - this is the Uniface component that is going to do most of the hard work for us...

  newinstance "UHTTP",vHandle
  if $status < 0 & $procerror < 0 )
    $sys_charset = vCharSet ;restore character set
    return -101             ;error handling!
  endif

Once you've got your instance, the next step is to define how the component responds to mismatched or expired certificates, and how it calculates the content length.  By default it will error if there is a certificate error and you have to manually calculate the content length yourself.  It's a binary switch, so to switch them all off we do the following...

  activate vHandle.SET_FLAGS(7)
  if $status < 0 & $procerror < 0 )
    $sys_charset = vCharSet ;restore character set
    return -102             ;error handling!
  endif

We're now ready to make the request.  In this example, I'm just going to grab the Google homepage...

  activate vHandle.SEND("http://www.google.co.uk","GET","","","",vContent,vResponse)
  if $status < 0 & $procerror < 0 )
    $sys_charset = vCharSet ;restore character set
    return -103             ;error handling!
  endif

The parameters are:

  1. URL (string : in) - the URL you're sending the request to.
  2. Method (string : in) - the method being used (this is not checked by "UHTTP").
  3. Username (string : in) - if you're using a secure URL, you may need to populate this.
  4. Password (string : in) - again, you may need to populate this.
  5. Headers (string : inout) - the HTTP request headers in and the response headers out.
  6. Content (string : inout) - in the case of a "GET" this is out only and the page contents.
  7. Response (string : out) - the HTTP response headers.
The $status should be set to 200 for a successful response (to match the HTTP status code for success) or it may be set to 1.  If it is set to 1 then this means that the content was larger than the parameter limit (10Mb) and therefore you will need to get the rest of the content from the buffer, like this...


  while $status = 1 )
    activate vHandle.READ_CONTENT(vExtra)
    if $status < 0 & $procerror < 0 )
      $sys_charset = vCharSet ;restore character set
      return -104             ;error handling!
    endif
    vContent = "%%vContent%%vExtra%%%"
  endwhile


You now have the full page contents, but don't forget to restore the character set...

  $sys_charset = vCharSet ;restore character set

Doing a "POST" is very similar, except for a couple of differences:
  1. To replicate a web browser "POST", which you are usually doing, you need to make sure the 5th parameter of the SEND call is populated with the following header.. "Content-Type=application/x-www-form-urlencoded"
  2. The 6th parameter of the SEND call needs to be populated with the data that you are sending in your request.  If this is larger than 10Mb then you will need to have a WRITE_CONTENT loop (similar to the READ_CONTENT loop) before the SEND, in order to populate the buffer with the full contents.
This all works rather well.  However, I've found two major limitations along the way.

Firstly, it was not possible to send a request to a URL which had a colon (:) character after the protocol.  For example, when trying to get a user's LinkedIn ID you would use the following URL...

  "https://api.linkedin.com/v1/people/~:(id)?oauth2_access_token=%%vToken%%%"

The colon (:) character after "/people/~" broke the URL.  Luckily I'm using the past tense here, this was fixed in patch X504 (for Uniface 9.6.05).  This works fine now we've installed the patch.

Secondly, the observant of you may have noticed the 6th parameter of the SEND call is defined as a string.  This is great for grabbing HTML page source from a website, or even doing most webservice calls, as these tend to return either XML and JSON strings of data.  However, it makes creating something like a Dropbox interface impossible, or at least very limited.  You can grab text files, but nothing else!  No images, no Office documents, no PDFs.

Unfortunately there's no word from Uniface on when this second issue might be resolved.

Summary: It's possible to use the "UHTTP" component to request text data, either from a web page or a webservice.  Just don't expect it to work with binary files, yet!