Fetching web content

I've been working on fetching web content recently, mostly for building authentication workflows, using the likes of Facebook and Google to act as authentication services.  I'm going to go into detail about how I've done those, not in this post at least, but more simply how I made the requests from my Uniface service form, out to those external services. 

There are a number of different HTTP methods.  Generally speaking, when you are retrieving data only then the method is "GET", whereas if you are sending data that may perform an action as well then the method is "POST".  There are others, but I'm going to stick with these two for now, as they are more commonly used.

In fact, I'm going to start with "GET" first.  

The first thing you need to do is consider the character set.  Your system may be set in different character sets, but you need to make sure the character set of your request matches the service you are calling, in my case it was UTF-8...

  vCharSet = $sys_charset ;backup current setting for later

  $sys_charset = "UTF8"   ;set to the character set we need

We then want to create a new instance of the "UHTTP" component - this is the Uniface component that is going to do most of the hard work for us...

  newinstance "UHTTP",vHandle

  if ( $status < 0 & $procerror < 0 )

    $sys_charset = vCharSet ;restore character set

    return -101             ;error handling!

  endif

Once you've got your instance, the next step is to define how the component responds to mismatched or expired certificates, and how it calculates the content length.  By default it will error if there is a certificate error and you have to manually calculate the content length yourself.  It's a binary switch, so to switch them all off we do the following...

  activate vHandle.SET_FLAGS(7)

  if ( $status < 0 & $procerror < 0 )

    $sys_charset = vCharSet ;restore character set

    return -102             ;error handling!

  endif

We're now ready to make the request.  In this example, I'm just going to grab the Google homepage...

  activate vHandle.SEND("http://www.google.co.uk","GET","","","",vContent,vResponse)

  if ( $status < 0 & $procerror < 0 )

    $sys_charset = vCharSet ;restore character set

    return -103             ;error handling!

  endif

The parameters are:

1. URL (string : in) - the URL you're sending the request to.
2. Method (string : in) - the method being used (this is not checked by "UHTTP").
3. Username (string : in) - if you're using a secure URL, you may need to populate this.
4. Password (string : in) - again, you may need to populate this.
5. Headers (string : inout) - the HTTP request headers in and the response headers out.
6. Content (string : inout) - in the case of a "GET" this is out only and the page contents.
7. Response (string : out) - the HTTP response headers.

The $status should be set to 200 for a successful response (to match the HTTP status code for success) or it may be set to 1.  If it is set to 1 then this means that the content was larger than the parameter limit (10Mb) and therefore you will need to get the rest of the content from the buffer, like this...

  while ( $status = 1 )

    activate vHandle.READ_CONTENT(vExtra)

    if ( $status < 0 & $procerror < 0 )

      $sys_charset = vCharSet ;restore character set

      return -104             ;error handling!

    endif

    vContent = "%%vContent%%vExtra%%%"

  endwhile

You now have the full page contents, but don't forget to restore the character set...

  $sys_charset = vCharSet ;restore character set

Doing a "POST" is very similar, except for a couple of differences:

1. To replicate a web browser "POST", which you are usually doing, you need to make sure the 5th parameter of the SEND call is populated with the following header.. "Content-Type=application/x-www-form-urlencoded"
2. The 6th parameter of the SEND call needs to be populated with the data that you are sending in your request.  If this is larger than 10Mb then you will need to have a WRITE_CONTENT loop (similar to the READ_CONTENT loop) before the SEND, in order to populate the buffer with the full contents.

This all works rather well.  However, I've found two major limitations along the way.

Firstly, it was not possible to send a request to a URL which had a colon (:) character after the protocol.  For example, when trying to get a user's LinkedIn ID you would use the following URL...

  "https://api.linkedin.com/v1/people/~:(id)?oauth2_access_token=%%vToken%%%"

The colon (:) character after "/people/~" broke the URL.  Luckily I'm using the past tense here, this was fixed in patch X504 (for Uniface 9.6.05).  This works fine now we've installed the patch.

Secondly, the observant of you may have noticed the 6th parameter of the SEND call is defined as a string.  This is great for grabbing HTML page source from a website, or even doing most webservice calls, as these tend to return either XML and JSON strings of data.  However, it makes creating something like a Dropbox interface impossible, or at least very limited.  You can grab text files, but nothing else!  No images, no Office documents, no PDFs.

Unfortunately there's no word from Uniface on when this second issue might be resolved.

Summary: It's possible to use the "UHTTP" component to request text data, either from a web page or a webservice.  Just don't expect it to work with binary files, yet!

What are handles and should I be using them?

The Uniface manuals describes a handle as...

A handle is a reference to an object, such as a component instance, entity, occurrence, or field. Handles can be used to call operations on the referenced object, or to pass these objects as variables and parameters.

If you're used to pointers, you may at first glance think that these are the same, but they're not.  A pointer contains the address of an item in memory and is therefore fixed, whereas a handle is an abstract reference, which means the address can be changed without breaking the reference.  This makes them very useful.

In Uniface you can have "public" or "partner" handles, which allow you to define which types of operations can access them.  They are also datatypes available for local, component and global variables, depending on the scope required.  I'm not going to discuss either of these points in detail, but I think it's handy to know.

As the description in the manual states, you can create a handle that references most Uniface objects, and there are different methods for creating these handles:  Component instances ($instancehandle), entity collections ($colhandle), entity occurrences ($occhandle), fields ($fieldhandle) and also OCX controls ($ocxhandle).  

I'm going to focus on component instances.  More specifically, I'm going to focus on creating new component instances, using newinstance and handles.  

As you might expect, creating a new instance of a form has a little lead time.  Using a handle doesn't stop this lead time, but what it does do, is mean that it only has to happen once.  This means that using a handle for a single form activation is not going to give you any advantage (unless you just love handles!) but when calling operations on the same form multiple times, handles could give you a performance improvement.

This is something like how you could activate a form...

  activate "TEST_FORM".test_oper("Test")

In fact this does an implicit newinstance as well (if it can't already find an instance with the same name as the form) and so explicitly your code could look like this...

  newinstance "TEST_FORM","TEST_FORM"

  activate "TEST_FORM".test_oper("Test")

If you were to replace this with a handle then you may use something like this...

  newinstance "TEST_FORM",theHandle

  theHandle->test_oper("Test")

Notice that in this case we have used a handle instead of an instance name in the newinstance call, and then we can use this handle in place of the activate command.  The -> is referred to by Uniface as the "dereference" operator, and is processed with a precedence of 2 (with only field indirection being higher).

So let's try these two different ways of activating an operation (in this case a very simple operation that simply returns out immediately).  The newinstance command is done before the loop, and the activate (or equivalent handle call) is done inside the loop, 200,000 times...

• Using activate: 2.56, 2.61, 2.60 (about 2.6 seconds)
• Using a handle: 2.39, 2.35, 2.39 (about 2.4 seconds)

As you can see, using a handle is marginally quicker, although you'd have to be making a lot of calls before you really noticed the difference.  

I thought that this would be even more useful when the handle references a component instance that is running on another server, such as when a form is mapped to run elsewhere, as the I thought the initial creation of the new instance would take longer here.  So I ran the same operation 20,000 times (after mapping the form)...

• Using activate: 13.39, 14.35, 12.99 (about 13.5 seconds)
• Using a handle: 13.56, 13.12, 12.91 (about 13.0 seconds)

Not much difference here either.  Having said that, maybe on a chatty network the difference would be greater.

Please note:  You must return out of the operation that you call, if you exit out then this will destroy the handle and future calls to the operation will fail with a $status of -1.  To clean up a handle you can either set it to 0 or "" (empty string). 


Summary: Handles are abstract references to pretty much any Uniface object.  They can be useful, especially if you want to call the same object multiple times, but they don't seem to have the performance benefits that I was expecting, not when it comes to component instances anyway.

Reserved operation names

This is something which is well documented by Compuware, but which I always forget, and it always drives me a little insane trying to figure out what's going on!  


Uniface has some reserved (or "predefined") operation names...

• init - executed when a component instance is created (newinstance or activate).
• cleanup - executed when a component instance is removed (deleteinstance or exit or completion of <accept> or <quit>).
• attach - executed when a contained DSP is attached to a parent DSP.
• detach - executed when a child DSP is detached from its parent.
• exec - executed when a component is activated (activate or run).
• accept - executed when an "accept" request is invoked on the current component (<accept>).
• quit - executed when a "quit" request is invoked on the current component (<quit>).

EDIT:  The following operation names have been reserved for future use...

• abort
• complete

This means that you cannot have an operation defined with this name in the operations (<oper>) trigger, unless you are using it for this purpose specifically.  If you do then this operation will be fired when you call it, and in the above situation as well.  

I had an AJAX operation on a SVC called "init" which I simply could not figure out why it was firing twice.  In this case, it was firing once as the form was being activated (by calling the operation) and then again because I had called the operation.  This baffled me for some time!

Summary: Do not accidently use reserved operation names for your operations, or you may get some unexpected behaviour!