URL Transactions and Formatting
This section describes how SenseTalk can be used to access resources on the Internet through their URLs. Through this mechanism, SenseTalk can access the contents of files and websites across the Internet and communicate with remote servers using standard protocols.
A number of supporting functions are available to make it easy to convert information between standard URL formats.
Referring to URL Resources in a Script
To refer to a file or other Internet resource accessed through a URL in a script, use the word url followed by an expression that evaluates to the URL of the file or resource.
The urlString may be a “file:”, “http:”, or “ftp:” type of URL. URLs of type “file:” can be treated as containers — you can store into or change their contents, provided you have write access to the file. If a urlPropertyList is used, it may contain “scheme”, “host”, “path”, “query” and other keys as defined by the makeURL() function.
When retrieving an http resource from the Internet using this syntax, an http GET operation is used. The urlString or urlPropertyList may include query data to simulate submitting a form. If you use a string, the makeURL() function can be used to easily construct a string containing a properly formatted query.
If with headers headerPropertyList is specified, headerPropertyList should be a property list containing any custom HTTP headers to be used in accessing that URL. The custom header information is passed along with any standard headers for both GET (when accessing a URL) and POST (when using the post command) operations.
After accessing a URL, you may check the status of the remote operation by calling the result() function on the next line of the script. This returns the status value of the request if it is less than the minimum level treated as an error (as specified by the URLErrorLevel global property) or an exception if the status is at that level or greater and the throwExceptionResults is set to false.
The data retrieved from the URL is interpreted as text according to the current setting of the defaultStringEncoding global property. Set this property to the correct encoding (often "UTF8") before fetching the URL data.
Syntax:
url urlString {with headers headerPropertyList}
url urlPropertyList {with headers headerPropertyList}
Example:
put url "http://www.somewhere.com/somepage" into htmlContents
put htmlContents into url "file://localhost/localCopy"
Example:
put {scheme:"http", host:"www.apple.com"} into applePage
Configuring URL Behavior
SenseTalk includes several global properties you can use to affect the behavior of URLs that you access from your scripts.
the URLCacheEnabled: controls whether the contents retrieved from a URL resource are cached.the URLTimeout: specifies the maximum time allowed before a URL request times out.the URLErrorLevel: sets the lowest URL status value that is treated as an error when fetching the contents of a URL.
These global properties are each described in detail on Global Properties for Working with URLs and Sockets.
Internet and URL Commands and Functions
SenseTalk provides several commands and functions for performing transactions over the Internet and manipulating URLs and data in standard formats.
The post command posts data to a URL and retrieves the results. The makeURL() function helps in constructing a properly-formed URL string from a given set of components, and its companion extractURL() function extracts the components from a URL string. The URLEncode and URLDecode functions convert text to and from the special encoded format commonly used in URL queries and data passed across the network.
Post Command
Behavior: Use the post command to conduct a transaction with an online web-based service, simulate the posting of a web form, or similar interactions with a web server. Results returned by the web server are stored in the variable it.
The data can be any text string, or it may be a property list, in which case it is formatted automatically in the standard query format. The post command waits up to the maximum time specified by the URLTimeout global property to receive a response back from the web server before proceeding with the next line of the script. Whatever response is received is stored in the variable it. If a response is not received within the allotted time or some other error occurs, it is empty, and the result is set to an exception object indicating the error.
Syntax:
post data to url theURL
Example:
post queryData to url "http://someservice.net" with headers {"Content-Type": "text/xml"}
Example:
post URLEncode(dataToPost) to url makeURL(urlComponents)
put it into postResults
Post Multipart Command
Behavior: Posts multipart data to a server. The contentToSend can be a single text value or list of values to send including the contents of one or more files, or may be a property list or list of property lists to provide more control over the format in which the data is transmitted or supply additional information about the contents. Specifying "as data" causes the file contents to be encoded in base64. Including the filename is optional, but recommended when sending a file.
Syntax:
post multipart contentToSend to URL serverURL
Example:
post multipart {filename:"/tmp/xyz", body:file "/tmp/xyz" as data} to URL serverURL
Tech Talk
The full set of rules for the content being posted is as follows:
- If the content value is a list, then each item in the list is treated as a separate part of the multipart content, with the rules below being applied to each part (the individual parts may not be lists, and subparts are not supported).
- If the content value is a binary data value (typically achieved by using the "as data" operator), the body will automatically be encoded in base64, with appropriate Content-Type and Content-Transfer-Encoding header values supplied.
- If the content value is not binary data and is not a property list, the string value of the content will be used as the body, with the "Content-Type: text/plain" header supplied.
- If the content value is a property list, it provides both the body content plus optional header information. The following properties have special meaning:
- Body: Required. Its value is the body content of this part, as either a data or text string value, or – at the top level only – as a list of content values, as described by these rules.
- Header: A property list defining additional header fields. This is only required if headers with the same name as other keys in the content property list are needed. Otherwise, header key/value pairs can be included directly in the content property list.
- Disposition: Used in forming the "Content-Disposition" header. Suitable values are "inline", "attachment", and so forth. If not given, "form-data" will be used.
- FieldName: Used in forming the "Content-Disposition" header. If a form field name is not given, a name such as "field1", "field2", etc. will be generated automatically.
- FileName: Used in forming the "Content-Disposition" header. If given, this is included as part of the Content-Disposition header. It is recommended that a FileName be included when uploading a file.
Some special rules apply if a property list is used at the top level (not as a part within a list). The Body at this level may be a list. Also, the following additional properties are recognized:
- MultipartBoundary: A text string used to separate the parts (if not given, a suitable string will be generated automatically). If a custom "Content-Type" header is supplied at the top level, it should include a boundary string, and MultipartBoundary must be included with that same boundary string as its value.
- TopHeader: A property list defining additional header fields at the top level of the message. Alternatively, top level header fields can be given with the URL being posted to by specifying with headers headerFields after the URL. If a "MIME-Version" property is not given in TopHeader, it will automatically be included with a value of "1.0". If a "Content-Type" property is not included, one will automatically be generated which specifies a content type of "multipart/form-data" and a boundary string as defined by the MultipartBoundary property (or generated automatically it none is specified).
Any additional properties in each part that don't have special meaning are treated as header fields for that part of the message. The following header fields are generated automatically if they are not included in either the content property list or in its Header property list:
- If a "Content-Disposition" property is not included, a Content-Disposition header is generated automatically using the values supplied for Disposition, FieldName, and FileName or their defaults, as described above for each of these properties.
- If a "Content-Type" property is not included, a Content-Type header is generated automatically.
MultipartMIME Function
Behavior: Produces a property list containing Body and Header information for the given content, as it would be sent by the post multipart command. This can be useful for troubleshooting a problem or for constructing an HTTP message as text. This function can be called with a single value, a list of values, or a property list or list of property lists. See the Post Multipart command for full details.
The value returned by the multipartMIME function is a property list with a Body property whose value is a fully-formatted string in multipart-MIME format. The Header property is a property list containing appropriate headers to accompany the Body text.
Example:
put (multipartMIME("Secret Message" as data) adding {request:"/"}) as http
JSONFormat Function
Behavior: Generates text in JSON format for any SenseTalk value. Use the JSONformat function to generate text in JSON format for use with another program, or for sending data to a web service or other processes that use JSON format. valueToFormat can be any SenseTalk value, including a list or a property list. Values other than lists, property lists, text, and numbers (such as a date or a color) are converted to text before formatting.
For the most part, values in SenseTalk will be mapped automatically to the corresponding type in the formatted output. A SenseTalk list will become a JSON list, a SenseTalk property list will become a JSON object, and so forth. Occasionally, though, you may need to specify “as number”, “as string” (or “as text”), or “as boolean" for individual elements to ensure you get the desired result. This is needed because SenseTalk doesn’t ordinarily make a distinction between string and numeric values — it simply converts values as needed depending on the context or operation. When formatting as JSON, however, SenseTalk has no way to know whether a string or a number is required.
Syntax:
JSONFormat( valueToFormat )
{the} JSONFormat of valueToFormat
Example:
put contentValue's JSONFormat into file outfile
Examples:
This example shows a SenseTalk call using the JSONFormat function to return a JSON string with the values listed:
set listOfValues to ["a",Yes,"0500",22]
put JSONFormat(listOfValues)
The JSON output from the above example looks like this:
["a",true,"0500",22]
In order to have "Yes" as text, 0500 as a number, and 22 as a string, you can specify that in your code like this:
set listOfValues to ["a", Yes as text, "0500" as a number, 22 as a string]
put JSONFormat(listOfValues)
The output from the above now looks like this:
["a","Yes",500,"22"]
JSONValue Function
Behavior: Use the JSONValue function for converting data in JSON format to a SenseTalk value for use within your script.
Syntax:
JSONValue( JSONstringToEvaluate )
{the} JSONValue of JSONstringToEvaluate
The JSONstringToEvaluate must be a JSON formatted string.
Example:
put JSONValue(file "/tmp/myJSON.txt") into fileContents
Example:
In this example, a JSON file on your system contains information about Denver, Colorado, in the United States:
{
"city" : "Denver",
"airports" : ["Denver International", "Rocky Mountain Metropolitan", "Centennial", "Front Range"],
"elevation" : 5280,
"coordinates" : [39.7392, 104.9903],
"locatedInMountains" : false,
"stateCapital" : true
}
The following SenseTalk uses the JSONValue function to access the JSON file's contents:
put file "/Users/elizabethsimons/Desktop/DenverInfo.json" into fileContents
put JSONValue(fileContents) into DenverInfo
put DenverInfo
When the above is run, the output looks like this:
{airports:["Denver International","Rocky Mountain Metropolitan","Centennial","Front Range"], city:"Denver", coordinates:[39.7392,104.9903], elevation:5280, locatedInMountains:"False", stateCapital:"True"}
MakeURL Function
Behavior: Use the makeURL function to create a properly-formatted URL string using a property list containing components of the desired URL. The makeURL function automatically encodes each part of the URL as appropriate. This function is called implicitly whenever a URL expression with a property list instead of a string is used.
Syntax:
makeURL( componentPropertyList )
{the} makeURL of componentPropertyList
Example:
get makeURL(scheme:"http", host:"myserver.net", path:"slinky")
Example:
put url makeURL(host:"google.com", path:"search", query:{"q":"EggPlant", "ie":"UTF-8"}) into searchResults
The componentPropertyList should be a property list containing the components that will be used to construct the resulting URL string. Any of the following properties may be specified; other properties will be ignored.
| scheme | the scheme or protocol, such as “http”, “ftp”, or “file” (if not specified, http will be assumed) |
| host | the host machine (web server) the URL refers to |
| path | the path to the resource on the host web server |
| port | the port number to connect to |
| user | the user name to log in on the server (used sometimes for ftp URLs) |
| password | the password used to log in on the server (used along with user for some ftp URLs) |
| parameters | additional parameters to include (such as a “type” parameter for an ftp URL) |
| fragment | the fragment identifies a particular anchor point within a resource |
| query | a property list containing keys and values to be passed to the web server, such as would be supplied by a form on a web page |
Related:
ExtractURL Function
Use the extractURL function to easily separate a URL string into a property list containing its component parts. The extractURL function will automatically decode each part of the URL as appropriate.
Syntax:
extractURL( urlString )
{the} extractURL of urlString
Example:
put extractURL("http://myserver.net/slinky") into urlParts
The urlString should be an expression that evaluates to a string in the standard URL format. The value returned by the extractURL function will be a property list containing the component parts that were extracted from that URL string. If urlString is empty, the returned property list will be empty, otherwise it will contain one or more of the following keys, depending on what the URL string contained:
| scheme | the scheme or protocol, such as “http”, “ftp”, or “file” |
| host | the host machine (web server) the URL refers to |
| path | the path to the resource on the host web server |
| port | the port number to connect to |
| user | the user name to log in on the server (used sometimes for ftp URLs) |
| password | the password used to log in on the server (used along with user for some ftp URLs) |
| parameters | additional parameters to include (such as a “type” parameter for an ftp URL) |
| fragment | the fragment identifies a particular anchor point within a resource |
| query | the value of this key will be a property list containing keys and values present in the urlString that would be passed to the web server, such as would be supplied by a form on a web page |
Related:
URLEncode Function
Behavior: The URLEncode function replaces each space in string with a ‘+’ character, and each non-alphanumeric character other than space with a three-character code of the form ‘%xx’, where xx is a two-digit hexadecimal number representing the replaced character’s ASCII value as used in URL strings. If the parameter is a property list, it is encoded as a URL query string in the same way that a query is encoded by the makeURL function. Use the URLEncode function to encode a string in a manner suitable for passing data as part of a URL string used to get an Internet resource.
Syntax:
URLEncode( string )
{the} URLEncode of string
Example:
put urlEncode(myData) into encodedData
Related:
URLDecode Function
Behavior: The URLDecode function replaces each triplet from encodedString of the form ‘%xx’ (where xx is a two-digit hexadecimal number) with the character having that ASCII value. Each ‘+’ character in encodedString is replaced by a single space character. Use the URLDecode function to decode data from a URL string. This is the inverse of the urlEncode function.
Syntax:
URLDecode( encodedString )
{the} URLDecode of encodedString
Example:
put urlDecode(encodedData) into myData
Related:
XMLRPCFormat Function
Behavior: Generates text in XML-RPC format for any SenseTalk value. Returns a string representing any value in the format required for use within the body of an XML-RPC message. Use the XMLRPCFormat function to generate text in XML-RPC format for use with another program, or for sending data to a web service or other processes that use XML-RPC format.
For the most part, values in SenseTalk will be mapped automatically to the corresponding type in the formatted output. A SenseTalk list will become a XML-RPC array, a SenseTalk property list will become a XML-RPC struct, and so forth. Occasionally, though, you may need to specify “as number”, “as string” (or “as text”), "as integer", "as a number", or “as boolean" for individual elements to ensure you get the desired result. This is needed because SenseTalk doesn’t ordinarily make a distinction between string and numeric values — it simply converts values as needed depending on the context or operation.
Syntax:
XMLRPCFormat( valueToFormat )
{the} XMLRPCFormat of valueToFormat
The valueToFormat can be any SenseTalk value, including lists, property lists, text, numbers, dates, and binary data. Other values will be converted to text before formatting.
Example:
put contentValue's XMLRPCFormat into file dataFile
Examples:
This example shows a SenseTalk call using the XMLRPCFormat function to return an XML-RPC string with the values listed:
set listOfValues to ["a",Yes,"0500",22]
put XMLRPCFormat(listOfValues)
The output from the above example looks like this:
<value><array><data>
<value><string>a</string></value>
<value><boolean>1</boolean></value>
<value><string>0500</string></value>
<value><int>22</int></value>
</data></array></value>
In order to have "Yes" as text, 0500 as an integer, and 22 as a number, you can specify that in your code like this:
set listOfValues to ["a", Yes as text, "0500" as integer, 22 as a number]
put XMLRPCFormat(listOfValues)
The output from the above now looks like this:
<value><array><data>
<value><string>a</string></value>
<value><string>Yes</string></value>
<value><int>500</int></value>
<value><double>22</double></value>
</data></array></value>
XMLRPCValue Function
Behavior: Use the XMLRPCValue function for converting data in XML-RPC format to a SenseTalk value for use within your script. The XMLRPCstringToEvaluate must be an XML-RPC formatted string.
Syntax:
XMLRPCValue( XMLRPCstringToEvaluate )
{the} XMLRPCValue of XMLRPCstringToEvaluate
Example:
put XMLRPCValue(file "/tmp/myXMLRPC.txt") into dataReceived