Technical debt is becoming an increasingly intractable problem for many organizations. Businesses that grew up fast because the only way to survive was to be first to market are now suffering the legacy of unbounded rapid development and workforce churn. There’s a raft of developers and operations folk out there tasked with maintaining a heap of spaghetti code they have little chance of understanding. This lack of comprehension leads to timidity when it comes to modifying or refactoring old codebases for fear of breaking something, especially in production systems which are often the only places the code still exists.
This blog explores the benefits to using Tcl to solve your tech debt plus gives you a practical example of wrapping an existing API with Tcl to quickly and easily solve your tech debt.Already convinced on the merits of using Tcl for your tech debt?You can try out our own version of Tcl for free or read more on our Tcl builds, ActiveTcl, and how you can get exactly the Tcl distro you need with our ActiveTcl.
Two Expensive Choices, 1 Solution
Organizations are often left with two unpleasant choices when dealing with legacy code:
- Hope the code keeps working forever with increasing cost of maintenance
- Refactor whole systems
Both choices are expensive, and the longer the code stays in production the more expensive each choice becomes. But it can be extremely difficult for a manager to justify rewriting a working system from scratch simply because they don’t understand how it works. Plus, the current manager’s tenure with the older system may be over before the legacy system breaks and they are blamed for the break. Consequently, legacy systems endure until they inevitably break or give way to superior competition.
Thankfully, there’s a solution to alleviate much of the cost associated with technical debt by integrating a scripting language. This solution is applicable for a large proportion of legacy codebases. Integrating a scripting language extends the usability of the legacy codebases with minimal risk and allows refactoring to occur at a steady pace. There are a bewildering array of scripting languages available. However, there is one language in particular designed for the task: Tcl.
Tcl was designed from the ground up to be embeddable. It is almost trivial to wrap an existing legacy API into commands you can call directly from scripts which opens up a whole new avenue of processing that would previously have been impossible. You can also embed an entire Tcl interpreter inside an existing application with a few lines of code that can open up extension points in that application allowing it to be trivially customised. This latter approach allows software sold to customers to be tailored specifically for that customer either as a professional service exercise or by the customer themselves.
4 Key Benefits to Tcl
Tcl has four desirable properties for this sort of work: size, design, speed and simplicity. Firstly, Tcl scripting code is small. An interpreter can add less than 200Kb to the memory footprint of an application, and the core Tcl library takes up around 2MB on disk.
Secondly, Tcl has a well designed C API. So it’s ideally suited to extend or embed in applications written in C or C++ and makes it simple to shift functions from C to Tcl for flexibility or Tcl to C for performance.
Thirdly, Tcl is fast. The tight coupling with C means that most inbuilt Tcl commands are fast enough as it is and when you need it to go faster, reimplementing Tcl routines in C and exposing them to Tcl with the same interface is simple.
Finally, Tcl is simple. Simple enough that there are a lot of folk out there who can write Tcl but don’t call themselves programmers, simple enough that people who use an application can be taught to extend it for their own purposes given access to Tcl.
Example: Use Existing APIs for Scripting
I’ll now show you how with a couple hundred lines of C and a few lines of Tcl, we’ll easily solve tech debt with a legacy API.
Here’s a simple demonstration of how easy it can be to instrument an existing API for scripting. There’s some sample code at https://github.com/shaunmlowry/tcl-tech-debt-blog-example which implements a trivial order processing system with a few deficiencies, and a scripting wrapper that extends it and can overcome some of these deficiencies.
The order processing system is split into 2 parts: a C library which implements the API and a C program which interacts with a user via the terminal, using the library to process orders. Firstly, the API. The API is declared in order.h:
#ifndef _ORDER_H
#define _ORDER_H 1
typedef struct _order_item {
char *item; // item name
double weight; // how much each weighs in kilos
int quantity; // how many
struct _order_item *next; // ptr to next item
} order_item;
typedef struct _order {
double total_weight; // weight of order in kilos
double shipping_cost; // total calculated shipping cost in dollars
order_item *items; // linked list of items
} order;
extern order *create_order();
extern order *add_to_order(order *my_order, char *name, int quantity, double weight);
extern double calculate_shipping_cost(double weight, int quantity);
extern void print_order(order *my_order);
#endif
The first structure is a typical linked-list of items including their properties and the second describes the order as a whole, including the item list. We have a number of functions that make up the API:
create_orderwhich creates a new, empty orderadd_to_orderwhich adds an item to an ordercalculate_shipping_costwhich given a weight in kilos and a quantity will tell you the shipping cost for an itemprint_orderwhich prints out the contents of an order, used here as a surrogate for a function which would submit the order to separate billing and shipping systems
The API is implemented in order.c which we’ll examine portions of later.
The command line program is implemented in process_order.c and just keeps requesting and adding items to an order until the user enters a blank line, at which point it calls print_order to show the full content of the order.
The API is pretty dumb. At some point way back when, the courier used by this fictitious company would charge by weight per package that was included in an order so the business logic was hard coded in. The API calculates a shipping cost (based on bands of weight) for each item entered and adds that to the running total. Fast forward to now, and someone in shipping has earned a gold star by negotiating a single charge for an order by weight on the basis that the entire order will be shipped in one package.
Unfortunately our API can’t cope with this and we have a couple of maintenance problems: i) Jerry, who wrote the original code, no longer works for the company; ii) the source code in order.c is lost and gone forever. Fortunately, Jerry left us a compiled library for the API and a header file for it (order.h) so we could make a new client program if we wanted.
Now, with a copy of ActiveTcl (ActiveState’s Tcl distro, ready to go out of the box) we can wrap the API and script our way around the problem. As our API is fairly simple, we can create a Tcl wrapper for it by hand. For more complex APIs, I suggest using a tool like SWIG (http://www.swig.org/) to automatically generate the Tcl wrappers.
Get our ActiveTcl data sheet and start solving tech debt today
Firstly, we create a new file for our API wrapping extension called order_tcl.c. This will contain some C code which exposes our API to Tcl as native commands. Tcl will treat them the same as any other command in the language and they’ll slot seamlessly into the syntax.
We’ll include tcl.h to get at the Tcl C library functions, string.h for some basic C string manipulation, and order.h to get at our API:
#include tcl.h
#include string.h
#include "order.h"
The first function we need to define is one that initializes our extension:
int Ordertcl_Init(Tcl_Interp *interp) {
Our API uses C pointers to various things and pointers aren’t something that any scripting language deals with well. In Tcl, we approach this by using a hash table in which the keys are unique string names for our pointers and the values are the pointers themselves. These mappings are referred to as ‘handles’ in Tcl and are a common way to uniquely represent underlying data structures that Tcl doesn’t necessarily need to manipulate directly.
/* create a hash table to store pointers to any order structs we create */
Tcl_HashTable *tblPtr = (Tcl_HashTable *)Tcl_Alloc(sizeof(Tcl_HashTable));
Tcl_InitHashTable(tblPtr, TCL_STRING_KEYS);
Now that we have the hash table, we can store it as data specific to this extension in a special area called the Associated Data. This is accessible only within our extension, and specific to this instance of the extension. This becomes important in multi-threaded situations or any situation where a program has multiple interpreters.
/* store a pointer to this hashtable in the extensions associated data
* so we can get at it from anywhere in the extension */
Tcl_SetAssocData(interp, "orders", NULL, (ClientData)tblPtr);
Next we tell Tcl about all the new commands we want to create in the interpreter, mostly just mapping the name you want for the command to a C routine that implements it.
Tcl_CreateObjCommand(
interp,
"create_order",
CreateOrderCmd,
(ClientData) 0,
(Tcl_CmdDeleteProc *) NULL
);
Finally, we tell Tcl that we’d like to call the extension “ordertcl” so our scripts can find it easily.
return(Tcl_PkgProvide(interp, "ordertcl", "0.1"));
}
Next, we’ll examine 2 of the wrapping functions. Firstly, a wrapper for create_order in the original API:
static int CreateOrderCmd(
ClientData clientData,
Tcl_Interp *interp,
int objc,
Tcl_Obj *CONST objv[])
{
The original call in the API takes no arguments and returns a pointer to an empty order. Our wrapper doesn’t need to do any argument processing, so we can get straight on declaring some variables.
char orderStructKey[24];
int madeANewHashEntry;
Tcl_HashEntry *orderEntry;
Firstly, we need to retrieve a pointer to the hashtable we’re going to store handles for our orders:
// retrieve the order hashtable from the associated data
Tcl_HashTable *orderHash = (Tcl_HashTable *)Tcl_GetAssocData(interp, "orders", NULL);
if(orderHash == NULL) {
Tcl_AddErrorInfo(interp, "Failed to retrieve order hash");
return TCL_ERROR;
}
Now we can call the underlying create_order function in the API to create an empty order
// create an order object and store a pointer to it in the hash table
order *my_order = create_order();
To create a unique string name we can use to reference our new order from Tcl, a typical approach is to use a prefix or suffix that denotes the type combined with a string representation of the pointer itself:
snprintf(orderStructKey, sizeof(orderStructKey), "order%p", my_order);
We now create an empty hash table entry using the key generated above, and set its value to the pointer to our new order:
orderEntry = Tcl_CreateHashEntry(orderHash, orderStructKey, &madeANewHashEntry);
if(orderEntry == NULL) {
Tcl_AddErrorInfo(interp, "Failed to create order hash entry");
return TCL_ERROR;
}
Tcl_SetHashValue(orderEntry, (ClientData)my_order);
Finally, we set the return value that our scripts will receive when they run our command to be the hash table key for our new order. This value can then be passed around into other Tcl commands that need to manipulate the underlying C representation.
// return the hash table key for our order
Tcl_SetObjResult(interp, Tcl_NewStringObj(orderStructKey, strlen(orderStructKey)));
return TCL_OK;
}
Now we’ll take a look at a slightly more complicated example, add_to_order. In the API, this takes a number of arguments – a pointer to an order to modify and the name, unit weight and quantity of the item to be added. Our Tcl command will need to take the same arguments but we can’t pass it a C pointer value. Luckily, we can retrieve the C pointer value from our hash table using its handle:
static int AddToOrderCmd(
ClientData clientData,
Tcl_Interp *interp,
int objc,
Tcl_Obj *CONST objv[])
{
We need to count the number of arguments that have been passed to our command to make sure there are enough. The C implementation of Tcl commands get passed the name of the command that was invoked as their first argument (in this case ‘add_to_order’). So in the normal case we’ll have 5 arguments passed, the 1 name of the command and the 4 arguments listed above.
if(objc != 5) {
Tcl_AddErrorInfo(interp,
"add_to_order: wrong number of args, should be order_handle name quantity weight");
return TCL_ERROR;
}
Arguments are passed to Tcl commands as Tcl_Obj objects which are polymorphic. They may have a number of internal representations and Tcl provides mechanisms for extracting particular representations. The simplest of these, and one that all objects can provide is a C string (char *). The first 2 arguments we need are strings, the handle for our order and the name of the item we want to add to it:
char *orderKey = Tcl_GetString(objv[1]);
char *orderName = Tcl_GetString(objv[2]);
Our other arguments need to translate to a C integer for the quantity and a C double for the unit weight:
int orderQuantity;
double orderWeight;
if(Tcl_GetIntFromObj(interp, objv[3], &orderQuantity) != TCL_OK) {
Tcl_AddErrorInfo(interp, "add_to_order: argument 3 must be an integer");
return TCL_ERROR;
}
if(Tcl_GetDoubleFromObj(interp, objv[4], &orderWeight) != TCL_OK) {
Tcl_AddErrorInfo(interp, "add_to_order: argument 4 must be numeric");
return TCL_ERROR;
}
Once we’ve processed our arguments, we retrieve the hash table from the associated data as before:
// retrieve the order hashtable from the associated data
Tcl_HashTable *orderHash = (Tcl_HashTable *)Tcl_GetAssocData(interp, "orders", NULL);
if(orderHash == NULL) {
Tcl_AddErrorInfo(interp, "Failed to retrieve order hash");
return TCL_ERROR;
}
We can then look up our order handle in the hash table:
// retrieve the order
Tcl_HashEntry *orderEntry = Tcl_FindHashEntry(orderHash, orderKey);
if(orderEntry == NULL) {
Tcl_AddErrorInfo(interp, "add_to_order: could not find order");
return TCL_ERROR;
}
order *my_order = (order *)Tcl_GetHashValue(orderEntry);
Now we have everything we need to call add_to_order from the underlying API
add_to_order(my_order, orderName, orderQuantity, orderWeight);
Tcl_SetObjResult(interp, Tcl_NewStringObj(orderKey, strlen(orderKey)));
return TCL_OK;
}
The rest of the functions in that C source file follow a similar pattern. There’s one more file of interest in the sample code – pkgIndex.tcl. This file tells Tcl where to find and how to load our extension. We can now build our extension and link it against our API library. The Makefile in the sample code can build it for linux and similar steps can be taken on other UNIX, MacOS or Windows systems. Tcl is portable too!Now we have our extension built, we can run scripts like the test.tcl script provided in the sample code:
lappend auto_path [pwd] ;# tell Tcl it’s OK to load extensions from here
package require ordertcl ;# load our ordertcl extension
# create an order using the API and print out the details
set my_order [create_order]
add_to_order $my_order "widgets" 5 11.3
add_to_order $my_order "flubbits" 7 10.0
array set details [get_order_details $my_order]
parray details
# recalculate the shipping cost with the new scheme
set_shipping_cost $my_order [shipping_cost $details(total_weight) 1]
# print out the details and see how much we saved!
array unset details
array set details [get_order_details $my_order]
parray details
Now with a couple hundred lines of C and a few lines of Tcl we’ve solved our problem without having to re-code our API. From here it’s trivial to re-implement the process_order CLI command and have it produce correct shipping costs.
The benefits don’t just stop there though. In future posts, we’ll explore much further – extending the API to eliminate memory leaks, creating a GUI tool for order entry using Tcl’s companion graphics toolkit, Tk, and even creating a web application.
Get our ActiveTcl data sheet and start solving tech debt today
