Today i want to talk a bit about “Relativity”, a new feature we’re shipping with Data Abstract this month (and which is actually available as “tech preview”in the current Winter 2009 releases).

“Relativity” is, simply put, a standalone server application that can be used to host Data Abstract schemas and serve data. In essence, it provides the complete Data Abstract middle tier, in a precompiled box and ready run. This offers several benefits, not the least of which are:

1. No need to build your own. Relativity provides all the power and abstraction of a true Data Abstract middle tier server, without you actually having to create, build, and deploy your own executable. This allows you to concentrate on the client development. It also is really handy for prototyping, if you just want to have database server to talk to, without much work, even if you plan to eventually implementing your own custom server.
2. Build on Data Abstract for .NET. .NET is platform with the most advanced version of DA – it supports our powerful DA SQL execution engine, and other features not available on all platforms. “Relativity” is being made available to all users of Data Abstract, so whether you’re a developer using .NET, Delphi, or Xcode, you can now write clients against Relativity that take full advantage of these features.

But let’s go and take a more detailed technical look at Relativity.

Relativity is a server application, so it does not provide any user interface. You can deploy a Relativity server in several ways, including:

• Running it as a regular user application, on Windows.
• Installing it as a Windows service.
• Running it as a Unix daemon with the mono-service2 tool, on Mac and Linux.
• Running it as a menubar item, on Mac.

Regardless of how you deploy and run it, Relativity will start making it’s Data Abstract services available on (by default) port 7099. But of course before it can start serving actual data, it will need to be configured with what data to serve. That is done thru the Relativity Server Admin tool, which is currently available for Windows (but versions for both Mac and iPhone are in the works).

When you first start it, the Admin tool will look something like this:

On the left hand side, you will see a tree view with all known instances of “Relativity” servers. This includes any servers (local or remote) that you connected to in the past, as well as any servers discovered on the local network using ROZeroConf.

Select any of the servers (here, “VEMMY” is the one instance running in my Windows VM), to connect and get more options.

The first two nodes allow you to configure how this “Relativity” server is accessed. This includes options such as what ports to listen on, what RemObjects SDK channels and messages to use, etc. Here you have the full choice of all the options provided by RO, you can choose the familiar HTTP, TCP, Super HTTP or other channels, BinMessage, SOAP Message, and so on.

Where it gets more interesting is the third node – Domains. This is where you configure the meat of the services Relativity is providing.

Each Domain corresponds to what typically would be a separate Data Abstract middle tier server. Domains are completely abstracted from one another and Relativity allows you to host as many separate domains as needed. With that, a single Relativity server can replace multiple DA servers, be it for related to completely independent applications or projects.

As you can see, i have three domains set up in my server, one for “BC7″, our new bug tracking system (using a PostgreSQL database running on our Xserve build server), one for “RemObjectsCom”, the database that drives our website (Microsoft SQL Server, running on the Windows server that hosts our website), and finally “OneSpace”, the database for a personal iPad project i am working on.

As we drill into a specific domain, you start seeing more things that will be familiar to you as a Data Abstract developer. Each domain can contain one or more Schemas – which correspond directly to .daSchema files or TDASchema components as you know them. You can manage schemas right there in the Admin tool, or directly launch Schema Modeler to edit the schema in question:

The domain also maintains a list of Connections to back-end databases available to it’s schemas. Once again, this directly corresponds to the .daConnections file or TDAConnectionManager component. Since Relativity is based on DA/.NET, these connections are based on the available DA/.NET drivers – regardless of whether your regular development platform is .NET, Delphi or Cocoa.

Another interesting part is the Login node. Relativity provides several options for handling authentication from client applications for data access (something you would normally need to manually implement, in a standalone DA server): in addition to relying on a static list of allowed username/password combinations, you can also choose to use a data table or command defined in one of the domain’s schemas to verify users (as i do in the example shown below) or query an external LDAP server for login.

The Login Provider architecture is extensible, so you can implement your own providers, for example if you need to do user authentication with a different method.

Client Access

Once you have Relativity set up to serve your data, your client application(s) can access it like any other Data Abstract server.

The only slight difference is that, since a Relativity server can host more than one domain, you need to tell the server which domain to access, by simply passing an additional Domain parameter as your login string, as in

User=foo;Password=bar;Domain=DomainName

You will also want to select which Schema to query. in regular DA applications, each schema usually has its own DataService, so keeping in line with that, Relativity allows you to append a schema name to the ServiceName property of your Remote Service. Instead of using DataService, simply use DataService.DomainName. (Alternatively, you can also pass a Schema to LoginEx).

In Closing…

So, this has been a brief introduction to Relativity – i hope you found it useful.

We think of Relativity as a strategic piece of Data Abstract, and it will play an increasing part in what we do for DA. We have lots of things planned for the future that will make Relativity even more powerful, such as our upcoming JavaScript Business Rules Scripting based on Carlo’s RemObjects Script for .NET.

You can try out the tech preview of Relativity now, and the official release will be part of what we ship at the end of this month. I’m looking forward to your thoughts and feedback!

marc

Core Plot is an open source graphing framework for Cocoa developers that makes it really easy to add graphs to your applications for the Mac and iPhone. Over the past half year, i’ve had the chance to use Core Plot in several of the internal applications i have been working on and have been very impressed with it, so i wanted to talk a little bit about how to use the library, in general, and how to make generate charts based on data in a Data Abstract for OS X application, in particular.

The example shown here is from a small iPhone application i wrote to keep track of the status of our Continuous Integration servers. The application shows the success (or errors) of any builds done on the machine, as well as – and this is the part where Core Plot comes in – graphs the number of tests that run and fail, over time:

In the application in question, the chart is being displayed in a cell within a UITableView, so our example starts out by implementing a custom UITableViewCell class that will host the graph. But the same general principles apply to show a chart anywhere else (and, replacing UIView with NSView, for using charts in a desktop app).

Core Plot uses a Core Animation as underlying technology, and as such draws itself in a specialized view class CPLayerHostingView. So we start out by crating this view, and adding it as sub-view:

- (id)initWithStyle:(UITableViewCellStyle)style 
{
  self = [super initWithStyle:style reuseIdentifier:@"MyChartCell"];
  if (self)
  {
    CGRect frame = self.contentView.bounds;
    CPLayerHostingView *chartView = [[CPLayerHostingView alloc] initWithFrame: frame];
    [self addSubview:chartView];
    //...

Once the CPLayerHostingView is created, we can add a graphs to it, and configure them. There are several graph types supported, but we want a regular line graph with X/Y coordinates, so we’ll choose a CPXYGraph class. We’ll add the graph to the view, and set a padding, to give us some room between the edges of the table cell:

  // create an CPXYGraph and host it inside the view
  CPTheme *theme = [CPTheme themeNamed:kCPPlainWhiteTheme];
  graph = (CPXYGraph *)[theme newGraph];	
  chartView.hostedLayer = graph;
 
  graph.paddingLeft = 10.0;
  graph.paddingTop = 10.0;
  graph.paddingRight = 10.0;
  graph.paddingBottom = 10.0;

The next step is to set up a plot space. A plot space defines the coordinate system for one or more charts in a graph, essentially mapping logical values to the area in the graph. For example, your chart might show values ranging from, say, 0 to 1000. The plot space defines the scale of those values in relation to the graph. It also defines the visual range of values that can be seen on screen.

To match our X/Y graph, we’ll create a CPXYPlotSpace, and set it to a range of 0-100 for both axis:

  CPXYPlotSpace *plotSpace = (CPXYPlotSpace *)graph.defaultPlotSpace;
  plotSpace.xRange = [CPPlotRange plotRangeWithLocation:CPDecimalFromFloat(0)
                                                 length:CPDecimalFromFloat(100)];
  plotSpace.yRange = [CPPlotRange plotRangeWithLocation:CPDecimalFromFloat(0)
                                                 length:CPDecimalFromFloat(100)];

Finally, we’ll set up to axis, for the X and Y coordinates. Axis provide labels and tick-marks for the viewer, to give context to the values being shown.

The X/Y graph already provides a set of axis in for of a – you guessed it – CPXYAxisSet class. There are several properties on the axis’ that are worth looking at at tweaking to to suite out needs:

• majorIntervalLength defines the number of units between “big” ticks on the axis. In this case it’s set to show one very 10 units. (if it were not for the exclusionRanges, discussed below, a numeric label would show for each major tick, as well).
• minorTicksPerInterval specified how many small ticks will be shown between each major one. in this case, a value of 2 indicated that small would show for 5, 15, 25, etc. (a value of 9 would show ticks for every single unit).
• Finally, exclusionRanges allows to define areas where no axis labels will be drawn. In my app, i don’t want to see any labels, so i set a range to exclude the entire visible graph

  CPXYAxisSet *axisSet = (CPXYAxisSet *)graph.axisSet;
 
  CPXYAxis *x = axisSet.xAxis;
  x.majorIntervalLength = length:CPDecimalFromFloat(10);
  x.constantCoordinateValue = length:CPDecimalFromFloat(2);
  x.minorTicksPerInterval = 2;
  x.borderWidth = 0;
  x.labelExclusionRanges = [NSArray arrayWithObjects:
                              [CPPlotRange plotRangeWithLocation:CPDecimalFromFloat(-100) 
                                                          length:CPDecimalFromFloat(300)], 
							  nil];;
 
  CPXYAxis *y = axisSet.yAxis;
  y.majorIntervalLength = length:CPDecimalFromFloat(10);
  y.minorTicksPerInterval = 1;
  y.constantCoordinateValue = length:CPDecimalFromFloat(2);
  y.labelExclusionRanges = [NSArray arrayWithObjects:
                              [CPPlotRange plotRangeWithLocation:CPDecimalFromFloat(-100) 
                                                          length:CPDecimalFromFloat(300)], 
							  nil];

Our graph is now fully set up – except for one important part: we still need to define one or more charts to display actual values. My application needs to show two graphs on top of each other – one showing the total number of tests that ran, and another showing the number of failures.

Core Plot supports a variety of chart (or, plot) types, such as bar and pie chart, but for this scenario, we want a simple line graph, which is done via a CPScatterPlot. (A scatter plot actually allows to draw a plot with points scattered all over the place, not just as string y = f(x) values, but we will use it in the more simplistic case).

We’ll create a new CPScatterPlot instance, configure some properties such as its lineWidth and lineColor, and most importantly, set the dataSource to the object that will provide the plot with its data – in this case, self. To round things off, we also tell the plot to fill the area underneath it with a gradient, going from almost-transparent green to nothing. This gives the plot some depth, without hiding the other plots.

  CPScatterPlot *dataSourceLinePlot = [[[CPScatterPlot alloc] init] autorelease];
  dataSourceLinePlot.identifier = @"AllTests";
  dataSourceLinePlot.dataLineStyle.lineWidth = 3.f;
  dataSourceLinePlot.dataLineStyle.lineColor = [CPColor greenColor];
  dataSourceLinePlot.dataSource = self;
  [graph addPlot:dataSourceLinePlot];
 
  // Put an area gradient under the plot above
  CPColor *areaColor = [CPColor colorWithComponentRed:0.3 
                                                green:1.0
                                                 blue:0.3
                                                alpha:0.3];
  CPGradient *areaGradient = [CPGradient gradientWithBeginningColor:areaColor 
                                                        endingColor:[CPColor clearColor]];
  areaGradient.angle = -90.0f;
  CPFill *areaGradientFill = [CPFill fillWithGradient:areaGradient];
  dataSourceLinePlot.areaFill = areaGradientFill;
  dataSourceLinePlot.areaBaseValue = CPDecimalFromString(@"1.75");

We repeat the above to add a second plot, except this time we use red for the line color and gradient, and set the identifier to @"FailedTests".

This was quite a bit setup, but our graph is now ready plot, and will ask its data source (our cell) for data by sending it messages from the CPPlotDataSource protocol. This protoco will be familiar to any Cocoa developer who worked with, for example, a UITableView or NSTableView, or any other control using a data source.

To the protocol, two methods need to be implemented. The first is numberOfRecordsForPlot: which will, simply enough, return the number of items (in case of a scatter plot, points, the chart will contain. The second can be one of three methods that return the actual data. We’ll implement numberForPlot:field:recordIndex:.

The data to display is retrieved via Data Abstract, using a simple DA SQL request on a table that contains all test runs:

	NSString *sql = [NSString stringWithFormat:@"SELECT TOP 50 ID, _TotalTestCount, \
                                     _FailedTestCount, FROM TestRuns ORDER BY ID DESC"];
	testRuns = [[rda getDataTableWithSQL:sql] retain];

The _TotalTestCount and _FailedTestCount fields are server-calculated based on a relation table and contain the total number of run and failed tests for each test run. We can Key Path operators to get the maximum value, and adjust our plot space accordingly, so that the entire graph fits into the chart:

  int count = [[testRuns rows] count];
  int maxTests = [[[testRuns rows] valueForKeyPath:@"@max._TotalTestCount"] intValue];
 
  CPXYPlotSpace *plotSpace = (CPXYPlotSpace *)graph.defaultPlotSpace;
  plotSpace.xRange = [CPPlotRange plotRangeWithLocation:CPDecimalFromFloat(0)
                                                 length:CPDecimalFromFloat(count)];
  plotSpace.yRange = [CPPlotRange plotRangeWithLocation:CPDecimalFromFloat(0)
                                                 length:CPDecimalFromFloat(maxTests)];

This is of course a bit redundant with the values we set earlier, but in real life application, the data access will usually happen in a different place than the initial setup (and data might get changed or refreshed during the course of the application), so i find it important to set up the graph properly to begin with, but then adjust the range to the real data.

With the data retrieved and the plot space adjusted, we can now implement the two data source methods.

numberOfRecordsForPlot: will simply return the number of rows in our table. numberForPlot:field:recordIndex will be called twice per data point, once for the X value (since we’re drawing a “non-scattered” plot, we’ll simply return the indexes) and one for the Y value, which we’ll retrieve from our data table. The different identifiers we assigned to the two plots will serve to distinguish which field to use for the Y value.

-(NSUInteger)numberOfRecordsForPlot:(CPPlot *)plot 
{
  return [rows count];
}
 
-(NSNumber *)numberForPlot:(CPPlot *)plot 
                     field:(NSUInteger)fieldEnum 
               recordIndex:(NSUInteger)index 
{
  switch (fieldEnum)
  {
    case CPScatterPlotFieldX: 
    {
        // inverse numbers, so first (latest) test run is on the right.
        int x = [rows count]-recordIndex; 
        return [NSDecimalNumber numberWithInt:x];
    }
    case CPScatterPlotFieldY:
    {
      if ([plot.identifier isEqual:@"AllTests"])
      {
        float v = [[[rows objectAtIndex:index] valueForKey:@"_TotalTestCount"] floatValue];
        return [NSNumber numberWithFloat:v];
      }
      else
      {
        float v = [[[rows objectAtIndex:index] valueForKey:@"_FailedTestCount"] floatValue];
        return [NSNumber numberWithFloat:v];
      }
    }
  }
  return nil;
}

And that’s it! You can find more info on Core Plot on their Google code page; i also recommend joining their mailing list for latest news and support.

One of the (few) concerns we hear from people looking to get started with Delphi Prism is that most of the code samples and snippets that are available on the net are written in C#.

Because .NET is fully language agnostic, reusing existing libraries and assemblies written in C# (or any other language) generally is no problem, but for smaller pieces of code – classes or even just small code snippets – it is inconvenient for the Prism developer to start a separate C# project – making manual translation (and sometimes, deeper understanding of the C# language and its intricacies) necessary.

We wanted to make this easier, so for the upcoming February* release of Delphi Prism, we’ve added two new features to the IDE, based on out open source C# to Oxygene tool.

The first is the new Paste C# as Oxygene command in the source editor. If you find snippets of C# code on the web (or elsewhere) that you want to reuse in your project, simply copy them to the clipboard, and then right-click into your code and choose Paste C# as Oxygene from the context menu:

The IDE will translate the code for you, and paste the Oxygene version in. This will work on just about any type of code – from a single line of code or a few compound statements, to entire method bodies, or even full classes. (Because classes are self-contained entities in C#, but split in declaration and implementation in Oxygene, pasting a whole class will actually generate a whole source module, including with “interface” and “implementation” sections. As such, pasting full classes works best when done into an empty file).

All of this is great when you have code snippets, especially from websites, forums or newsgroups. But what about C# files already on disk? That’s where the second feature comes in: if you right-click your Delphi Prism project and go to the Add menu, you’ll see a new Import C# sub-item. Select this, and you’ll be presented with a regular File Open dialog, where you can browse to and select one or more .cs files on your harddisk. When you click Ok, each .cs file will be translated to Oxygene, and a corresponding new .pas file will be added to your project:

How well does this work? C# is a complex language, so while Oxygene provides a complete superset of what can be done in C#, there might be some cases where complicated C# code will confuse the translator, but in the vast majority f cases, the translate code will work as is.

One of our test bases is the entire RemObjects SDK for .NET library, and the tool translates that very satisfactory.

Paste C# and Import C# are probably not features you want to use to blindly translate whole projects with and reuse the resuiting code without further testing or review, but in our experience it is working very well or the intended purpose – reuse onf code snippets found online.

Just like Oxfuscator, covered in my previous post, this is in the current beta drop, so if you already a Delphi Prism user (or buy it now), make sure to apply for the beta.

Bugs 7, the new bug-tracking application i have been working on over the past month for internal use here at RemObjects, employs several different data access paradigms (all based on Data Abstract for OS X), to accommodate the different nature of the data in individual tables.

The most interesting one is the main Issues table that contains all the bugs and tasks that are logged in the system. That is due to the fact that (a) it is a huge table (with over 25,000 records as of now, sized at about 11 MB when transferred over the wire, compressed) and that (b) by the very nature of a bug tracking system, this table changes frequently, and needs to be updated on the clients, often.

Rather then downloading the entire table anew every time, we optioned for a solution that allowed us to incrementally fetch only those records that have changed, and integrate them with the local dataset. This way, only minimal traffic is occurred for the regular refresh (which our client, by default, does every two minutes).

In the following post, i want to give you a quick glimpse at how this was accomplished, and how you can leverage the same technology in your own Data Abstract applications.

The Server

A couple things happen on the middle tier server (written using Data Abstract for .NET), to support the incremental refresh. Like all tables in the Bugs database, our Issues table has an UpdatedDate field, which gets automatically adjusted by the business logic code on the server. Every time a new issue is created, or an existing issue is updated, the server puts the current UTC time into the UpdatedDate field, clearly marking the order in which issues have been touched.

This is handled by a simple BeforeProcessChange event handler on the server, which simply adjusts the received delta, as such:

method BugsDataService.bp_Issues_BeforeProcessChange(aSender: BusinessProcessor; ea: DeltaChangeEventArgs);
begin
  ea.DeltaChange.NewValues['UpdatedDate'] := DateTime.Now;
  ea.DeltaChange.NewValues['UpdatedByID'] := Session['UserID'];
end;

(Of course the actual code in our server performs a lot more checks and changes, to enforce business logic for our database – but that’s beyond the scope of this post.)

Also, our Issues table does not permit deleting of records (only closing of issues, which sets their status accordingly, but does not remove the rows from the database). This alleviates the problem of worrying about rows disappearing from the table altogether.

The Client

On the client, a bit more custom logic is necessary, to perform the incremental updating.

When the client application (“Bugs 7”) is first started, it checks whether a briefcase file with a local copy of the data is found from a previous run, or not.

If not, the client will start a request to download the complete set of data from the server. This is a one-time process, and will download the entire table with it’s (currently) 11MB across the wire. Once downloaded, it is stored in a briefcase file, so on next application start, the data can be loaded locally. After the download is finished, the application also takes note of the latest UpdatedDate value it can find in the table, for future reference. This is made easy by Cocoa’s KVC and Key Paths:

- (void)downloadIssueData 
{
	// Initial download can take long, server side, to gather data. 
	// allow for a longer timeout.
	[(ROHTTPClientChannel *)[[rda dataService] channel] setTimeout:360];
 
	// fetch full table from server	
	issues = [[rda getDataTable:@"Issues"] retain];
	maxDate = [[[issues rows] valueForKeyPath:@"@max.UpdatedDate"] retain];
 
	// save table to briefcase
	[self saveIssues];
}

If instead a briefcase was found, the Issues table simply gets loaded from that file, alongside the stored maxDate:

- (void)loadIssueDataFromBriefcase
{
	DABriefcase *briefcase = [DABriefcase briefcaseWithFolder:briefcasePath];
	issues = [[briefcase tableNamed:@"Issues"] retain];
	maxDate = [[briefcase.properties valueForKey:@"BugsMaxUpdateDate"] retain];
}

Whichever path was taken, the application now holds a local copy of the Issues table it can work with. The next step it to schedule the regular refreshes, and for that an NSTimer is configured, to fire at regular intervals, on a background thread.

This NSTimer will trigger our beginRefreshBugs method, which uses asynchronous requests to start checking for new issues. It uses the previously stored maxDate and a feature of Data Abstract called DA SQL, to fetch only those issues that have newly changed:

- (void)beginRefreshBugs
{
	// prevent two refreshs happening at the same time, if the NSTimer
	// triggers again before the previous refresh has finished.
	if (refreshingBugs) return; 
	refreshingBugs = YES;
 
	// build the DA SQL query
	NSString *sql = [NSString stringWithFormat:"SELECT * FROM Issues WHERE UpdatedDate >= '%d'",
						(int)[maxDate timeIntervalSince1970]];
 
	//and start an Async Request to the server
	DAAsyncRequest *ar = [rda beginGetDataTable:@"Issues" withSQL:sql start:NO];
	[ar setDelegate:self]; 
	[ar setContext:@"RefreshIssues"];
	[ar start];
}

The DAAsyncRequest, once started, will communicate with the server in a background thread, without blocking the caller. beginRefreshBugs will return right away, and not wait for the request to complete (or fail).

Once the request did complete, it will call back to a delegate method (in this case we assigned self as the delegate, above), called asyncRequest:didReceiveTable:. It is here that we handle integrating the received data back with our big issues table by sending it the mergeTable:withPrimaryKey: message. This will replace the data in any rows that have changed, as well as add any new rows to the table:

- (void)asyncRequest:(DAAsyncRequest *)request didReceiveTable:(DADataTable *)table
{
	// our class may handle access to more tables. check the context
	// to make sure we handle the right requests.
	if ([[request context] compare:@"RefreshBugs"] == NSOrderedSame)
	{
		// nothing received? nothing to do!
		if ([table rowCount] > 1) 
		{
			@try
			{
				[issues mergeTable:table withPrimaryKey:@"ID"];
				maxdate = [[[table rows] valueForKeyPath:@"@max.UpdatedDate"] retain];
				[self saveBugs];
			}
			@finally 
			{
				refreshingBugs = NO;
			}
		}
	}
}

Of course a refresh might also fail (for example due to a broken internet connection, for this case, we implement a second delegate method, called asyncRequest: didFailWithException:, as follows:

- (void)asyncRequest:(ROAsyncRequest *)request didFailWithException:(NSException *)exception
{
	if ([[request context] compare:@"RefreshBugs"] == NSOrderedSame)
	{
		refreshingBugs = NO;
	}

The View

The last step after receiving new issues is to update any affected views. This happens more or less automatically, as every view that shows one one more issues (whether it’s the regular grid view of issues, a chart visualizing issue data, or an individual issue’s detail view) will be have registered itself to observe DA_NOTIFICATION_TABLE_CHANGED notifications on issues. And like any other change to a data table, mergeTable:withPrimaryKey: will send such a notification if changes happened, allowing all views to update themselves.

In Bugs, all of this happens in the background, so over time the view(s) presented to the user just seamlessly adjust themselves, as changes happen – new issues come into views; issues resolved by other users disappear on their own, etc.

Today i want to take a quick peek at a new code obfuscation solution for .NET we have been working on.

Back sometime in mid-2009, when we started migrating the Oxygene/Delphi Prism IDE code base to a new managed project system in order to support Visual Studio 2010 and MonoDevelop alongside to the existing Visual Studio 2008 support, we ran into limitations with our existing obfuscation solution (Xenocode). So we started to look for alternatives.

We found Obfuscar, a seemingly abandoned open source project that provided a great feature set and was based on Mono’s Cecil reader/writer library for .NET assemblies – the same library that we’re using for Oxygene and that Carlo had a lot of experience working with (and contributing to). So Carlo took that, expanded it and fixed some issues, and we published the result as a new open source project on code.remobjects.com, our open source SVN repository. Obfuscar, at that stage, was a command line application, and we have been using it in this form for production, for over half a year now (although the August release of Delphi Prism still did ship with XenoCode-obfuscated binaries).

But, we figured, we could do more than that. So we decided to integrate the product with both the IDE and the MSBuild build system, to make obfuscation an integral part of a solution. The result is RemObjects Oxfuscator, the name indicating that it is intended to complement Oxygene – although it will of course work language agnostic on assemblies created with any language (or even those you only have the binary version of).

Let’s See How This Works in Action

Oxfuscator provides it’s own project type, and a new project template that goes along with it. In the most common case, you would use the solution’s Add|New Project… context menu to add an Oxfuscator project to your existing solution, and have obfuscation happen as part of the build. But of course you can also use Oxfuscator on its own, only referencing existing assemblies.

Once added, the Oxfuscator project will show in Solution Explorer, much like a general language project (be it Delphi Prism or C#) would. Different than a “real” project, an Oxfuscator project contains no source files, only an “Assemblies to Obfuscate” item. This node works much (if not exactly) like the “References” node in language projects, and you can use it to configure the assemblies Oxfuscator will work on. You can right-click it to use the common Add Reference dialog to add assemblies, including libraries registered with the .NET framework, files browsed to from disk, and “project references” to other language projects that are part of the same solution.

Just as with project references in normal projects, the Oxfuscator project will take part in MSBuild’s automatic dependency resolution, to make sure Oxfuscator runs only after all the projects it depends on have been built.

So What Exactly Does Oxfuscator Do?

Oxfuscator will take all the assemblies the project references, and globally obfuscate all shared names, according to the rules defined for it. For every identifier defined in in any of the assemblies, it will generate a new name that is unique (in that scope), and adjust both the definition and all pieces of code that reference that identifier. The result is a new set of classes and types that performs the same function as the old code – but with names that are completely senseless and next to impossible to reverse-engineer.

Oxfuscator provides several settings to provide less readable or completely unreadable identifier names. In the simplest case, they will be made up of gibberish letters, which make no sense to the reader, but are still easily distinguishable (this setting is useful for debugging your obfuscation, as “a”, “aa” and “b” can at least be told apart in Reflector and when seeing them in call stacks and exception messages). In the best case, identifiers will leverage the full power of Unicode and made up of totally illegible characters. Oxfuscator will use all XX different types of whitespaces, as well as obscure punctuation marks, to make sure what your hacker is seeing is, well, almost invisible.

Did i say debugging your obfuscation? Why would you need debugging? Well. maybe not most, but many projects are not completely standalone, but need to interact with external libraries, and thus need certain classes or class members excluded from obfuscation. A good example is our IDE integration for Visual Studio, which, after all, drove Oxfuscator. If Visual Studio loads your project as an add-in and expects it to expose (say) a class called Bar with a method called Foo – well, suffice to say those names better not be obfuscated!

Which brings us to the final topic: exclusions. Oxfuscator provides two ways to exclude certain identifiers from obfuscation. The first, simplest, and (imho) much preferred way – at least if you have access to the code – is to simply decorate your classes or members with the [System.Reflection.Obfuscation(Exclude := true)] attribute, to exclude them from obfuscation. Oxfuscator will automatically detect and honor that. Alternatively, you can specify a list of exclusions “manually” in the project options, as shows here:

In either case, excluding an identifier will of course automatically apply both to the definition and any references, within the obfuscated assemblies.

Oxfuscator will automatically not obfuscate any identifiers referenced from external assemblies (such as the framework base library itself), including overridden methods or implementations of interfaces defined externally.

When Can You Get it?

Oxfuscator is currently with beta testers, and we’ll talk more about availability, soon. Stay tuned to this space for more.

One of the new features we introduced to Data Abstract in the latest release is the Local Data Adapter, or LDA, for short.

What is a Local Data Adapter, and What Does it Do?

If you’ve used Data Abstract before, you are probably familiar with the LDA’s older sibling, the Remote Data Adapter. Data Abstract is built around the concept of multi-tier data access, and the Remote Data Adapter (RDA) forms the central hub on the client side to access data from the remote server, ie from the middle tier. The RDA knows how to request data for one or more tables from the server, how to post changes back, and how to handle reconciliation problems. The RDA understands various client side technologies such as Dynamic Where and DA SQL, and – on .NET – it comes in two versions, one that support regular DataSets, and another one that uses DA LINQ.

Sometimes, however, it is also necessary to work with data locally, within the middle tier. There are several scenarios for this – for example, your business rules code might need to access data to verify or adjust changes submitted from a client, or independent code running in the middle tier server might need to work on data, for example to generate reports, or mine data.

In the past, accessing data within the middle tier has been less straight-forward than it should have been: one either had to work with the lower-level APIs of Data Abstract or use ADO.NET directly, bypassing DA (and the Schemas and everything that goes along with them) altogether. The LDA changes that.

The Local Data Adapter is, essentially, a component that works just like the RDA, except that it is specifically designed to work with data within the same tier. It provides the same APIs for data access as the RDA does (including DA LINQ, on .NET, and DA SQL on both platforms) but does not use RO to talk to a remote server and instead works directly with a local data service.

Setting up an LDA

As mentioned above, the LDA works directly against a local DataService, so it needs none of the setup of RemObjects SDK communication infrastructure that the RDA does. There’s no need for a Channel, a Message or a RemoteService. Instead, you just connect the LDA to a service directly, in one of two ways.

1. If your code already has a reference to a concrete service instance, that reference can simply be assigned to the LDA’s ServiceInstance property, and be directly usable.

This is especially helpful if you’re using the LDA within event handlers on the Data Service or Business Processors. If your code is already executing in the context of a Data Service, so why not use it?

2. Alternatively, you can set the ServiceName string property to the name of a data service that exists within your server. As needed, the LDA will obtain a reference for this service using RemObjects SDK’s regular server manager infrastructure. The LDA and RO will take care of instantiating (and releasing) the service as needed, and the regular instantiation mechanics will apply (for example, your chosen class factory will be used).

This option is useful if your code does not have a reference to a service yet, for example when you are accessing data outside of a client request. In a sense, this second option more closely resembles how the RDA works – it connects to a data service by name; it just does not do it across the network, but locally.

One thing to keep when using the LDA is that while it is bypassing the the RO network infrastructure to directly call the local data service as needed, any restrictions implemented in your service will still apply. For example, if your data service requires login or uses Roles to restrict access, it will need a session with the appropriate privileges. When assigning ServiceInstance to self, that session will be inherited from the outer request, but when using ServiceName, you might need to set the SessionID and manually set up a session as expected by your service.

Two Examples

Let’s look at the LDA in action with two examples. The code shown here uses the DA LINQ version of the LDA, in Data Abstract for .NET, but the same principles would apply using DataSets, or using the Delphi version of the LDA.

First, let’s write a BeforeProcessChange event for a Business Processor, to validate and influence changes as they are applied from the client.

To keep the code snippet short, we’ll focus on the update case only. The code first uses the LDA to obtain the original record that is being updated. In the real life project (Bugs 7) that this code is taken from, the “Issues” table is filtered by what individual users are allowed to see, so this lookup not only returns the original row, it also verifies that the user is indeed allowed to access it. Once obtained, we can do arbitrary validation of the data. Finally, we also use the LDA to insert an extra row into an entirely separate “History” table, in this case serving as a sort of log of each changes:

void bpIssues_BeforeProcessChange(...)
{
  // instantiate a new LINQ LDA that uses the same Data Service
  lda = new LinqLocalDataAdapter();
  lda.ServiceInstance = self;
  if (ea.DeltaChange.Type == ChangeType.Update)
  {
    // get the Primary Key, "ID&" from the Delta...
    Int64 oldID := ea.DeltaChange.OldValues["ID"] as Int64;
 
    // ... and look up the issue as it is now
    currentRow = (from i in lDA.GetTable<issues>
        where i.ID == lOldID).FirstOrDefault;
 
    // if not found, the user probably isn't allowed to touch it!
    if (currentRow != null) throw new Exception(String.Format("No access to original row {0}, or row does not exist.", oldID));
 
    // do some more checks of ea.DeltaChange vs currentRow, if needed.
    if (currentRow.locked) throw new Exception("Row has been locked to prevent future changes"));
 
    // ...
    // once all is approved, insert an entry into a "History" table.
    var history = new History();
    history.IssueID = lOldID;
    history.UpdatedByID = Session["UserID"] as Int32; // who changed it?
    history.UpdatedDate = DateTime.Now;
    lda.InsertRow<History>(lHistory);
    lda.ApplyChanges();
  }
  else ...
}

So far so good. As a second example, we’ll use the LDA outside of a DataService. In this case (again a trimmed-down real life example from Bugs7), we have a scheduling process that looks at all the rows in the “Issues” table to calculate the appropriate execution order and assign issues to the proper developers. This code runs on a Systems.Timers.Timer, to trigger at regular intervals.

void TimerElapsed(...)
{
  using (LinqLocalDataAdapter  lda = new LinqLocalDataAdapter())
  {
    lda.ServiceName = 'Bugs7DataService';
    lda.SessionID = Guid.NewGuid;
 
    // create a session and set the roles needed for the tasks at hand
    ISession s = SessionManager.GlobalSessionManager.GetSession(lda.SessionID);
    s.Roles = ["Internal", "Scheduler", "RealDB"];
 
    // load the data we need, via DA LINQ
    var issues = from i in fLDA.GetTable<scheduledissues>
        where (i.StatusID != Status_Closed)
        orderby i.CreatedDate;
    // do our processing on the "Issues" sequence.
  }
}

As you see, we create the new LINQ LDA, and initialize it with the name of our data service (in this case, this is the same service that is also exposed publicly, but it could just as well be a specific service provided only for internal use).

Because our service requires login, we set its SessionID property to a new GUID, and then set up a session with that ID (essentially faking a successful “login”), and also configure a couple of Roles that our service expects; in this case “Internal” tells the service it is being accesses from within the server (i.e. not as a client request), which relaxes some security restrictions, and “RealDB” will prompt it to connect to the production database (rather than a second test database that it also serves).

Summary

This post is only meant to give a brief introduction to the LDA. You can read more it on the Local Data Adapter in our wiki.

Yours,
marc

I wish all of our customers and readers a happy and successful new year of 2010. We have great things coming up, so 2010 should be a very exciting year!

Photo shot in Berlin Friedrichshain at the Frankfurter Allee, tonight at 12:43 AM.

We’re getting close to shipping our first release of Data Abstract for OS X as well as the 1.1 update for RemObjects SDK for OS X at the end of this month, and as part of crossing all the final Is and dotting all the remaining Ts, Carlo and i have been working on leveraging the tools in our new documentation tool chain to integrate the RemObjects Wiki (or, the relevant part of it) as Xcode documentation:

What you are seeing here is our wiki as standard Xcode doc set. You can browse and search it just as you would the standard Apple docs; you can navigate to it by Cmd+Option-Double-clicking identifiers in your code – in other words, you get the full documentation experience.

The version shown here works directly on the live wiki – meaning as soon as changes and updates are made to the pages online, you will see them reflected in the wiki as well. In the long term (not for this release, yet), we’ll also be providing an offline version, that contains a snapshot of the entire wiki, for offline use (i.e, when you’re working on the proverbial plane).

The Docset will also update via RSS feed, so content (in the offline version) and structure changes (such as the addition of new pages to the indexes, etc) will automatically update within Xcode, on a regular basis.

Stay tuned for more info on RO/OSX and DA/OSX, as we near release.

One common scenario when developing iPhone apps is that you have a desktop application of some sorts that you want your iPhone to pair with – be it to sync data, remote control one with the other, or establish any other kind of communication.

In this scenario, the desktop app would be the “server”, in RemObjects SDK terminology, and the iPhone would be the client.

For the next release of RemObjects SDK for OS X, coming next month, we’ve created a nice sample that illustrates how to easily achieve this, and i figured i would post a small screencast of the sample in action, below.

What you see is a desktop app written in Delphi (although .NET will do just fine, as well), which for sample purposes has a pretty minimal UI, but in real life would be your full-fledged desktop UI app. When the iPhone client is a launched, it automatically detects any instance of your desktop app running on the local network and shows them in a list.

(The UI for this would be up to the application designer, obviously; for simplicity we simply show them all in a UITableView. In real life, if only one server was found you’d probably want to forgo the list altogether and just ask the user if he wants to connect).

Once the user picks a server, communication is initiated, by the iPhone calling a service method and passing it a unique GUID it generated. Since this is the first time these two are talking to each other, the server rejects the phone, but displays a random 4-digit code in it’s UI (this code is not known to the phone, at this stage). The phone asks the user to enter the code and, once done, sends it to the server to confirm the pairing. Once successful, the server stores the client’s GUID for future reference – and any future calls (even after the phone app has been restarted) will work right away.

Of course instead of showing an UIAlertView, your application would instead commence doing it’s real work.

Here’s the app in action:

If your internal body clock has gotten attuned to our regular release cycle here at RemObjects (or even if you just did the math ;), you might be expecting our new set of Fall 2009 releases right around this week.

No worries, these Fall releases are in the works, but this time around, we have decided to grant ourselves another two weeks and ship them to you in the second week of September, instead. This has several reasons:

For one, as you might know, we also shipped a new major release of Delphi Prism this month, which is making its way to customers (and available as trial) now. Originally, RTM for Delphi Prism was planned for a few weeks earlier than when it actually happened, for a variety of reasons, so Delphi Prism has kept us busy a bit longer than we had anticipated. We did not want the other products to suffer from that and see a rushed release, so we made extra time.

For another, Embarcadero also just shipped a new version of Delphi/Win32, and we wanted to get final and official support for that into the new releases, as well. Of course we’ve been testing our products with pre-release versions for a while now, and had begun to get ready for Delphi 2010 early on (and there weren’t really any complications, either), but while that is all fine and well in the proverbial lab, there was still a lot to do and test once we received the final binaries and could install those on the build servers that crank out the shipping products. We could have done it in time, but once again, we did not want to rush things and give more than a good week of testing to builds coming off our build machines with D2010 support (and ship a beta or two of those to our external testers, as well). So two extra weeks came in really handy here, as well.

If you’re detecting a general theme here, you are not wrong. We’ve been asking our customers (that’s you) what we can improve, and one consistent feedback we got was: Quality. New features and bug fixes are great, but (understandably), you have told us that you need to rely on new releases to be of high quality and on upgrades to be as risk-free to install as possible. And while i believe we’ve made great improvements over the past year or so in that regard, there’s still room for more improvement, and we intend to leverage that.

So we have scaled down the scope of both this upcoming Fall release and the next Winter release (due in November), in order to focus more of our resources onto better quality assurance. More unit tests. Way more unit tests. More automated non-unit tests. But also way better infrastructure to exercise these unit tests. Over the past month, we’ve started to set up a sophisticated infrastructure for Continuous Integration (and i’ll blog more about that in a separate post, soon); we’re setting up enhanced infrastructure for automated testing and reporting based on our internal Afterhours testing framework. We’re also reworking programs such as our beta tests, and TeamRO, to provide a better experience for everyone involved, more frequent beta builds (with more details about them) and tighter feedback loops.

I hope the upcoming Fall release, two weeks from now, will show the first, tentative, fruits of this shift, and that there will be a noticeable impact on quality with the Winter releases.

Of course this does not mean that the upcoming releases will be entirely without new goodies. Two IMHO very exciting new features we will be shipping are integrated support for Roles in RemObjects SDK (i’ll write more on this, soon), and the Delphi port of our new PCTrade sample suite for Data Abstract. We’ve also been working on a new component for Data Abstract for .NET, the “Local Data Adapter” which will work similar to its Remote counterpart, but allow easy access to data from within the middle tier.

The LDA will not ship in the Fall releases (another factor of our renewed focus on quality-over-features), but i’ve been using it myself in a couple of internal projects for a while now, and it is shaping up nicely. I’ll be talking more about the LDA, and other things we have in the works for November, soon.

In the mean time: new beta builds of the upcoming Fall release are available for all active customers, on beta.remobjects.com. If you want to get an early peak, or are desperate for Delphi 2010 support, make sure to grab those; they are pretty stable, at this stage, and we’ll be updating them frequently as we near release. Or, sit back and relax for the next two weeks, until the official Fall 2009 releases will be available, September 12.

yours,
marc