Archive for the ‘Code’ Category

Friday, April 5th, 2013

This picks up with the code generation series. See other posts:

In this post we will be using project templates for code generation. Project templates are great when you are going to create many projects with similar boilerplate type code. For example if there is a plugin architecture and each plugin has one (or many) interfaces that it must implement.

In order to get started create a new project inside a solution. Write all the boilerplate code that is needed. Use the Template Parameters in the boilerplate code to make replacements. Note: it’s okay if the project doesn’t compile after adding template parameters. Make sure to setup any project specific build events, assembly references, output paths, etc. inside the project.


/*************************
* File: $safeprojectname$Plugin.cs
* Author: $username$
* Created: $time$
************************/

using Main;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace My.Plugins.Plugin.$safeprojectname$
{
///

/// Plugin: $projectname$
///

public class $safeprojectname$Plugin : IPluginInterface
{
///

/// Does work for the plugin
///

void IPluginInterface.DoWork()
{
// TODO: This MUST be implemented
throw new NotImplementedException();
}

///

/// User visible description of $safeprojectname$
///

///
string IPluginInterface.GetDefaultInfo()
{
return "Plugin ($safeprojectname$) created by $username$";
}

///

/// Returns true when the plugin is ready to do work
///

///
bool IPluginInterface.ReadyToGo()
{
return false;
}
}
}

After the boilerplate code is in go to File > Export Template…

ExportTemplate

This will bring up the “Export Template Wizard”. Make sure to select the project with the boilerplate code and click next.

ExportTemplateWizard1

The next page of the wizard allows some customization of the project name. Make sure to automatically import into visual studio.

ExportTemplateWizard2

That it. Write code once, reuse multiple times. To test out the new project template go to New > Project in Visual Studio. Since the project was automatically imported the new project template is available.

NewProject

The new project will have the file(s) specified with all the template parameters filled in:


/*************************
* File: My_Plugin_Template1Plugin.cs
* Author: Sirchris
* Created: 3/28/2013 6:52:06 AM
************************/

using Main;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace My.Plugins.Plugin.My_Plugin_Template1
{
///

/// Plugin: My Plugin Template1
///

public class My_Plugin_Template1Plugin : IPluginInterface
{
///

/// Does work for the plugin
///

void IPluginInterface.DoWork()
{
// TODO: This MUST be implemented
throw new NotImplementedException();
}

///

/// User visible description of My_Plugin_Template1
///

///
string IPluginInterface.GetDefaultInfo()
{
return "Plugin (My_Plugin_Template1) created by Sirchris";
}

///

/// Returns true when the plugin is ready to do work
///

///
bool IPluginInterface.ReadyToGo()
{
return false;
}
}
}

That’s a quick bear minimum of what can be done with project templates. Project templates can package up any set of files. In the exported template created look at the vstemplate file. There you can start to get a feel for some of the additional items that could be accomplished with templates. Make sure to look at the VSTemplate section on the MSDN.

Here is the vstemplate file created from this simple example




My Plugin Template
This is to create plugins faster
CSharp


1000
true
My Plugin Template
true
Enabled
true
__TemplateIcon.ico



Class1.cs

AssemblyInfo.cs




This is one of the simplest (and powerful) ways to get started with code generation. Keep them mind next time creating multiple project that are similar in nature. There are still two more examples of code generation coming up.

Monday, March 11th, 2013

One of the benefits of a startup is there is a very rapid code, test, deploy to production cycle. For a while at CollectedIt we had a manual (but documented!) process to deploy code to production. This worked well for a while, but it started to get tedious. Plus as anybody who has done any number of production installs knows the more steps a human does, the more chances for an error.

It was time for an automated way to deploy to our code. First we looked into using something like TFS or Jenkins. These tools however required installation somewhere. CollectedIt is very lean so we prefer not installing excess services in production, spinning up a new server in the cloud, or investing in a physical server just for an automated build tool (to be clear we would spend the resources if we deemed it necessary). Next our thoughts turned to writing something homegrown.

CollectedIt runs on Windows servers in the cloud. I had been exploring PowerShell on and off for a little while and seemed like the perfect solution for a quick and easy homegrown deployment script.

PowerShell comes with a very powerful feature called Remoting which is a technology that lets you use PowerShell to remotely control one (or many) remote computers. There were however 2 major obstacles that we needed to be overcome.

  1. Remoting has no out of the box way to copy files from server to server
  2. Remoting over the Internet is not the most straight forward of configurations

Not having a way out of the box to copy bits to a server with PowerShell is annoying. There are ways to copy bits over the Remoting (such passing over a byte[] parameter when doing a remote call that has the contents of the file). However there was no way that was robust enough, or performed well enough for our tastes. We went ahead and configured an FTP server as a file server. Since PowerShell is built on top of .NET we can use FTP with Microsoft.NET. The code samples in the using FTP with Microsoft.NET blog entry are all in C#, but they translate to PowerShell fairly easily. Here is an example of using FTP over explicit SSL to upload a file.


$ftp = [System.Net.WebRequest]::Create($ftpuri)
$ftp.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftp.Credentials = New-Object System.Net.NetworkCredential($username, $password)
$ftp.EnableSsl = $true
$ftp.ContentLength = $filebytes.Length
$s = $ftp.GetRequestStream()
$s.Write($filebytes, 0, $filebytes.Length)
$s.Close()

Now that we have a way to copy bits we turned to the Remoting part. In order to use Remoting over the Internet first we had to enable Remoting on the server.

PS> Enable-PSRemoting -Force

If we were on a secured domain we would have no more steps. Getting Remoting to work securely over the Internet however, we are just getting started.

  1. Create a certificate for SSL. We used the makecert command that comes with the windows SDK.

    PS> makecert.exe -r -pe -n "CN=collectedit.com" `
    >> -eku 1.3.6.1.5.5.7.3.1 -ss my -sr localmachine -sky exchange`
    >> -sp "Microsoft RSA SChannel Cryptographic Provider" -sy 12
    >>

    This places the certificate inside the “Local Computer\Personal” certificate store.
  2. Get the thumbprint for the certificate.

    PS> ls Cert:\LocalMachine\My

    Directory: Microsoft.PowerShell.Security\Certificate::LocalMachine\My

    Thumbprint Subject
    ---------- -------
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX CN=collectedit.com

  3. Now we needed to create an HTTPS endpoint. We can use the winrm command to help with that. One note of warning: winrm was made for use at regular old cmd.exe . Using it with PowerShell we end up with a lot of backticks. If you run into frustration using winrm with PowerShell just switch to cmd.exe (it’s okay I won’t tell).

    PS> winrm create `
    winrm/config/Listener?Address=*+Transport=HTTPS`@`{Hostname=`"`collectedit.com`"`
    ;CertificateThumbprint=`"`XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX`"`}
  4. At this point the server is able to accept Remoting connections over the internet over SSL. To disable the HTTP remoting listener it’s as easy as finding the listener then removing it.

    PS> ls WSMan:\localhost\Listener

    WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Listener

    Type Keys Name
    ---- ---- ----
    Container {Address=*, Transport=HTTP} Listener_809701527
    Container {Address=*, Transport=HTTPS} Listener_1353925758

    PS> Remove-Item WSMan:\localhost\Listener\Listener_809701527

To use PowerShell Remoting over SSL there are additional parameters we needed to set when creating a remote session. The first is to tell PowerShell to use SSL and the second is to ignore the certificate authority since our certificate is self signed. This is as easy as

PS> $so = New-PSSessionOption -SkipCACheck # skip certificate authority check
PS> Enter-PSSession localhost -UseSSL -SessionOption $so # note the "UseSSL"

If there are any issues connecting first check firewall settings to allow port 5986 then check out this awesome blog post on Remote PSSession Over SSL finally if you still have issues use the about_Remote_Troubleshooting help page

With the two major hurdles solved we were confident that we could use PowerShell for our uses. Now we just needed to piece together code for

  1. Building of the project
  2. Zipping up the project
  3. Installing the project

We could have leveraged something like PSake to do our dirty work however coming from a background of .NET/bash/batch it was actually easier to build up our script ourselves, this may change in the future.

To build the project we just used MSBuild.

$msbuild = "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\MSBuild.exe"
if (![System.IO.File]::Exists($msbuild)) {
# fall back to 32 bit version if we didn't find the 64 bit version
$msbuild = "C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe"
}
$buildcmd = "$msbuild $SolutionFile /t:Rebuild /p:Configuration=$Config"
Invoke-Expression $buildcmd

Zipping up the project we chose to use the zip method that (finally!) comes with .NET 4.5 (note: this required us to use PowerShell v3), System.IO.Compression.ZipFile.CreateFromDirectory in PowerShell it looks like this


[System.Reflection.Assembly]::LoadWithPartialName("System.IO.Compression.FileSystem")
[System.IO.Compression.ZipFile]::CreateFromDirectory($dir, $zip)

Installation of CollectedIt code was straight forward. From the beginning we created a setup.exe (that uses the excellent Insight Schema Installer from the Insight Micro-ORM project) to do an install of the SQL database that we could use on the command line (hence we could use it with PowerShell. The website only required the output from the build be copied to the production location. Again this was straight forward using PowerShell. We only had to Invoke a few commands remotely on the server to get this going. It looks something like this


Invoke-Command -SessionOption $so -ComputerName $Servers -Credential $remotecreds -ArgumentList ($Config) -ScriptBlock {
Param(
$Config
)

[System.Reflection.Assembly]::LoadWithPartialName("System.IO.Compression.FileSystem")
[System.IO.Compression.ZipFile]::ExtractToDirectory("$Config.zip", "c:\$Config")
Invoke-Expression "C:\$Config\Setup\setup.exe"
Copy-Item -Verbose -Recurse -Force "C:\$Config\Web" "D:\webroot"
}

That’s all the piece we had to put together for our ‘one click’ install. I should mention that our $remotecreds variable is populated with the Get-Credential cmdlet. For more of a streamlined process we are investigating securely storing the creds. Something along the lines of what is covered in this blog post on Importing and Exporting Credentials in PowerShell.

Hope this helps you build a streamlined process for deploying your own code with PowerShell. Drop me a line with any questions or comments with your own PowerShell deployment scripts.

Tuesday, February 19th, 2013

One of the most vivid memories I have from reading the “The Pragmatic Programmer” is the tidbit about code generation it felt like my eyes were opened to this new and exotic world. All of a sudden the DRY principle had a new weapon (to be fair I think this was the point of that section of “The Pragmatic Programmer”). Up until that point I never thought about writing a code generator.

The biggest reason for amazement was that at the time I was in the middle of developing a Windows service to kick off Microsoft Office automation tasks. I came up with a decent pattern for spinning up an external program, doing work, and ending the external process. There was retry logic with a WatchDog that added some reliability to an inherently unreliable task. The problem was that the pattern was not abstracted very well (if I was solving the problem today I would have gone for a completely different pattern, probably implementing something more like CQRS like the eMoney Nexus; live and learn). In order to keep DRY I needed to

  1. Rewrite my service (that I just demoed as working) with better encapsulation or
  2. Use code generation!

I opted for code generation.

From memory the pseudocode for the basic pattern that I wanted to reuse for different external programs:


// Initialize the office process
private Process word = new Process(...);
public void DoWork(string method)
{
// Start up the office program
word.Start();

// Spin up the watch dog
ThreadPool.QueueUserWorkItem(new WaitCallback(Watchdog, word));

// Find the method the caller wants to invoke via reflection
MethodInfo m = this.GetType().GetMember(method)[0];
m.Invoke(this, null);
}

static void ThreadProc(Watchdog stateInfo)
{
// waits for a timeout, then check process state
// if process is still running first try to stop
// the process, if that doesn't work try to kill
// the process. Then check to see if the worker
// thread is still around, and tell it to quit
// as a last resort use Thread.Abort()
}

Basically the pattern started an external program, started a watchdog, then called a method via reflection. Anytime somebody wanted to write a new task for the external program to do they would just need to write a new method in the class and call DoWork with the name of their method. It worked great when dealing with the same process. The issue was when wanting to use excel, or acrobat, or any other external program this whole pattern would need to be reimplemented with the logic that knew how to spin up that particular program, and have method that worked with that program as well.

What I came up with was this whole pattern could be injected into the class as compile time. I changed the basic files to look like this:


private Process word = new Process(...);
public void DoWork(string method) { /***** DOWORK *****/ }

This allowed anybody writing a method inside the class to still get IntelliSense on the object they wanted to work with, and they could still call DoWork. The rest of the pattern I moved to an external console program that knew how to open the class file, find the “public void DoWork(string method) { /***** DOWORK *****/ }” string and drop in custom code per external process. Something like this:


private static readonly string wordWatchDog = @"
private Process word = new Process(...);
public void DoWork(string method)
{
// Start up the office program
word.Start();

// Spin up the watch dog
ThreadPool.QueueUserWorkItem(new WaitCallback(Watchdog, word));

// Find the method the caller wants to invoke via reflection
MethodInfo m = this.GetType().GetMember(method)[0];
m.Invoke(this, null);
}

static void ThreadProc(Watchdog stateInfo)
{
// waits for a timeout, then check process state
// if process is still running first try to stop
// the process, if that doesn't work try to kill
// the process. Then check to see if the worker
// thread is still around, and tell it to quit
// as a last resort use Thread.Abort()
}"
private static readonly magicString = "public void DoWork(string method) { /***** DOWORK *****/ }";
public void UpdateWordDotCS()
{
string wordDotCSAsString = ReadWordFile();
int magicStringStart = wordDotCSAsString.IndexOf(magicString);
using (FileStream fs = File.Open(wordDotCSFile, FileMode.Create))
{
byte[] file = ASCII.ASCIIEncoding.GetBytes(wordDotCSAsString);
// Write beginning of file
fs.Write(file, 0, magicStringStart);

// Inject our custom code
byte[] customCode = ASCII.ASCIIEncoding.GetBytes(wordWatchDog);
fs.Write(customCode, 0, customCode.Length);

// Write beginning end file
fs.Write(file, magicStringStart + magicString.Length, file.Length - magicStringStart - magicString.Length);
}
}

Then our release build process was able to first call the console program that did the code generation before it started to build the actual solution.

This first foray into code generation while crude cemented code generation as a tool in my toolbox of how to get stuff done. Since this first pass I went on to make a few more code generators.

  • A custom C# project templates with default code.
  • A program that reads a DB and creates C# classes to populate the tables.
  • An app that could read a copy/pasted table for a spec on how to read a 3rd party file and generate a class that would allow the 3rd party file to be serialized into our class.

None of these generators were particularly difficult to write and the pay off in terms of productivity was noticeable. Which makes me wonder why I don’t see more people writing generators. My theory is twofold, first I think code generation just does not come to people’s mind (which is odd since 99% of the coding we do is to simplify other work, why not simplify our own coding) and second I think the term ‘code generation’ sounds scary to people.

My next few posts will go into more details on the aforementioned code generators to illustrate how they really are not scary, and to show how there is more than one way to generate code.