November 12th, 2017

After reading Scott Berkun‘s excellent book on Making Things Happen. I started to think through how I approach projects and mapped the topics covered the book to my approach(s).

When tackling “things to do” or projects I tend to put on a pragmatic lens of getting things done. However when I think back through all projects that have been successful in my past they all went through basically the same stages. The scope of the stages change with the size of the project, however a little request such as “can you build this query for me” to large projects such as “build the realtime data pipeline”.

Project Phases

Beginning Phases

The beginning of the project is usually brimming with questions. Why are we doing this? What exactly will we be doing? Who’s doing it? There is always a discovery piece in the beginning. To address these questions and bring clarity to the project there needs to be 3 things.

  • Vision
  • Scope
  • Timeline

These things can be large and formal, or small and informal. The point is in successful projects they exist.

Vision

A vision is the “why”. It is deceptively hard to have a good vision. Many times a vision creeps in as the answer to “what” instead of “why”. A good vision will allow you to stand behind the work that will get done as part of the project. Everything that gets built needs to be able to look back to the vision and see it’s role in the project.

https://commons.wikimedia.org/wiki/File:Iris_-_left_eye_of_a_girl.jpg

Bad visions:

  • To be in line with the industry we’re going to build a realtime data pipeline
  • Mary wants an answer by Friday
  • Pink is the best color, so all images on the website will be a pink hue.

Better visions:

  • Faster insights into our operations will allow for intra-day corrections in the business.
  • Mary is trying to find out how much money we spend weekly on lunches, next budget deadline is Friday.
  • Latest report showed people are mad when navigating our website.

There are always reasons why we are doing a project/task. Once those are known it allows for discovery on¬†what is going to be built. Also¬†inevitability at some point during a project (particularly larger projects) there will be roadblocks and frustrations. A good vision allows people to ask “why are we doing this again?” and get a reasonable answer.

It seems simple, but in my opinion a good vision is critical to a projects success. People feel like there is a meaningful goal and empowered to make their own decision as they line up with the vision. You’ll know you have a good vision when you find yourself and other repeating it..constantly.

A vision doesn’t necessarily have to be written down, but it helps if it is. It could be a document by itself, a section in a document, a email, a tweet, anything that’s searchable and reference-able.

Scope

The scope is more “what” are we going to build. There is usually a set of documents, or artifacts that come out of the scope phase. Scope is where you would write down the requirements, specs, and how we know the project is over.

Again the coming up with scope can be a large formal process that in and of itself takes a while to complete the documents for everything. Or it can be smaller items written down in an email, whiteboard, etc.

Timeline

After you have the general scope you will should get a sense on how long this will take. Are there milestones along the way. Your timeline will be the most externally communicated item in a project.

Note on “Agile”

Maybe if you are on an “agile” team you’ve already given up on this article (“sounds like waterfall, ugh”) and and never got to this point. If you haven’t given up yet though, I want to say this article is compatible with the agile process. Agile is a way to deliver on projects. A well run agile team will have the documents and phased already broken down into smaller chunks so the whole process can get iterated on and get done faster.

Mid Phases

The middle phase is what most people thing of as “work” this is when the concrete tasks of completing whatever you set out to complete is getting down.

  • Design
  • Building
  • Testing

Again depending on the scope of the projects these phases can take varying amounts of time. From weeks to minutes.

Design

Design can start at the beginning phases, however in the mid phase design is really about thinking completely through what is getting built. What trade off will there be. What needs to be done. There may have been specs already, designing here will take those specs and really do a deep dive onto what those specs really mean.

Building

TODO

Testing

TODO

End Phases

There gets to be a transition point toward the end of the project where things are “close” but not “done”. This is the toughest part of every project. If you read any sort of project management literature you’ll see some sort of chart like like

Where in reality I think it’s something more like (forgive my crude free form drawing skills)

There is a hump of effort right at the transition from Mid-End. This is where there usually the most time pressure, there may be a few late nights, tensions are usually the highest as well. The phase at the end are.

  • Winddown
  • Testing
  • Exit

Work is still happening at the start of the end phase, but there should be a mentality shift. From building to shipping.

Winddown

TODO

Testing

Testing shows up again… (TODO)

Exit Criteria

TODO

Why is this project inception?

I didn’t really talk much as to why I called this “project inception”. The thought behind “project inception” is that bigger projects will need to be broken down. All the pieces that are broken down should follow the same basic steps.

  • Vision. Scope, Timeline
  • Design
  • Building
    • Vision, Scope, Timeline
    • Design
    • Building
      • Vision, Scope, Timeline
      • Design
      • Building
      • Testing, Wind down
      • Exit Criteria
    • Testing, Wind down
    • Exit Criteria
  • Testing, Wind down
  • Exit

The outer projects would have formal documents, timelines in terms of months or even quarters, then they get narrowed down to the inner projects where items are documented in emails, or project management software, or GitHub issues (my personal favorite) and the timeline is in hours or days.

Projects are always ebbing and flowing, there is pragmatism throughout these stages. Nothing is ever set in stone, you can always go back, revisit and revise (granted some things will be much hard to change at a certain point). Successful project, large and small, do go through them and are cognizant of the fact that there are stages and which stage the current project(s) is in.

February 24th, 2015

The initial prototype of StockJock was a simple python web application (using Flask) with PostgreSQL backend. It was fun writing python for a while, however when development work really started to pick up it was decided to move to ASP.NET MVC and use C#. (I won’t get into why right now. If you really want details hit me up on Twitter.) We also wanted to keep the database in Postgres (again for details there is Twitter).

We also wanted to take advantage of the Universal Providers provided when using ASP.NET MVC out of the box. The problem was out of the box Simple Membership expected to connection to SqlServer with the “DefaultConnection” connection string. The data is all managed through EntityFramework. Luckily

ASP.NET SimpleMembership schema with PostgreSQL syntax

First step was to create a database in which to keep the membership data.

	CREATE DATABASE my_mvc_web_app;

Once we have a DB connect to the DB with a user who has permissions to create tables to create the schema. There are few particularities to pay attention to between Postgres & SQLServer.

  • The default schema in Postgres is “public”, in SQLServer it’s “dbo”. There are ways to tell Entity framework which schema to use or the way we chose was to create a “dbo” schema inside our database.
  • Postgres will treat any unquoted identifier as lowercase, where SQLServer is case agnostic with identifiers. The easiest way to keep EntityFramework from failing to find table was to quote our table identifiers to preserve case.

Here is the SQL we used to create our DB.


	CREATE SCHEMA IF NOT EXISTS dbo;

	CREATE TABLE IF NOT EXISTS dbo."AspNetUsers" (
	    "Id"                   VARCHAR   NOT NULL,
	    "Email"                VARCHAR   NULL,
	    "EmailConfirmed"       BOOLEAN   NOT NULL,
	    "PasswordHash"         VARCHAR   NULL,
	    "SecurityStamp"        VARCHAR   NULL,
	    "PhoneNumber"          VARCHAR   NULL,
	    "PhoneNumberConfirmed" BOOLEAN   NOT NULL,
	    "TwoFactorEnabled"     BOOLEAN   NOT NULL,
	    "LockoutEndDateUtc"    TIMESTAMP NULL,
	    "LockoutEnabled"       BOOLEAN   NOT NULL,
	    "AccessFailedCount"    INT       NOT NULL,
	    "UserName"             VARCHAR   NOT NULL,
	    CONSTRAINT PK_AspNetUsers PRIMARY KEY ("Id"),
		CONSTRAINT UQ_AspNetUsers UNIQUE ("UserName")
	);

	CREATE TABLE IF NOT EXISTS dbo."AspNetUserLogins" (
	    "LoginProvider" VARCHAR NOT NULL,
	    "ProviderKey"   VARCHAR NOT NULL,
	    "UserId"        VARCHAR NOT NULL,
	    CONSTRAINT PK_AspNetUserLogins PRIMARY KEY ("LoginProvider", "ProviderKey", "UserId"),
	    FOREIGN KEY ("UserId") REFERENCES dbo."AspNetUsers" ("Id") ON DELETE CASCADE
	);

	DO $$
	    BEGIN
	        CREATE INDEX IX_AspNetUserLogins_UserId ON dbo."AspNetUserLogins" ("UserId");
	    EXCEPTION
	        WHEN others THEN RAISE NOTICE 'Could not add IX_AspNetUserLogins_UserId. Does it already exist?';
	    END
	$$;


	CREATE TABLE IF NOT EXISTS dbo."AspNetRoles" (
	    "Id"   VARCHAR NOT NULL,
	    "Name" VARCHAR NOT NULL,
	    CONSTRAINT PK_AspNetRoles  PRIMARY KEY ("Id"),
		CONSTRAINT UQ_RoleName UNIQUE ("Name")
	);

	CREATE TABLE IF NOT EXISTS dbo."AspNetUserClaims" (
	    "Id"         SERIAL,
	    "UserId"     VARCHAR NOT NULL,
	    "ClaimType"  VARCHAR NULL,
	    "ClaimValue" VARCHAR NULL,
	    CONSTRAINT PK_AspNetUserClaims PRIMARY KEY ("Id"),
	    FOREIGN KEY ("UserId") REFERENCES dbo."AspNetUsers"("Id") ON DELETE CASCADE
	);

	DO $$
	    BEGIN
	        CREATE INDEX IX_AspNetUserClaims_UserId ON dbo."AspNetUserClaims" ("UserId");
	    EXCEPTION
	        WHEN others THEN RAISE NOTICE 'Could not add IX_AspNetUserClaims_UserId. Does it already exist?';
	    END
	$$;

	CREATE TABLE IF NOT EXISTS dbo."AspNetUserRoles" (
	    "UserId" VARCHAR NOT NULL,
	    "RoleId" VARCHAR NOT NULL,
	    CONSTRAINT PK_AspNetUserRoles PRIMARY KEY ("UserId", "RoleId"),
	    FOREIGN KEY ("RoleId") REFERENCES dbo."AspNetRoles"("Id") ON DELETE CASCADE,
	    FOREIGN KEY ("UserId") REFERENCES dbo."AspNetUsers"("Id") ON DELETE CASCADE
	);

	DO $$
	    BEGIN
	        CREATE INDEX IX_AspNetUserRoles_UserId ON dbo."AspNetUserRoles"("UserId");
	    EXCEPTION
	        WHEN others THEN RAISE NOTICE 'Could not add IX_AspNetUserRoles_UserId. Does it already exist?';
	    END
	$$;


	DO $$
	    BEGIN
	        CREATE INDEX IX_AspNetUserRoles_RoleId ON dbo."AspNetUserRoles"("RoleId");
	    EXCEPTION
	        WHEN others THEN RAISE NOTICE 'Could not add IX_AspNetUserRoles_RoleId. Does it already exist?';
	    END
	$$;

Hooking up EntityFramework with PostgreSQL

Now that we have a database we need our web application to communicate with it. The code the Microsoft provides for SimpleMembership uses EntityFramework. Francisco Figueiredo Jr., the creator of Npgsql project, has a great writeup on how to use npgsql with EntityFramework. I’d recommend following his blog post. The post summed up in bullet points:

  • Install the Npgsql NuGet package
  • Edit your web.config to hook in the Npgsql as DbProvider
    <System.data>
        <DbProviderFactories>
            <add name="Npgsql Data Provider" 
                    invariant="Npgsql"
                    description="Data Provider for PostgreSQL"
                    type="Npgsql.NpgsqlFactory, Npgsql" />
        </DbProviderFactories>
    </system.data>
    		
  • Edit your web.config to set DefaultConnection to your DB
    <connectionStrings>
        <add name="DefaultConnection"
             providerName="Npgsql"
              connectionString="server=127.0.0.1;userid=yoursqluser;password=sqlpass;database=my_mvc_web_app"/>
    </connectionStrings>
    	

Using SimpleMembership

Now you are all set to start hooking in authentication into your web application. Following articles like this should be able to followed without modifications since the interaction with the database is all done through the EntityFramework just setup.

January 12th, 2015

This is a quick start for getting SQL Server unit tests going inside Visual Studio. This assumes that you already have a database and that you already have a development process and an installation process for your database. If you are starting completely from scratch then you may want to just follow Microsoft’s Walkthrough for Creating and Running SQL Server Unit tests.

Steps to create

  1. Open your solution add a new ‘Unit Test Project’
    NewProject
  2. Add a new Sql Server Unit Test file by right clicking the new unit test project and selecting ‘Add New Item…’
    AddSqlServerUnitTest
  3. You will be prompted to setup your SQL Server connection. Setup a connection to a new DB.
  4. Double click on SqlServerUnitTest1.cs to bring up database test designer.
    Database Test Designer
  5. The database test designer take a little bit getting used to. The top navigation bar has 3 areas. The first area is a drop down to select a test, the second area (circled in red in the image above) lets you pick pre-test, test, or post-test, and the 3rd area is for test management (add/remove/rename).
  6. In the database test designer on the top bar click the green plus sign to create a new test.
  7. Use the 2nd drop down in the top bar (circled in red) to select a pre-test. In the main window pane write whatever SQL is needed to setup your test. Note of caution: The pre-tests and post-tests also allow for test conditions. If you accidently add a condition to a pre-test your test may fail.
  8. Again using the 2nd drop down in the top bar (circled in red) select the test. Here you will write SQL that you want to test. For example this would be where you would call a stored procedure that you wish to test.
  9. The test will need test conditions. The bottom of the designer are test conditions. There are a limited set of built in test conditions. To create your own follow the Microsoft article on Custom Test Conditions. Just select a condition and click the green plus to add it. To edit what it tests use the properties window.

At this point you will have a SQL server test that you can run. There should have been a file, SqlDatabaseSetup.cs, that was automatically created that you can edit to automatically install your DB. You can also put any setup that you need to do, i.e. dropping the DB and creating test data inside this file.