Playback Papertrail Audit Logs

We use paper_trail quite often, it’s a wonderful gem and generally part of our standard suite for business apps, we primarily use it for audit logging, today we needed to get data out of this history

There are quite a few ways to do this, from having daily server backups or parsing logs. They all have there quirks, so does this solution.  In this example we want to playback and take a snapshot of our orders table, let’s say value grouped by status.

Using Olark for In-App Annoucments

Anyone who supports a web application find themselves with a need to get users pumped about new features, or explain a downtime. And one thing that using Olark over the years is that conversations are always better.

Olark has a feature that can easily be abused to produce In-App Announcements, that are app, and or page specific.   The basic idea is that for every visitor (unique), we send them a message of our announcement, and we link to our blog for further details.

Shows the various settings required to use olark as in-app annoucments.Shows the various settings required to use olark as in-app annoucments.

 

In our example,

  1. We only want to message the user why someone is on to answer questions
  2. When they are a specific application (we support a lot of applications)
  3. And the important one, “Once per unique visitor”

Gets you this, hopefully with a more useful message

Screenshot 2015-10-19 10.08.58

 

Learn more about Olark or just chat with us! Olark is in the bottom right!

Feldfix

Feldfix is an online marketplace that brings together extraordinary farmers and their conscientious customers, allowing them to communicate and do business directly, just like a local farmer’s market.

Through innovative packaging and delivery, customers receive field fresh food at their doorstep from farmers who take pride in their craft, all while enjoying a unique and simple shopping experience that is founded on honesty.

As a result, customers get better food, shipped with a real story, from a farmer they know and trust.

We are currently Live and taking orders in Germany, and will soon expand to Austria.   Here at Burningpony we helped Feldfix extend and modified the Spree ecommerce platform into a marketplace, built marketing websites, and help drive the international team towards a successful launch.

Checkout the site and store at feldfix.com.   Though beware most of the products are in german only :/

That feature took how long?!?

Every programmer at one point or another is going to be asked by a manager to keep track of their time, be it for a particular client, or cost management exercise or just a time management tool. We all dream up beautiful schedules to block off our time to make it easy to bill to our clients and to help us focus, and boy is it wonderful when it plays out that way.   But in the reality of a consultancy, your phone rings at the worst time, and you have to answer it, the person calling pays for your house.

Personally this has been a battle of mine, from Profit Train, to Billable, to Money Tracker, to Timecard, to Pay Dirt, to Fresh Books, and finally Quickbooks.   Nothing was perfect, either time it was to hard to enter or it was a total pain to get the company books up to date.  Something had to change.

Then I found this.

Screenshot 2015-02-12 23.09.59

It’s concept was simple, Wakatime plugs into your text editor and sends heartbeats containing the files I am working on, including the Git project and branch. Suddenly I had all the data I needed to build the effortless time tracking software.  The idea was simple,  use wakatime’s data to determine how much time I spent on each commit, and use the commit messages to build a entry into Quickbooks, add a little cron, profit.

Correlating the data of wakatime was no where near as simple as I thought. The first attempt which focused on the time spent between each commit was clearly flawed and produced large commit times after a few hours hacking on a feature. I realized this approach was too simplistic, people might forget to commit, or work on two separate things before doing a two separate commits (rapid fire commits).

So the next solution was designed for the “rapid fire commits”, since Wakatime stores data by file we have the information we need to establish “Time per File” and use that to build “Time per Commit”.  This took a commit’s modified files, and determines when each of the file was last committed, and establishes a ‘last_modified_at’ time for that file.  We use this to get heartbeat data from Wakatime for how much time was spent on that file for that time frame, culminating in “Time per Commit” and “Time per Day per Project”.  Time per File massively improved the “rapid fire commit” issue and produced much more believeable data, but when run against my large repo’s produced very strange results.

5db7ef62b 2015-02-06 17:47:06 -0500 28 hours 5 secs         Adding ability to add documents to proposals.
        275d6d630                                  app/controllers/documents_controller
        c78df01c1                                  app/models/document.rb
        c0b5dc90a                                  app/models/invoice.rb
        c78df01c1                                  app/models/proposal.rb
        92b4f8e3f                                  app/views/documents/_documents.html
        4a4a2c462            1 sec                 app/views/documents/_files.html.haml
        3e0ff455a            4 secs                app/views/documents/index.js.erb
        932e8dcb0                                  app/views/invoices/show.html.haml
        a83734121                                  app/views/proposals/_form.html.haml
        b6eff8597                                  app/views/proposals/show.html.haml
        36e5aa268                                  config/routes.rb

Eventually we found at least one issue, diagramed below, which I will refer to as the “Split Tree” problem. In our below example, the highlighted commits have either added or modified a file titled “Readme” this file was modified in a separate branch that was later merged into master. While mapping the git tree commit 1, and commit 2 both belong to the commit at 5:05am.  Which caused this time to be counted twice. This example was solved by checking for any 2 committed files, with the same path dependent on the same SHA, we then split the difference given the time between 8:05 -> 8:20 to the “Added Feature A” commit.

split-tree

Results


a95f65cda 2015-02-10 8:20:11 -0500                                Added Feature A
                9bcccb12b        8 minutes 10 seconds                         README.md
6444dd692 2015-02-10 08:05:35 -0500                                Fixing Bug
                9bcccb12b        3 minutes 3 seconds                          README.md
9bcccb12b 2015-02-10 05:05:11 -0500                                Added Readme
                                20 minutes 48 seconds                        README.md

Obviously this is just one of the many possible issues that could arise from complex git trees, but it yields much better granular results that lends itself to further use, but this lays a foundation of truly actionable data.   In a world of noise a useable a signal is few and far between.  With Wakatime and GitWakatime you can fundamentally understand how much time was spent and where.  Make business choices and bill your clients

You can try this out on your repo right now,  if you use wakatime.   Install the code to produce these reports from https://github.com/rposborne/gitwakatime it currently outputs in text and json for further use, and has facilities to produce enumerable’s in ruby.

The Quickbooks code that plugs into the gitwakatime gem is available on request but not quite ready for open sourcing, it may or may not have secret keys in it :).

Cloud66 vs Opsworks, 1 year later

Cloud 66 is a managed, BYOS (bring your own server) PAAS that runs on many cloud platforms,  we have been using them for our flagship produce www.lumber.io, for about 2 years now.

Opsworks is a service provided by AWS that can manage EC2 or bare metal servers, with some handy pre built chef recipes. We have hosted our time keeping software with Opsworks for about 1 year now.

Similarities,

  • Manage Servers outside of their domain. Opsworks can manage bare metal, so can Cloud 66
  • Provides 90% of devops for you.
  • Provides handy management interfaces and monitoring of servers
  • Click to deploy

Cloud 66

Has been great for us, minus a major security issue early in their life, (delete all managed servers, yea it was bad) they have performed very well.  99% of rails just simple works on their platform, be it setting up the whenever gem for cron tasks, or sharding memcached between your servers it just works.  They tend to be a week or two behind bleeding edge releases which I feel is a very good balance between the latest and greatest and maintaining a stable production environment.

There service is built around “Stacks” or a set of servers built in the same manner with various add ons.  These stacks are meant to be replaced if you upgrade ruby or do anything major, and Cloud 66 provides methods to cleanly migrate and upgrade your stacks, IF you are using their managed backup.

All and all our time with them has been great,  with our only major complaint is the “nickel and diming”, most of the features are free but things like backup, even if you host it yourself costs money. Which I think is pretty poor practice,  a customer loosing data is VERY bad, and encouraging people to back up with there unmanaged server to AWS or something similar should not cost extra.

Opsworks

Is a whole different beast, it presents a surface of being easy to manage, but woah nelly is the learning curve high.  Using opsworks for anything beyond a rails getting started app you must know chef.  We use opswork to manage EC2 instances 2 app server 1 ELB + 1 RDS instance,  memchaced is installed on both app servers and shared between the two. Performance is great.  Managing, and deploying is not, very very little just works.  Any type of background jobs, cron scripts, imagemagick, gs, or anything beyond extremely basic will require custom chef scripts.

The service is also built around “Stacks” very similarly to cloud 66, you add various layers for your application, i.e. Rails, DB, Memcached. and opsworks will provision on new or existing servers.  Building servers tends to be a slow process about 20-25 minutes from the outgoing request, so don’t expect to scale so fast. Though it does provide methods to autoscale or scale based upon time which is very nice.

Opsworks, is a big boys tool, you need to know what you are doing but it can be a good blend between a fully managed PAAS and rolling your own.   My overall experience has been good, but painful at times, we are not a chef shop and have struggled with weird deployment issues.  Most of our grips are beyond the opsworks platform itself, like ELB returning a white page with no error in a 503 event, or deploys failing and bringing down the service.

Conclusion

Cloud 66 is a fantastic service that uses my favorite support tool, and clearly is here to make developers lives easier, providing PAAS on many different cloud platforms you get performance you need and the service that keeps you coming back.   Opsworks, is hosted chef, and that’s the end of the story, it will do very little for you if you don’t understand devops,  there is no “15 minute” getting started guide on this one, once it works it’s a solid tool, but you will save very little time in the devops arena.

 

Running Microsoft AD Primary Domain Controller on AWS

Everyone has aging servers, it seems they are old by the time they turn on.   Coupled with Microsofts complex licensing for server, when it came around that we needed some of the newer features naming custom certificates from Microsoft Certificate Authority we chose to spin up a simple EC2 server to make our lives easier vs jumping through hoops to upgrade our old server, or purchase new hardware.

This configuration requires a bit more than just an EC2 Instance.

  • A new Amazon VPC, if you run AD exposed to the public, you are insane.
  • A Direct VPN connection to our office
  • A Nat Instance for the private VPC to connect to the internet without having to pipe through the vpn.
  • A VPN connection, or a “bastion” instance to connect to the VPC if the primary VPN is down.
  • A security device, to act as a VPN endpoint on site. (Sonicwall TZ-210) in our case.

Below is what we came up withscilucent-ad

Bot the ELB and the Nat instance sit in a “Public” vpc subnet, while CA-1 sits in a private one.   Only the private subnet has access to the local network link in the direct VPN connection.

The Sonicwall providers monitoring via ping over the dual vpn connections to AWS and will attempt to rectify any issues by renegotiating the tunnel.  In reality this has been about 80% reliable.

For the AWS windows instance you must use an EBS backed instance or you will have a bad time.  Treat this like any other windows server you have, ensure your are backing it up properly, security is well configured and that you have a disaster plan in place.

 

 

eCTD Checker

A quick validation of the FDA eCTD spec to verify all md5’s.

This is not a full validation tool, but can save your team some hassle by rapidly checking the submission and correcting any invalid md5 Checksums.

https://github.com/rposborne/ectd_checker

Update:  We now have a cross platform version of this tool that can be used by everyday users.  Email us if you are interested!

 

Migrating to Heroku: WordPress Edition

Update:We have since migrated our systems to Pressable.

Starting the new year we decided to do some spring cleaning. We noticed we had a few instances running on rackspace that were doing next to nothing. Basically only hosting this site. Well it’s time to find a simpler host and close down some older (Ubuntu 10.10) servers.

1st finding a solution.

After a quick look around I found this https://github.com/mhoofman/wordpress-heroku. I have played with this before for some clients but they had a few needs that could not be met. But this site is VERY simple, everything should work great.

2nd Moving your Site

For the purpose of this walkthrough I am going to assume a few things

  • You have an active s3 account with Access and Secret Keys handy
  • You have followed the installation steps for wordpress-heroku
  • You have a local install of Mysql and Postgres

What do we need to do.

  1. Migrate our sites DB toPostgres
  2. Migrate Theme
  3. Upload all wp-content/uploads to s3.

DB Migration Heroku has a fantastic guide here. https://devcenter.heroku.com/articles/heroku-mysql I cheated a bit and migrated directly to the Heroku DB. Here is my config file (not the real passwords but this is what it should look like, my local environment is MAMP)

Once you have your config file set up for mysql2psql run from the directory containing your mysql2psql.yml file.
$ mysql2psql

Themes

Copy your theme to the wp-content directory of wordpress-heroku. Add it to version control and deploy.
$ git commit -a -m "Adding Theme"

$ git push heroku production:masterAt this point the site should be functional, assets should be broken but the admin should work.
FYI: Double check that the url is staying on the heroku url. WordPress may try to redirect you to the WordPress URL saved in the DB.You can add the below code to your wp_config.php to have wordpress respond the to any incoming host and ignore the setting in the DB.
define('WP_SITEURL', 'http://' . $_SERVER['HTTP_HOST']); define('WP_HOME', 'http://' . $_SERVER['HTTP_HOST']);

Assets

  1. Configure WPRO at Settings -> WPRO
  2. Log into console.aws.amazon.com and navigate to s3.
  3. Navigate to your bucket
  4. Ensure Permissions are set as such. (Everyone List)
  5. Upload the contents of wp-content/uploads to the bucket.

Cavets

  • I have had some issues uploading media from new post window.
  • Using APEX domains on heroku is dangerous and can interfere with mail delivery.

I will update this as I test this more.

Time Card

Timecard is a amazingly easy way to get your team to fill out time
sheets. 100% web based, timecard is designed to prevent errors and let employees fill out repeat time entries automatically. With many different visualizations your employees will actually enjoy doing their timesheet… (maybe).

Features


  • On the fly reporting
  • Employee Benchmarks and multi year reporting
  • As you type entry and word suggestion.
  • 100% Web Based.
  • 10 Different User Time reports
  • project management tools
  • Invoice approval and delivery.
  • Automated Late Timesheet notifications.

Screenshots